Which JVM version ?
Can you see if OS is paging ?
What is the result of jmap -heap before serving applications and near the OOME ?
I really recommend for you to generate and analyze heap dump and see
if there are memory leaking somewhere or big objects.
A good software to try out is Eclipse Memory Analyzer
(eclipse.org/mat), to analyze heap dumps, checkout their blogs for
useful tips.
Claudio Miranda
On Thu, Nov 6, 2008 at 4:57 PM, Dru Devore <ddevore_at_duckhouse.us> wrote:
> I have had a chance to study the way the memory is acting and I have
> found some very interesting things.
>
> First I will explain what the application is doing.
>
> 1. Queries a set of data from the database. The object I am getting is
> the bottom table of a 4 table structure with lazy loading. I have not
> been able to get the constructor expressions working so I am stuck with
> getting the structure out.
>
> 2. The tests that I have been working with has approximately 20,000
> objects. I batch them into a list of 25 at a time and manipulate some
> strings and traverse up the relations to the top element to get a piece
> of information. Then I save off the bottom element.
>
> Thats it.
>
> I can't keep the application from growing beyond the memory available to
> it.
>
> I started with a heap of 256, trying to kill it sooner, and ended up
> getting heap space errors. I made some changes to the code, of which I
> am still working on, and have slowed down the death but none the less
> the death still comes. I have been able to make it to about 10000
> elements before it actually dies.
>
> I then increased the heap size to 512 and ended bringing the server to
> its knees. The really interesting thing is that I don't make it through
> as many elements this way though, I got to 8600 before death. Yes, it
> was the same code, I didn't even redeploy the application I just
> restarted.
>
> I have tried to remove the elements from the list as I go through them
> and persist them back to the DB and even request a GC every 100 elements
> but the heap just keeps growing.
>
> When it gets the heap error it retries and after about 3 retries it
> finally gets to the place where it will no longer run.
>
> When I have the larger heap size the heap just jumps to max almost
> immediately but the virt and res reported by top is simply unacceptable.
> Almost immediately after starting the application it jumps to 1.5G on
> virt and 1G on res.
>
> I know I am not being nice to GF and I am not trying to be, I am trying
> to kill, but this should be harder than this. This memory issue is
> simply hard for me to explain to my customer.
>
> Now I would like to get constructor expression working thinking that
> this will help but I have no proof right now that the memory growth will
> not stop. I have to be able to run this processing for an extended time
> and have GF still be running. How can I prove to my customer and myself
> that GF is a good solution?
--
Claudio Miranda
_______________________________________________