I have had a chance to study the way the memory is acting and I have
found some very interesting things.
First I will explain what the application is doing.
1. Queries a set of data from the database. The object I am getting is
the bottom table of a 4 table structure with lazy loading. I have not
been able to get the constructor expressions working so I am stuck with
getting the structure out.
2. The tests that I have been working with has approximately 20,000
objects. I batch them into a list of 25 at a time and manipulate some
strings and traverse up the relations to the top element to get a piece
of information. Then I save off the bottom element.
Thats it.
I can't keep the application from growing beyond the memory available to
it.
I started with a heap of 256, trying to kill it sooner, and ended up
getting heap space errors. I made some changes to the code, of which I
am still working on, and have slowed down the death but none the less
the death still comes. I have been able to make it to about 10000
elements before it actually dies.
I then increased the heap size to 512 and ended bringing the server to
its knees. The really interesting thing is that I don't make it through
as many elements this way though, I got to 8600 before death. Yes, it
was the same code, I didn't even redeploy the application I just
restarted.
I have tried to remove the elements from the list as I go through them
and persist them back to the DB and even request a GC every 100 elements
but the heap just keeps growing.
When it gets the heap error it retries and after about 3 retries it
finally gets to the place where it will no longer run.
When I have the larger heap size the heap just jumps to max almost
immediately but the virt and res reported by top is simply unacceptable.
Almost immediately after starting the application it jumps to 1.5G on
virt and 1G on res.
I know I am not being nice to GF and I am not trying to be, I am trying
to kill, but this should be harder than this. This memory issue is
simply hard for me to explain to my customer.
Now I would like to get constructor expression working thinking that
this will help but I have no proof right now that the memory growth will
not stop. I have to be able to run this processing for an extended time
and have GF still be running. How can I prove to my customer and myself
that GF is a good solution?
> -------- Original Message --------
> Subject: RE: Memory Keeps Growing
> From: Scott Oaks <Scott.Oaks_at_Sun.COM>
> Date: Fri, October 31, 2008 2:13 pm
> To: users_at_glassfish.dev.java.net
> Cc: GlassfishAdmin <admin_at_glassfish.dev.java.net>
>
>
> On Fri, 2008-10-31 at 12:04, Dru Devore wrote:
> > I knew about the heap not getting larger than what is specified, I
> > included the information to be complete. What I needed to get to was the
> > rest of it so thank you very much. I have a couple of
> > questions/confirmations.
> >
> > We are planning on increasing the memory for this machine. The powers
> > that be would like to get the memory under control so we can be assured
> > that increasing the memory doesn't simply delay the death. So since this
> > is my first stint on admining GF for production can I expect the memory
> > to ever max out? or will it simply keep growing tell this happens no
> > matter what I do. I am guessing that it will eventually stop but how big
> > should I expect this to get before it finally settles down?
>
> You can expect the memory to max out -- all the socket connections and
> everything else I mentioned are made gradually (e.g., a connection pool
> typically has a min and max size where the min starts at 1). Once they
> all hit the max size, they' have allocated the memory.
>
> So typically, I always set the min == max for connection pool sizes and
> request-processing threads and java heap, just so that all the memory is
> allocated up front.
>
> If the memory doesn't max out under those circumstances, there is a bug
> somewhere -- either in glassfish or the JVM itself (which we are unaware
> of), or in the postgres JDBC drivers.
>
> -Scott
>
> > Are there any docs out there for this tuning. I have been looking and
> > find a tidbit here and a tidbit there but nothing complete.
> >
> > > -------- Original Message --------
> > > Subject: Re: Memory Keeps Growing
> > > From: Scott Oaks <Scott.Oaks_at_Sun.COM>
> > > Date: Fri, October 31, 2008 10:12 am
> > > To: users_at_glassfish.dev.java.net
> > > Cc: GlassfishAdmin <admin_at_glassfish.dev.java.net>
> > >
> > >
> > > The memory usage of any Java application, including glassfish, isn't
> > > really at all dependent on your data set. The Java heap will grow to
> > > whatever size you have specified (256M in your case), and all Java
> > > objects will be managed by the JVM in that heap. There's nothing you can
> > > do in the application to really affect that -- if you tell Java to use
> > > 256M of heap, that's what it will use.
> > >
> > > The remaining memory used by the JVM is native memory for its libraries,
> > > plus any native memory your application is using (that would be the case
> > > if you are using a Type 2 JDBC driver). Assuming you're using a type 4
> > > JDBC driver, then the remaining memory is all program text and data.
> > >
> > > The program text and native memory will occupy a certain amount of space
> > > which you can't really control. But one thing is that every socket used
> > > by the application uses some native memory for its socket buffers, so
> > > you should make sure that the size of all your socket-related tunables
> > > is reasonable, particularly for your very small machine. This includes
> > > the size of your JDBC connection pool and the number of simultaneous
> > > connections your allow into the server (which means the number of
> > > keep-alive connections).
> > >
> > > The other thing that affects native memory is the thread stacks -- you
> > > can limit the size of the thread stacks by adding -Xss128k to your
> > > jvm-options; you should also make sure that the size of the HTTP
> > > request-processing threads is small (again, given your small machine).
> > >
> > > -Scott
> > >
> > > On Fri, 2008-10-31 at 09:47, Dru Devore wrote:
> > > > We are having a major memory problem and I need a solution.
> > > >
> > > > The Glassfish process continually takes memory tell there is no more and
> > > > postgres dies. I think that postgres has a little blame her for our
> > > > overall memory problem but not as much as Glassfish because Glassfish
> > > > hogs memory.
> > > >
> > > > This is our current setup:
> > > > Glassfish v2 (asadmin reports Sun Java System Application Server 9.1_01)
> > > > (-Xmx256m)
> > > > Ubuntu 8.04
> > > > Postgres 8.3
> > > > Running under VM ware
> > > > 1 GB total memory
> > > > 512 MB virtual
> > > > The application is mostly services implemented in EJB3/JPA. This is not
> > > > a large data set.
> > > >
> > > > Situation:
> > > > When GF is started it jumps to around 160m RES (from top) with around
> > > > 545m VIRT (from top). Postgres has multiple threads that with one
> > > > jumping to around 19m RES and 42408 VIRT.
> > > > Over time the GF process will be grow to up around 1250m VIRT and 683m
> > > > RES. Postgres will grow its threads to 8 with 28m RES and 42408 VIRT. At
> > > > this time pieces of the application start to fail due to postgress
> > > > failing. Postgres will eventually fail and all postgres processes will
> > > > die out.
> > > >
> > > > Like I said above the data set is not large. Basically the application
> > > > accepts around 5000 service requests/day and stores the information to
> > > > the database. It then has some EJB3 timer beans that run and pull
> > > > information out of the database, performs some business logic, and
> > > > stores some information back to the database. I am not storing any
> > > > information, besides configuration settings, in the application. All the
> > > > services are serviced with stateless session beans, multiple in most
> > > > cases.
> > > >
> > > > There are performance improvements we are looking at now which will
> > > > streamline the application and help but all we have been able to
> > > > accomplish up to this point is a delay in the errors.
> > > >
> > > > Questions:
> > > >
> > > > 1. How can I limit the GF memory?
> > > >
> > > > 2. Any tips for performance settings we should look at before moving the
> > > > application to production?
> > > >
> > > > 3. Anything I should look at in my EJB3/JPA code that could help
> > > > eliminate unneeded caching or holding of data?
> > > >
> > > > I am being forced into an admin position on this.
> > > >
> > > > Thanks for any help.
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > > > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: admin-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: admin-help_at_glassfish.dev.java.net