admin@glassfish.java.net

RE: Memory Keeps Growing

From: Dru Devore <ddevore_at_duckhouse.us>
Date: Fri, 31 Oct 2008 10:26:15 -0700

Thanks for the information. It is appreciated.


>
> See http://wiki.glassfish.java.net/Wiki.jsp?page=PerformanceTuningGuide
>
> Also see Scott's blog:
> http://weblogs.java.net/blog/sdo/archive/2007/12/a_glassfish_tun.html
>
> The default heap size is 512M. You seem to have already reduced it to
> 256M. Is it because you don't have more than 1G physical memory?
> IMO, 256M looks "small".

256 is running fine for us right now. This server does not have any web
pages to it. It only performs business logic fed by web services.

>
> Analysis with top is ok, but you should also analyze it with:
> - JConsole
> - asadmin generate-jvm-report

I have been using JConsole and I will give asadmin a try also. (I like
the jvm-report thanks)

>
> It is also possible that your application leaks memory or retains references
> to heap locations you don't need. So, I'd still think that you analyze
> what the application does (e.g. for each new request).

We have been analyzing the application and are continuing to do so but
it doesn't seem to be the heap, it is hanging out around 225m, it
appears to be the DB drivers. Its like when it grabs something out of
the database the non heap memory increases. Part of the application
simply queries the database performs business logic and saves the
results back to the database. If I have elements that I need to perform
the business logic to the memory goes up. If the timer queries the DB
for an hour without any elements needing to be pulled out then the
memory is stable. It is really perplexing to me.

I am quickly coming down to increasing the memory of the VM to 2GB and
put postgres on another server. This will alleviate part of the memory
problem and allow GF to have what it needs without interrupting
postgres.
(The decision to run the whole thing together was not my decision. It
was made by people a lot more important than me)

>
> -Kedar
>
> Dru Devore wrote:
> > I knew about the heap not getting larger than what is specified, I
> > included the information to be complete. What I needed to get to was the
> > rest of it so thank you very much. I have a couple of
> > questions/confirmations.
> >
> > We are planning on increasing the memory for this machine. The powers
> > that be would like to get the memory under control so we can be assured
> > that increasing the memory doesn't simply delay the death. So since this
> > is my first stint on admining GF for production can I expect the memory
> > to ever max out? or will it simply keep growing tell this happens no
> > matter what I do. I am guessing that it will eventually stop but how big
> > should I expect this to get before it finally settles down?
> >
> > Are there any docs out there for this tuning. I have been looking and
> > find a tidbit here and a tidbit there but nothing complete.
> >
> >> -------- Original Message --------
> >> Subject: Re: Memory Keeps Growing
> >> From: Scott Oaks <Scott.Oaks_at_Sun.COM>
> >> Date: Fri, October 31, 2008 10:12 am
> >> To: users_at_glassfish.dev.java.net
> >> Cc: GlassfishAdmin <admin_at_glassfish.dev.java.net>
> >>
> >>
> >> The memory usage of any Java application, including glassfish, isn't
> >> really at all dependent on your data set. The Java heap will grow to
> >> whatever size you have specified (256M in your case), and all Java
> >> objects will be managed by the JVM in that heap. There's nothing you can
> >> do in the application to really affect that -- if you tell Java to use
> >> 256M of heap, that's what it will use.
> >>
> >> The remaining memory used by the JVM is native memory for its libraries,
> >> plus any native memory your application is using (that would be the case
> >> if you are using a Type 2 JDBC driver). Assuming you're using a type 4
> >> JDBC driver, then the remaining memory is all program text and data.
> >>
> >> The program text and native memory will occupy a certain amount of space
> >> which you can't really control. But one thing is that every socket used
> >> by the application uses some native memory for its socket buffers, so
> >> you should make sure that the size of all your socket-related tunables
> >> is reasonable, particularly for your very small machine. This includes
> >> the size of your JDBC connection pool and the number of simultaneous
> >> connections your allow into the server (which means the number of
> >> keep-alive connections).
> >>
> >> The other thing that affects native memory is the thread stacks -- you
> >> can limit the size of the thread stacks by adding -Xss128k to your
> >> jvm-options; you should also make sure that the size of the HTTP
> >> request-processing threads is small (again, given your small machine).
> >>
> >> -Scott
> >>
> >> On Fri, 2008-10-31 at 09:47, Dru Devore wrote:
> >>> We are having a major memory problem and I need a solution.
> >>>
> >>> The Glassfish process continually takes memory tell there is no more and
> >>> postgres dies. I think that postgres has a little blame her for our
> >>> overall memory problem but not as much as Glassfish because Glassfish
> >>> hogs memory.
> >>>
> >>> This is our current setup:
> >>> Glassfish v2 (asadmin reports Sun Java System Application Server 9.1_01)
> >>> (-Xmx256m)
> >>> Ubuntu 8.04
> >>> Postgres 8.3
> >>> Running under VM ware
> >>> 1 GB total memory
> >>> 512 MB virtual
> >>> The application is mostly services implemented in EJB3/JPA. This is not
> >>> a large data set.
> >>>
> >>> Situation:
> >>> When GF is started it jumps to around 160m RES (from top) with around
> >>> 545m VIRT (from top). Postgres has multiple threads that with one
> >>> jumping to around 19m RES and 42408 VIRT.
> >>> Over time the GF process will be grow to up around 1250m VIRT and 683m
> >>> RES. Postgres will grow its threads to 8 with 28m RES and 42408 VIRT. At
> >>> this time pieces of the application start to fail due to postgress
> >>> failing. Postgres will eventually fail and all postgres processes will
> >>> die out.
> >>>
> >>> Like I said above the data set is not large. Basically the application
> >>> accepts around 5000 service requests/day and stores the information to
> >>> the database. It then has some EJB3 timer beans that run and pull
> >>> information out of the database, performs some business logic, and
> >>> stores some information back to the database. I am not storing any
> >>> information, besides configuration settings, in the application. All the
> >>> services are serviced with stateless session beans, multiple in most
> >>> cases.
> >>>
> >>> There are performance improvements we are looking at now which will
> >>> streamline the application and help but all we have been able to
> >>> accomplish up to this point is a delay in the errors.
> >>>
> >>> Questions:
> >>>
> >>> 1. How can I limit the GF memory?
> >>>
> >>> 2. Any tips for performance settings we should look at before moving the
> >>> application to production?
> >>>
> >>> 3. Anything I should look at in my EJB3/JPA code that could help
> >>> eliminate unneeded caching or holding of data?
> >>>
> >>> I am being forced into an admin position on this.
> >>>
> >>> Thanks for any help.
> >>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> >>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> >> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: admin-unsubscribe_at_glassfish.dev.java.net
> > For additional commands, e-mail: admin-help_at_glassfish.dev.java.net
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: admin-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: admin-help_at_glassfish.dev.java.net