I thank you for your help and suggestions. Over the last week I have
greatly expanded my knowledge and I thank you. I am not out of the woods
yet, but I do have a lot more knowledge on what is going on.
Yesterday I was getting ready to report a bug with the persistence but
then I found that it wasn't me getting information from all the parent
tables. It was a lib I am using for some string manipulation that was
the major problem. I should have looked at it sooner. If I remove the
call to this lib my memory calms down and actually operates as I was
expecting. I haven't had the chance to investigate the lib yet but it is
grabbing memory for each string it manipulates and doesn't release it
resulting in memory growing out of control.
I think my next step, after fixing this problem, is to find the right
tools for investigating the memory and performance better.
Once again thanks for the suggestions and help they have been VERY
helpful.
> -------- Original Message --------
> Subject: Re: Memory Keeps Growing
> From: Marina Vatkina <Marina.Vatkina_at_Sun.COM>
> Date: Thu, November 06, 2008 7:16 pm
> To: users_at_glassfish.dev.java.net
>
>
> In another thread you mentioned that the constructor expressions are moving in
> the right direction.
>
> Unless something changed that I'm not aware of, TLE doesn't really support
> reading in portions of the query result. All of the result is brought in into
> memory (and into the cache unless you disable it - any entity returned from a
> query is cached), then iterated over the chunk that you asked for.
>
> If you are still trying this, try to put conditions on the 'where' clause to
> really get back parts of the result, and disable the 2nd level cache.
>
> Also, (AFAIK) em.clear() doesn't clear em cache. Only em.close() does.
>
> HTH,
> -marina
>
> Dru Devore wrote:
> > I have had a chance to study the way the memory is acting and I have
> > found some very interesting things.
> >
> > First I will explain what the application is doing.
> >
> > 1. Queries a set of data from the database. The object I am getting is
> > the bottom table of a 4 table structure with lazy loading. I have not
> > been able to get the constructor expressions working so I am stuck with
> > getting the structure out.
> >
> > 2. The tests that I have been working with has approximately 20,000
> > objects. I batch them into a list of 25 at a time and manipulate some
> > strings and traverse up the relations to the top element to get a piece
> > of information. Then I save off the bottom element.
> >
> > Thats it.
> >
> > I can't keep the application from growing beyond the memory available to
> > it.
> >
> > I started with a heap of 256, trying to kill it sooner, and ended up
> > getting heap space errors. I made some changes to the code, of which I
> > am still working on, and have slowed down the death but none the less
> > the death still comes. I have been able to make it to about 10000
> > elements before it actually dies.
> >
> > I then increased the heap size to 512 and ended bringing the server to
> > its knees. The really interesting thing is that I don't make it through
> > as many elements this way though, I got to 8600 before death. Yes, it
> > was the same code, I didn't even redeploy the application I just
> > restarted.
> >
> > I have tried to remove the elements from the list as I go through them
> > and persist them back to the DB and even request a GC every 100 elements
> > but the heap just keeps growing.
> >
> > When it gets the heap error it retries and after about 3 retries it
> > finally gets to the place where it will no longer run.
> >
> > When I have the larger heap size the heap just jumps to max almost
> > immediately but the virt and res reported by top is simply unacceptable.
> > Almost immediately after starting the application it jumps to 1.5G on
> > virt and 1G on res.
> >
> > I know I am not being nice to GF and I am not trying to be, I am trying
> > to kill, but this should be harder than this. This memory issue is
> > simply hard for me to explain to my customer.
> >
> > Now I would like to get constructor expression working thinking that
> > this will help but I have no proof right now that the memory growth will
> > not stop. I have to be able to run this processing for an extended time
> > and have GF still be running. How can I prove to my customer and myself
> > that GF is a good solution?
> >
> >
> >>-------- Original Message --------
> >>Subject: RE: Memory Keeps Growing
> >>From: Scott Oaks <Scott.Oaks_at_Sun.COM>
> >>Date: Fri, October 31, 2008 2:13 pm
> >>To: users_at_glassfish.dev.java.net
> >>Cc: GlassfishAdmin <admin_at_glassfish.dev.java.net>
> >>
> >>
> >>On Fri, 2008-10-31 at 12:04, Dru Devore wrote:
> >>
> >>>I knew about the heap not getting larger than what is specified, I
> >>>included the information to be complete. What I needed to get to was the
> >>>rest of it so thank you very much. I have a couple of
> >>>questions/confirmations.
> >>>
> >>>We are planning on increasing the memory for this machine. The powers
> >>>that be would like to get the memory under control so we can be assured
> >>>that increasing the memory doesn't simply delay the death. So since this
> >>>is my first stint on admining GF for production can I expect the memory
> >>>to ever max out? or will it simply keep growing tell this happens no
> >>>matter what I do. I am guessing that it will eventually stop but how big
> >>>should I expect this to get before it finally settles down?
> >>
> >>You can expect the memory to max out -- all the socket connections and
> >>everything else I mentioned are made gradually (e.g., a connection pool
> >>typically has a min and max size where the min starts at 1). Once they
> >>all hit the max size, they' have allocated the memory.
> >>
> >>So typically, I always set the min == max for connection pool sizes and
> >>request-processing threads and java heap, just so that all the memory is
> >>allocated up front.
> >>
> >>If the memory doesn't max out under those circumstances, there is a bug
> >>somewhere -- either in glassfish or the JVM itself (which we are unaware
> >>of), or in the postgres JDBC drivers.
> >>
> >>-Scott
> >>
> >>
> >>>Are there any docs out there for this tuning. I have been looking and
> >>>find a tidbit here and a tidbit there but nothing complete.
> >>>
> >>>
> >>>>-------- Original Message --------
> >>>>Subject: Re: Memory Keeps Growing
> >>>>From: Scott Oaks <Scott.Oaks_at_Sun.COM>
> >>>>Date: Fri, October 31, 2008 10:12 am
> >>>>To: users_at_glassfish.dev.java.net
> >>>>Cc: GlassfishAdmin <admin_at_glassfish.dev.java.net>
> >>>>
> >>>>
> >>>>The memory usage of any Java application, including glassfish, isn't
> >>>>really at all dependent on your data set. The Java heap will grow to
> >>>>whatever size you have specified (256M in your case), and all Java
> >>>>objects will be managed by the JVM in that heap. There's nothing you can
> >>>>do in the application to really affect that -- if you tell Java to use
> >>>>256M of heap, that's what it will use.
> >>>>
> >>>>The remaining memory used by the JVM is native memory for its libraries,
> >>>>plus any native memory your application is using (that would be the case
> >>>>if you are using a Type 2 JDBC driver). Assuming you're using a type 4
> >>>>JDBC driver, then the remaining memory is all program text and data.
> >>>>
> >>>>The program text and native memory will occupy a certain amount of space
> >>>>which you can't really control. But one thing is that every socket used
> >>>>by the application uses some native memory for its socket buffers, so
> >>>>you should make sure that the size of all your socket-related tunables
> >>>>is reasonable, particularly for your very small machine. This includes
> >>>>the size of your JDBC connection pool and the number of simultaneous
> >>>>connections your allow into the server (which means the number of
> >>>>keep-alive connections).
> >>>>
> >>>>The other thing that affects native memory is the thread stacks -- you
> >>>>can limit the size of the thread stacks by adding -Xss128k to your
> >>>>jvm-options; you should also make sure that the size of the HTTP
> >>>>request-processing threads is small (again, given your small machine).
> >>>>
> >>>>-Scott
> >>>>
> >>>>On Fri, 2008-10-31 at 09:47, Dru Devore wrote:
> >>>>
> >>>>>We are having a major memory problem and I need a solution.
> >>>>>
> >>>>>The Glassfish process continually takes memory tell there is no more and
> >>>>>postgres dies. I think that postgres has a little blame her for our
> >>>>>overall memory problem but not as much as Glassfish because Glassfish
> >>>>>hogs memory.
> >>>>>
> >>>>>This is our current setup:
> >>>>>Glassfish v2 (asadmin reports Sun Java System Application Server 9.1_01)
> >>>>>(-Xmx256m)
> >>>>>Ubuntu 8.04
> >>>>>Postgres 8.3
> >>>>>Running under VM ware
> >>>>>1 GB total memory
> >>>>>512 MB virtual
> >>>>>The application is mostly services implemented in EJB3/JPA. This is not
> >>>>>a large data set.
> >>>>>
> >>>>>Situation:
> >>>>>When GF is started it jumps to around 160m RES (from top) with around
> >>>>>545m VIRT (from top). Postgres has multiple threads that with one
> >>>>>jumping to around 19m RES and 42408 VIRT.
> >>>>>Over time the GF process will be grow to up around 1250m VIRT and 683m
> >>>>>RES. Postgres will grow its threads to 8 with 28m RES and 42408 VIRT. At
> >>>>>this time pieces of the application start to fail due to postgress
> >>>>>failing. Postgres will eventually fail and all postgres processes will
> >>>>>die out.
> >>>>>
> >>>>>Like I said above the data set is not large. Basically the application
> >>>>>accepts around 5000 service requests/day and stores the information to
> >>>>>the database. It then has some EJB3 timer beans that run and pull
> >>>>>information out of the database, performs some business logic, and
> >>>>>stores some information back to the database. I am not storing any
> >>>>>information, besides configuration settings, in the application. All the
> >>>>>services are serviced with stateless session beans, multiple in most
> >>>>>cases.
> >>>>>
> >>>>>There are performance improvements we are looking at now which will
> >>>>>streamline the application and help but all we have been able to
> >>>>>accomplish up to this point is a delay in the errors.
> >>>>>
> >>>>>Questions:
> >>>>>
> >>>>>1. How can I limit the GF memory?
> >>>>>
> >>>>>2. Any tips for performance settings we should look at before moving the
> >>>>>application to production?
> >>>>>
> >>>>>3. Anything I should look at in my EJB3/JPA code that could help
> >>>>>eliminate unneeded caching or holding of data?
> >>>>>
> >>>>>I am being forced into an admin position on this.
> >>>>>
> >>>>>Thanks for any help.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>---------------------------------------------------------------------
> >>>>>To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> >>>>>For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >>>>>
> >>>>
> >>>>
> >>>>---------------------------------------------------------------------
> >>>>To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> >>>>For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >>>
> >>>
> >>>---------------------------------------------------------------------
> >>>To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> >>>For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >>>
> >>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: admin-unsubscribe_at_glassfish.dev.java.net
> >>For additional commands, e-mail: admin-help_at_glassfish.dev.java.net
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net