My comments are in-line below.
Eric Chamberlain
VentriPoint, Inc. | www.ventripoint.com | Software Engineer
Helping heart care through innovative diagnostic solutions
>> -----Original Message-----
>> From: Jeanfrancois.Arcand_at_Sun.COM
>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>> Sent: Wednesday, June 11, 2008 5:46 PM
>> To: users_at_glassfish.dev.java.net
>> Subject: Re: Glassfish Slowing Down
>>
>> Salut,
>>
>> Eric Chamberlain wrote:
>> >> Greetings all.
>> >>
>> >> I am seeing a problem in which my Glassfish-hosted
>> service slows down over time. Over
>> >> a few weeks, it slows down to 10x its original response
>> time. The
>> >> Glassfish framework is used only to support this one
>> service (which,
>> >> conveniently, has but one external API). When I stop the
>> Glassfish
>> >> instance and re-start it, the response time immediately
>> goes back to its expected (short) interval.
>> Can you gives more information about your service? Mainly,
>> are you using only JSP and Servlet (no db, no remote call,
>> no extrenal component)?
There is no JSP or servlet involved here. All we have is a web service.
>> Most of the time this problem is cased by an external
>> component that locks our threads.
>>
I don't know anything external that would do that. The individual calls are short and
there is no database involved.
>> >>
>> >> We have conducted stress testing on the service code and
>> there is no
>> >> slow down detectable even when we simulate many weeks of use in a
>> >> short time. We also could detect no bloat of the heap in
>> our stress
>> >> testing. BTW, the service does *not* access any databases.
>> >>
>> >> The next suspect is the Glassfish framework itself. How
>> can I find
>> >> out more information on when and if Glassfish is slowing down
>> >> handling requests? Is there someway that I can post an automatic
>> >> monitor which will help me track down changes in response
>> times over a multi-week time frame and correlate them with
>> memory usage?
>>
>> Are you able to reproduce the problem easily? Do you think
>> are you able to get a thread dump when it starts slowing down?
>>
Reproducing the problem takes a while but within a two or three weeks the problem is
perceptable. It is not possible to reproduce the problem within a day. I probably could
see the change over a week if I tracked closely the response times.
>> >>
>> >> Another hypothesis is that we're losing information from
>> >> dis-connection and re-connections that happen over time
>> (and thus are not seen in a stress test because of
>> >> the compressed time frame). Have there been any
>> problems in this area found by others?
>> >> I could find nothing myself.
>>
>> Grizzly (the http front end of GlassFish) will close
>> connections under two circumstances:
>>
>> (1) A connection is idle for more that 30 seconds
>> (2) More than 250 requests has been made on a persistent connection.
>>
>> Based on the above information, I suspect (1) might happens.
>> But usually slow down are observed because
>>
>> (1) All the worker thread takes times to execute. During
>> that time, all incoming requests are queued. As soon as one
>> thread is free, it dequeue one request and execute it. By
>> default, the queue can accept 4096 connection. After
>> reaching that limit, Grizzly just start refusing requests.
>>
>> (2) Your application/framework is doing something wrong by
>> caching/storing data (wild guess).
I cannot reproduce the problem by stressing the app over the short run. This leads me to
think it is not in the app.
>>
>> So, first let's do the usual configuration stuff. First, can
>> you add, in domain.xml, the following property:
>>
>> <jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=fa
lse</jvm-options>
I do not know what effect I should expect from this change. Please explain.
>>
>> Restart GlassFish and try to reproduce the problem. Can you
>> also send your domain.xml? Are you changing the
>> http-listener's acceptor-threads value by any chance? If
>> yes, set it to 1 (the default) and see if the problem still happens.
>>
I have not changed the domain.xml at all. The domain.xml lists acceptor-threads = 1.
>> Also please let us know which JDK and OS version you are using.
>>
We're using JDK 1.6 OS is Windows Server 2003.
>> Thanks
>>
>> -- Jeanfrancois
>>
Thank you for the quick response.
== Eric ==