Jeanfrancois,
Hmm. Your comments about Windows 2003 prompt some questions:
1. If Windows 2003 is leaking nbpool wouldn't that cause the server to crash? We have had
no crashes of the server, only Glassfish slowing down. The server has been solid even
with other active IIS services running on it.
2. If the problem was in the OS, then why does restarting Glassfish clear the problem? Is
the nbpool problem a *per-process* issue?
3. If I configure to use blocking rather than non-blocking sockets, what would that do to
my throughput?
Eric Chamberlain
VentriPoint, Inc. | www.ventripoint.com | Software Engineer
Helping heart care through innovative diagnostic solutions
>> -----Original Message-----
>> From: Jeanfrancois.Arcand_at_Sun.COM
>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>> Sent: Wednesday, June 18, 2008 7:14 AM
>> To: users_at_glassfish.dev.java.net
>> Subject: Re: Glassfish Slowing Down
>>
>> Salut,
>>
>> almost forget to reply....
>>
>> Eric Chamberlain wrote:
>> > My comments are in-line below.
>> >
>> > Eric Chamberlain
>> > VentriPoint, Inc. | www.ventripoint.com | Software Engineer
>> > Helping heart care through innovative diagnostic solutions
>> >
>> >
>> >>> -----Original Message-----
>> >>> From: Jeanfrancois.Arcand_at_Sun.COM
>> >>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>> >>> Sent: Wednesday, June 11, 2008 5:46 PM
>> >>> To: users_at_glassfish.dev.java.net
>> >>> Subject: Re: Glassfish Slowing Down
>> >>>
>> >>> Salut,
>> >>>
>> >>> Eric Chamberlain wrote:
>> >>>>> Greetings all.
>> >>>>>
>> >>>>> I am seeing a problem in which my Glassfish-hosted
>> >>> service slows down over time. Over
>> >>>>> a few weeks, it slows down to 10x its original response
>> >>> time. The
>> >>>>> Glassfish framework is used only to support this one
>> >>> service (which,
>> >>>>> conveniently, has but one external API). When I stop the
>> >>> Glassfish
>> >>>>> instance and re-start it, the response time immediately
>> >>> goes back to its expected (short) interval.
>> >>> Can you gives more information about your service? Mainly,
>> >>> are you using only JSP and Servlet (no db, no remote call,
>> >>> no extrenal component)?
>> >
>> > There is no JSP or servlet involved here. All we have is
>> a web service.
>>
>> OK they might be a problem there, but I doubt.
>>
>>
>> >
>> >>> Most of the time this problem is cased by an external
>> >>> component that locks our threads.
>> >>>
>> > I don't know anything external that would do that. The
>> individual calls are short and
>> > there is no database involved.
>> >
>> >>>>> We have conducted stress testing on the service code and
>> >>> there is no
>> >>>>> slow down detectable even when we simulate many weeks
>> of use in a
>> >>>>> short time. We also could detect no bloat of the heap in
>> >>> our stress
>> >>>>> testing. BTW, the service does *not* access any databases.
>> >>>>>
>> >>>>> The next suspect is the Glassfish framework itself. How
>> >>> can I find
>> >>>>> out more information on when and if Glassfish is slowing down
>> >>>>> handling requests? Is there someway that I can post
>> an automatic
>> >>>>> monitor which will help me track down changes in response
>> >>> times over a multi-week time frame and correlate them with
>> >>> memory usage?
>> >>>
>> >>> Are you able to reproduce the problem easily? Do you think
>> >>> are you able to get a thread dump when it starts slowing down?
>> >>>
>> > Reproducing the problem takes a while but within a two or
>> three weeks the problem is
>> > perceptable. It is not possible to reproduce the problem
>> within a day. I probably could
>> > see the change over a week if I tracked closely the response times.
>>
>> When it happens, are you able to do a jstack <PID> >
>> dump.txt and send
>> it here?
>>
>>
>>
>> >
>> >>>>> Another hypothesis is that we're losing information from
>> >>>>> dis-connection and re-connections that happen over time
>> >>> (and thus are not seen in a stress test because of
>> >>>>> the compressed time frame). Have there been any
>> >>> problems in this area found by others?
>> >>>>> I could find nothing myself.
>> >>> Grizzly (the http front end of GlassFish) will close
>> >>> connections under two circumstances:
>> >>>
>> >>> (1) A connection is idle for more that 30 seconds
>> >>> (2) More than 250 requests has been made on a persistent
>> connection.
>> >>>
>> >>> Based on the above information, I suspect (1) might happens.
>> >>> But usually slow down are observed because
>> >>>
>> >>> (1) All the worker thread takes times to execute. During
>> >>> that time, all incoming requests are queued. As soon as one
>> >>> thread is free, it dequeue one request and execute it. By
>> >>> default, the queue can accept 4096 connection. After
>> >>> reaching that limit, Grizzly just start refusing requests.
>> >>>
>> >>> (2) Your application/framework is doing something wrong by
>> >>> caching/storing data (wild guess).
>> >
>> > I cannot reproduce the problem by stressing the app over
>> the short run. This leads me to
>> > think it is not in the app.
>> >
>> >>> So, first let's do the usual configuration stuff. First, can
>> >>> you add, in domain.xml, the following property:
>> >>>
>> >>> <jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=fa
>> > lse</jvm-options>
>> >
>> > I do not know what effect I should expect from this
>> change. Please explain.
>>
>> It is just disabling the following mechanism:
>>
>> http://weblogs.java.net/blog/binod/archive/2005/09/lazy_initi
>> aliza.html
>>
>> shouldn't make a difference, but just to make sure.
>>
>>
>> >
>> >>> Restart GlassFish and try to reproduce the problem. Can you
>> >>> also send your domain.xml? Are you changing the
>> >>> http-listener's acceptor-threads value by any chance? If
>> >>> yes, set it to 1 (the default) and see if the problem
>> still happens.
>> >>>
>> > I have not changed the domain.xml at all. The domain.xml
>> lists acceptor-threads = 1.
>> >
>>
>> OK then the default config run with only 5 threads:
>>
>> <request-processing ...thread-count="5".../>
>>
>> You might want to increase that number to see if it helps.
>>
>>
>>
>> >
>> >>> Also please let us know which JDK and OS version you are using.
>> >>>
>> >
>> > We're using JDK 1.6 OS is Windows Server 2003.
>>
>> Haaaaa....Window 2003 leaks nbpool and eventually the TCP
>> stack will go
>> down completely. We have reported that problem to Microsoft
>> and as far
>> as I can tell, no patch has been provided so far. So that's
>> possibly the
>> problem you are facing. Note that this is *not* a
>> GlassFish/Java issues,
>> but a win32/2003 issue. The workaround is to avoid using non
>> blocking
>> socket and instead use blocking. Add the following property
>> in domain.xml:
>>
>> -Dcom.sun.enterprise.web.connector.useCoyoteConnector=true
>>
>> to see if that help.
>>
>> A+
>>
>> -- Jeanfrancois
>>
>>
>>
>> >
>> >>> Thanks
>> >>>
>> >>> -- Jeanfrancois
>> >>>
>> > Thank you for the quick response.
>> >
>> > == Eric ==
>> >
>> >
>> >
>> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>