Salut,
Eric Chamberlain wrote:
> Jeanfrancois,
>
> Hmm. Your comments about Windows 2003 prompt some questions:
>
> 1. If Windows 2003 is leaking nbpool wouldn't that cause the server to crash? We have had
> no crashes of the server, only Glassfish slowing down. The server has been solid even
> with other active IIS services running on it.
Right, but it takes time before it crash. Are you monitoring nbpool
value (use the task monitor)
> 2. If the problem was in the OS, then why does restarting Glassfish clear the problem? Is
> the nbpool problem a *per-process* issue?
It shouldn't make any difference I agree, but since this is an OS issue,
I can't speak for the JDK. Maybe the JDK is the issue. I would monitor
the nbpool to be sure...might takes time, but it will eventually crash.
> 3. If I configure to use blocking rather than non-blocking sockets, what would that do to
> my throughput?
Will improve a little at the cost of scalability (one thread per
connection instead on one thread per request).
Thanks
-- Jeanfrancois
>
>
> Eric Chamberlain
> VentriPoint, Inc. | www.ventripoint.com | Software Engineer
> Helping heart care through innovative diagnostic solutions
>
>
>>> -----Original Message-----
>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>> Sent: Wednesday, June 18, 2008 7:14 AM
>>> To: users_at_glassfish.dev.java.net
>>> Subject: Re: Glassfish Slowing Down
>>>
>>> Salut,
>>>
>>> almost forget to reply....
>>>
>>> Eric Chamberlain wrote:
>>>> My comments are in-line below.
>>>>
>>>> Eric Chamberlain
>>>> VentriPoint, Inc. | www.ventripoint.com | Software Engineer
>>>> Helping heart care through innovative diagnostic solutions
>>>>
>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>>>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>>>> Sent: Wednesday, June 11, 2008 5:46 PM
>>>>>> To: users_at_glassfish.dev.java.net
>>>>>> Subject: Re: Glassfish Slowing Down
>>>>>>
>>>>>> Salut,
>>>>>>
>>>>>> Eric Chamberlain wrote:
>>>>>>>> Greetings all.
>>>>>>>>
>>>>>>>> I am seeing a problem in which my Glassfish-hosted
>>>>>> service slows down over time. Over
>>>>>>>> a few weeks, it slows down to 10x its original response
>>>>>> time. The
>>>>>>>> Glassfish framework is used only to support this one
>>>>>> service (which,
>>>>>>>> conveniently, has but one external API). When I stop the
>>>>>> Glassfish
>>>>>>>> instance and re-start it, the response time immediately
>>>>>> goes back to its expected (short) interval.
>>>>>> Can you gives more information about your service? Mainly,
>>>>>> are you using only JSP and Servlet (no db, no remote call,
>>>>>> no extrenal component)?
>>>> There is no JSP or servlet involved here. All we have is
>>> a web service.
>>>
>>> OK they might be a problem there, but I doubt.
>>>
>>>
>>>>>> Most of the time this problem is cased by an external
>>>>>> component that locks our threads.
>>>>>>
>>>> I don't know anything external that would do that. The
>>> individual calls are short and
>>>> there is no database involved.
>>>>
>>>>>>>> We have conducted stress testing on the service code and
>>>>>> there is no
>>>>>>>> slow down detectable even when we simulate many weeks
>>> of use in a
>>>>>>>> short time. We also could detect no bloat of the heap in
>>>>>> our stress
>>>>>>>> testing. BTW, the service does *not* access any databases.
>>>>>>>>
>>>>>>>> The next suspect is the Glassfish framework itself. How
>>>>>> can I find
>>>>>>>> out more information on when and if Glassfish is slowing down
>>>>>>>> handling requests? Is there someway that I can post
>>> an automatic
>>>>>>>> monitor which will help me track down changes in response
>>>>>> times over a multi-week time frame and correlate them with
>>>>>> memory usage?
>>>>>>
>>>>>> Are you able to reproduce the problem easily? Do you think
>>>>>> are you able to get a thread dump when it starts slowing down?
>>>>>>
>>>> Reproducing the problem takes a while but within a two or
>>> three weeks the problem is
>>>> perceptable. It is not possible to reproduce the problem
>>> within a day. I probably could
>>>> see the change over a week if I tracked closely the response times.
>>> When it happens, are you able to do a jstack <PID> >
>>> dump.txt and send
>>> it here?
>>>
>>>
>>>
>>>>>>>> Another hypothesis is that we're losing information from
>>>>>>>> dis-connection and re-connections that happen over time
>>>>>> (and thus are not seen in a stress test because of
>>>>>>>> the compressed time frame). Have there been any
>>>>>> problems in this area found by others?
>>>>>>>> I could find nothing myself.
>>>>>> Grizzly (the http front end of GlassFish) will close
>>>>>> connections under two circumstances:
>>>>>>
>>>>>> (1) A connection is idle for more that 30 seconds
>>>>>> (2) More than 250 requests has been made on a persistent
>>> connection.
>>>>>> Based on the above information, I suspect (1) might happens.
>>>>>> But usually slow down are observed because
>>>>>>
>>>>>> (1) All the worker thread takes times to execute. During
>>>>>> that time, all incoming requests are queued. As soon as one
>>>>>> thread is free, it dequeue one request and execute it. By
>>>>>> default, the queue can accept 4096 connection. After
>>>>>> reaching that limit, Grizzly just start refusing requests.
>>>>>>
>>>>>> (2) Your application/framework is doing something wrong by
>>>>>> caching/storing data (wild guess).
>>>> I cannot reproduce the problem by stressing the app over
>>> the short run. This leads me to
>>>> think it is not in the app.
>>>>
>>>>>> So, first let's do the usual configuration stuff. First, can
>>>>>> you add, in domain.xml, the following property:
>>>>>>
>>>>>> <jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=fa
>>>> lse</jvm-options>
>>>>
>>>> I do not know what effect I should expect from this
>>> change. Please explain.
>>>
>>> It is just disabling the following mechanism:
>>>
>>> http://weblogs.java.net/blog/binod/archive/2005/09/lazy_initi
>>> aliza.html
>>>
>>> shouldn't make a difference, but just to make sure.
>>>
>>>
>>>>>> Restart GlassFish and try to reproduce the problem. Can you
>>>>>> also send your domain.xml? Are you changing the
>>>>>> http-listener's acceptor-threads value by any chance? If
>>>>>> yes, set it to 1 (the default) and see if the problem
>>> still happens.
>>>> I have not changed the domain.xml at all. The domain.xml
>>> lists acceptor-threads = 1.
>>> OK then the default config run with only 5 threads:
>>>
>>> <request-processing ...thread-count="5".../>
>>>
>>> You might want to increase that number to see if it helps.
>>>
>>>
>>>
>>>>>> Also please let us know which JDK and OS version you are using.
>>>>>>
>>>> We're using JDK 1.6 OS is Windows Server 2003.
>>> Haaaaa....Window 2003 leaks nbpool and eventually the TCP
>>> stack will go
>>> down completely. We have reported that problem to Microsoft
>>> and as far
>>> as I can tell, no patch has been provided so far. So that's
>>> possibly the
>>> problem you are facing. Note that this is *not* a
>>> GlassFish/Java issues,
>>> but a win32/2003 issue. The workaround is to avoid using non
>>> blocking
>>> socket and instead use blocking. Add the following property
>>> in domain.xml:
>>>
>>> -Dcom.sun.enterprise.web.connector.useCoyoteConnector=true
>>>
>>> to see if that help.
>>>
>>> A+
>>>
>>> -- Jeanfrancois
>>>
>>>
>>>
>>>>>> Thanks
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>> Thank you for the quick response.
>>>>
>>>> == Eric ==
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>