Salut,
catching up...
glassfish_at_javadesktop.org wrote:
> Thanks for the quick reply. The situation is a bit complicated but if you have time to review it I would appreciate it.
>
> Environment--
> Sun Java System Application Server 9.1_02
> 2 physical machines
> 4 Applications deployed to 4 different clusters. One is heavily used, one is moderately used, and the other two are minimally used.
> 2 Sun Web Server 6 for load balancing. (Soon to be removed from environment.)
> https passthrough enabled.
> F5 big-ip in Front of Web Servers, for load balancing to the Web Servers. (Will load balance the applications directly soon.)
>
> The heavily used application will usually slows down in the mornings, M-F, until it becomes unresponsive, usually there is just a blank white page and the browser keeps working on the requests. (On the weekends the application usage is cut in half and we do not see the problem.) The other applications don't appear to be affected. If I go to the application directly (through a proxy the fronts our DMZ) on the http listener then there is no problem with the application. Unfortunately our proxy didn't allow https on the non-standard ports the applications were using. I have since changed this so I can check if the https listener is having problems when users say the application is slow. The logs don't show anything that would point to a root cause from what I can see. When I shutdown the cluster instances there are errors but the instances do shutdown. I don't know if that is normal as I have not watched the logs while shutting them down in the past.
>
> Initially I thought it was a problem with the web server. I was able to increase the performance significantly by changing these settings on the web server--
> upped the Acceptor Threads from 1 to 4.
> increased the Max Queue length 8182 (I don't think that mattered now as the peak never reached very high.)
> increased the RqThrottle to 512.
>
> But improving the performance on the web server didn't fix the problem. There were some application code changes which didn't help either. The last change I made was changing the http "engine" from grizzly to coyote. This was done last Friday. There have not been any confirmed problems since then.
You might not have configured enough threads. The difference between
Coyote and Grizzly is that Coyote magically creates 150 threads if you
switch to it.
(This morning there was one site that said the application was slow
again but I wasn't able to duplicate the problem.
That probably means the application's execution is blocked, most of the
time by a database call that never ends up. When you observe a slow
down, would you be able to grab a thread dump so I can see where the
lock happens? Mainly, all Grizzly worker threads (5 by default) are
blocked and all other requests are pilling in a queue, waiting to be
executed. When you are getting blank page, it just means Grizzly start
rejecting requests because its internal queue is full. You can increase
the queue's size by increasing the connection-pool element of
domain.xml, but that will not fix your issue. You either need to
increase the number of threads, or understand why all threads deadlock.
If you can send me a
${java.home}/bin/jstack <PID> (using JDK 6 if you can), It will tells us
where it locks.
When asked for more details they said the performance issues went away.)
Yes because Coyote have more threads :-)
>
> So hopefully by changing the http listener or engine (what is the proper term?) to coyote fixed the problem.
It will eventually lock also with Coyote, just need a much more heavier
load.
But since there have been no errors in the logs that would say were the
problem is I am still looking into other possible explanations as to why
this is/was happening.
>
> That is why I am thinking it is a Memory Leak in the App Server. The other possibility I am thinking of is a JDBC connection leak.
I would say lock ( I doubt it is related to a memory leak).
I am looking into enabling monitoring it since it doesn't appear to be
enabled by default.
>
> Anyway, if you have read until here then thank you for your time.
You are welcome.
A+
-- Jeanfrancois
> [Message sent by forum member 'jfaldmo' (jfaldmo)]
>
> http://forums.java.net/jive/thread.jspa?messageID=324113
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>