users@glassfish.java.net

Re: GlassFish 2.1.1 Hanging

From: Oleksiy Stashok <Oleksiy.Stashok_at_Sun.COM>
Date: Thu, 29 Apr 2010 16:55:35 +0200

update....

(attaching latest jvm.log from Marios)

-------------------------------------

thank you for the snapshot. It just confirmed my previous guess.
You can check yourself, all the threads "8080-<N>" (search in log by
"8080-" token) are occupied by doing [1].

So, as I told, here we might face one of the following problem:

1) Web-app, by its nature requires more worker threads (I've written
in prev. mail how to increase that number).
2) Issue in web-app application logic, when applications call each
other "transitionly", using port 8080 (as I described in prev. email)
and this way occupy all the worker threads (increasing threads numbers
may help here also).
3) Issue in the web-app or web framework code. If you'll take a look -
all the occupied threads are doing the same thing - putting something
into hashmap, may be there is some neverending cycle or something like
that.
4) another issue :)

Hope this will help.


[1]

    at java.util.HashMap.put(HashMap.java:374)
    at
elas
.hr
.roster
.viewcontroller
.backing.FylloBacking.refreshKathikontaMap(FylloBacking.java:176)
    at
elas
.hr
.roster
.viewcontroller
.backing
.FylloBacking.refreshKolyomenoi_actionListener(FylloBacking.java:651)
    at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
    at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
    at
com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)
    at
org
.apache
.myfaces
.trinidadinternal
.taglib
.util
.MethodExpressionMethodBinding
.invoke(MethodExpressionMethodBinding.java:53)
    at
org
.apache
.myfaces
.trinidad
.component
.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1288)
    at
org
.apache
.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:183)
    at
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:444)
    at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:
541)
    at
com
.sun
.faces
.lifecycle
.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:78)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
    at
com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
333)
    at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at
elas
.hr
.generic
.viewcontroller
.utilities.minichat.MiniChatFilter.doFilter(MiniChatFilter.java:49)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
    at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at
elas
.hr
.generic
.viewcontroller
.utilities
.sessioncounter
.SessionCounterFilter.doFilter(SessionCounterFilter.java:51)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
    at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at
org
.apache
.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:
341)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
    at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at
org
.apache
.myfaces
.trinidadinternal
.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:247)
    at
org
.apache
.myfaces
.trinidadinternal
.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:157)
    at
org
.apache
.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
    at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
    at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at
org
.apache
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
313)
    at
org
.apache
.catalina
.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
    at
org
.apache
.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
218)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
    at
com
.sun
.enterprise
.web
.PESessionLockingStandardPipeline
.invoke(PESessionLockingStandardPipeline.java:98)
    at
org
.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
222)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
587)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
    at
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
    at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
    at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
587)
    at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
    at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
    at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter
(DefaultProcessorTask.java:666)
    at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess
(DefaultProcessorTask.java:597)
    at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process
(DefaultProcessorTask.java:872)
    at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask
(DefaultReadTask.java:341)
    at
com
.sun
.enterprise
.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
    at
com
.sun
.enterprise
.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
    at com.sun.enterprise.web.portunif.PortUnificationPipeline
$PUTask.doTask(PortUnificationPipeline.java:382)
    at
com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
    at
com
.sun
.enterprise
.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)


> I put a pause in the script, after the generation of the jvm.log and
> the stop of the server.
> Today it hung again. I think this time the file is complete.
>
>> from the latest jvm.logs I see that all the threads, responsible
>> for processing connections on port 8080, are occupied.
>> The easiest thing we can do - is just increase the threads number
>> and check if it will help. The problem could be also in the
>> application design, where you have several web applications,
>> deployed on your machine, which communicate with each other on port
>> 8080 and finally consume all the threads. Here what I mean... Let's
>> say we have 6 web applications/services: A, B, C, D, E, F and
>> assume GF, by default, has 5 threads, which process events on port
>> 8080. If you have design like:
>>
>> A --- calls-via-8080 --> B --- calls-via-8080 --> C --- calls-
>> via-8080 --> D --- calls-via-8080 --> E --- call BLOCKED --> F
>>
>> Anyway, let's try to increase thread-num first.
>> In the domain.xml pls. fine the element:
>> <request-processing header-buffer-length-in-bytes="8192" initial-
>> thread-count="2" request-timeout-in-seconds="30" thread-count="5"
>> thread-increment="1"/>
>>
>> and change attribute thread-count to let's say 20.
>