update....
(attaching latest jvm.log from Marios)
-------------------------------------
thank you for the snapshot. It just confirmed my previous guess.
You can check yourself, all the threads "8080-<N>" (search in log by
"8080-" token) are occupied by doing [1].
So, as I told, here we might face one of the following problem:
1) Web-app, by its nature requires more worker threads (I've written
in prev. mail how to increase that number).
2) Issue in web-app application logic, when applications call each
other "transitionly", using port 8080 (as I described in prev. email)
and this way occupy all the worker threads (increasing threads numbers
may help here also).
3) Issue in the web-app or web framework code. If you'll take a look -
all the occupied threads are doing the same thing - putting something
into hashmap, may be there is some neverending cycle or something like
that.
4) another issue :)
Hope this will help.
[1]
at java.util.HashMap.put(HashMap.java:374)
at
elas
.hr
.roster
.viewcontroller
.backing.FylloBacking.refreshKathikontaMap(FylloBacking.java:176)
at
elas
.hr
.roster
.viewcontroller
.backing
.FylloBacking.refreshKolyomenoi_actionListener(FylloBacking.java:651)
at sun.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
at
com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)
at
org
.apache
.myfaces
.trinidadinternal
.taglib
.util
.MethodExpressionMethodBinding
.invoke(MethodExpressionMethodBinding.java:53)
at
org
.apache
.myfaces
.trinidad
.component
.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1288)
at
org
.apache
.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:183)
at
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:444)
at javax.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:
541)
at
com
.sun
.faces
.lifecycle
.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:78)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at
com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
333)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
elas
.hr
.generic
.viewcontroller
.utilities.minichat.MiniChatFilter.doFilter(MiniChatFilter.java:49)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
elas
.hr
.generic
.viewcontroller
.utilities
.sessioncounter
.SessionCounterFilter.doFilter(SessionCounterFilter.java:51)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org
.apache
.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:
341)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org
.apache
.myfaces
.trinidadinternal
.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:247)
at
org
.apache
.myfaces
.trinidadinternal
.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:157)
at
org
.apache
.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
246)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org
.apache
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
313)
at
org
.apache
.catalina
.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at
org
.apache
.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
218)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at
com
.sun
.enterprise
.web
.PESessionLockingStandardPipeline
.invoke(PESessionLockingStandardPipeline.java:98)
at
org
.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
222)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
587)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
648)
at
org
.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:
593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:
587)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter
(DefaultProcessorTask.java:666)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess
(DefaultProcessorTask.java:597)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process
(DefaultProcessorTask.java:872)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask
(DefaultReadTask.java:341)
at
com
.sun
.enterprise
.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at
com
.sun
.enterprise
.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.portunif.PortUnificationPipeline
$PUTask.doTask(PortUnificationPipeline.java:382)
at
com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
at
com
.sun
.enterprise
.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
> I put a pause in the script, after the generation of the jvm.log and
> the stop of the server.
> Today it hung again. I think this time the file is complete.
>
>> from the latest jvm.logs I see that all the threads, responsible
>> for processing connections on port 8080, are occupied.
>> The easiest thing we can do - is just increase the threads number
>> and check if it will help. The problem could be also in the
>> application design, where you have several web applications,
>> deployed on your machine, which communicate with each other on port
>> 8080 and finally consume all the threads. Here what I mean... Let's
>> say we have 6 web applications/services: A, B, C, D, E, F and
>> assume GF, by default, has 5 threads, which process events on port
>> 8080. If you have design like:
>>
>> A --- calls-via-8080 --> B --- calls-via-8080 --> C --- calls-
>> via-8080 --> D --- calls-via-8080 --> E --- call BLOCKED --> F
>>
>> Anyway, let's try to increase thread-num first.
>> In the domain.xml pls. fine the element:
>> <request-processing header-buffer-length-in-bytes="8192" initial-
>> thread-count="2" request-timeout-in-seconds="30" thread-count="5"
>> thread-increment="1"/>
>>
>> and change attribute thread-count to let's say 20.
>