Below is the setting that suggested by Sun Engineer during their visit to customer site in the early stage. FYI, the suggested configuration still didn't help and their already collect the thread dump during that time. But as I know they still didn't come back yet with the outcome.
1. Change the compiler from "client" to "server"
2. Added Initial heap size to "1024M" (-Xms settting)
3. Added parallel garbage collector setting -XX:+UseParallelOldGC
4. Added Concurrent Garbage Collector setting -XX:+UseConcMarkSweepGC
After that our engineer change the jvm setting to what I've given earlier and the OOM error don't show up but the system still hang till today.
This is the OOM message during that time.
[#|2009-11-10T14:35:43.738+0800|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=13;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=d4766bf0-6703-4a2e-8ddf-9ec11185c4ea;|ApplicationDispatcher[/sppekl] PWC1231: Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
at java.lang.Class.getDeclaredField(Class.java:1880)
at com.thoughtworks.xstream.mapper.AttributeMapper.getField(AttributeMapper.java:120)
at com.thoughtworks.xstream.mapper.AttributeMapper.getConverterFromAttribute(AttributeMapper.java:101)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromAttribute(MapperWrapper.java:114)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:145)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:45)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:46)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:195)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:125)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:56)
|#]
Some warning message that I capture this evening.
[#|2010-01-12T14:42:10.100+0800|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=17;_ThreadName=httpSSLWorkerThread-8080-2;_RequestID=0636be83-81de-4690-be54-23a7749d99f2;|
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at com.htp.filter.LoginFilter.doFilter(LoginFilter.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
|#]
Is there any ways to know our application is caching lots of data?
[Message sent by forum member 'badul' (dzafarul_at_gmail.com)]
http://forums.java.net/jive/thread.jspa?messageID=380280