Reposting to users_at_glassfish.java.net. (e-mail to the issues list, might
be lost in the bug e-mail)
Anthony
If you have opened a JIRA on this (GlassFish, or Shoal), please let us
know the issue ID. If you haven't you might want to use that as a place
to coordinate all this information -- as well as the responses.
-- Ed Bratt
-------- Original Message --------
Subject: Fwd: Session Replication
Date: Thu, 18 Jul 2013 23:02:16 +0200
From: Anthony Levesque <alev50_at_gmail.com>
Reply-To: issues_at_glassfish.java.net
To: issues_at_glassfish.java.net
Hello,
Can you help me with the issue in history of this mail ? I have been
redirected here by Joe Fialli from project Shoal.
Thank you.
Anthony
---------- Forwarded message ----------
From: *Fialli Joe* <joe.fialli_at_oracle.com <mailto:joe.fialli_at_oracle.com>>
Date: 2013/7/18
Subject: Re: Session Replication
To: issues_at_shoal.java.net <mailto:issues_at_shoal.java.net>
Anthony,
Thanks for providing a full set of server logs just containing the test
scenario that is failing for you.
I just want to make an overall statement that we are trying to track
down class loading issues in the web container.
So you might want to post this issue to glassfish alias instead of the
shoal alias.
So it is overkill to turn on the shoal logging to FINEST since that
subsystem could not even be responsible
for corrupted stream (grizzly is the transport for messages between
cluster members and the web container passes a
byte of serialized content into Shoal messaging system and when trying
to reconstitute replicated session,
the byte array is taken out of shoal messaging subsystem and
deserialized should be deserialized using web container class path.
While the failure is stating corrupted stream, the failure is always
exactly the same type code of "00". If it truely was a corrupted stream,
we would see different values all the time. The best bet for Type Code
00 in stream is a class not being found.
The ClassNotFoundException is getting consumed and not reported.
Possible reason for the ClassNotFoundException is the following.
My observations when looking at the server log you sent yesterday was
that it was incorrect
for there not to be a Web Container classloader context to deserialize
the session context
(after all, the class is the app loaded in the container, the default
ObjectInputStream is not going
to be able to deserialize that.)
So this failure is significant and needs interpretation by someone who
works on web container. It happens quite a bit in the server logs that
you sent in.
I am not aware why this would happen. The following might explain
trying to deserialize replicated content in web container without using
a web container
class path.
[#|2013-07-18T16:00:02.639+0200|FINEST|glassfish3.1.2|org.apache.catalina.loader.WebappLoader|_ThreadID=10;_ThreadName=Thread-2;ClassName=null;MethodName=null;|getClasspath
java.lang.NoSuchMethodException:
com.sun.enterprise.v3.server.APIClassLoaderServiceImpl$APIClassLoader.getClasspath()
at java.lang.Class.getMethod(Class.java:1607)
at
org.apache.catalina.loader.WebappLoader.getClasspath(WebappLoader.java:1196)
at
org.apache.catalina.loader.WebappLoader.setClassPath(WebappLoader.java:1145)
at org.apache.catalina.loader.WebappLoader.start(WebappLoader.java:692)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:5298)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at
org.apache.catalina.core.ContainerBase.startChildren(ContainerBase.java:1518)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1184)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:995)
at
org.apache.catalina.core.ContainerBase.startChildren(ContainerBase.java:1518)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1184)
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:529)
at org.apache.catalina.startup.Embedded.start(Embedded.java:942)
at
com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:604)
at
com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at
com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at
com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at
org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:93)
at
com.sun.enterprise.v3.services.impl.WebContainerStarter.startWebContainer(WebContainerStarter.java:202)
at
com.sun.enterprise.v3.services.impl.WebContainerStarter.postConstruct(WebContainerStarter.java:134)
at
com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at
com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at
com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
at
com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
at
com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
at
com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
|#]
*************
Here is time of first stream corrupted warning log message.
#|2013-07-18T16:06:35.588+0200|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.ha.session.management|_ThreadID=75;_ThreadName=Thread-2;|Exception
occurred in getSession
java.io.StreamCorruptedException: invalid type code: 00
Here is ClassNotFoundException that was probably related to that
failure. (Note that this is a nested anonymous class. Typically
difficult to serialize correctly.)
So the class below is probably one to look at.
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
--> Passing on ClassNotFoundException|#]
Here are a large number classes that are being looked for just before
this ClassNotFoundException log message.
[#|2013-07-18T16:06:34.749+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
--> Passing on ClassNotFoundException|#]
Note that the classes are mostly UI classes related to ajax, java faces.
[#|2013-07-18T16:06:37.493+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.web.messages)|#]
[#|2013-07-18T16:06:37.508+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.web.messages_fr)|#]
[#|2013-07-18T16:06:37.514+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.web.messages_fr_FR)|#]
[#|2013-07-18T16:06:38.123+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities)|#]
[#|2013-07-18T16:06:38.129+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities_fr)|#]
[#|2013-07-18T16:06:38.137+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities_fr_FR)|#]
[#|2013-07-18T16:06:35.921+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.component.AjaxViewRootBeanInfo)|#]
[#|2013-07-18T16:06:35.925+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIViewRootBeanInfo)|#]
[#|2013-07-18T16:06:35.926+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIComponentBaseBeanInfo)|#]
[#|2013-07-18T16:06:36.012+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.messages_fr)|#]
[#|2013-07-18T16:06:36.019+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.ajax4jsf.messages_fr_FR)|#]
[#|2013-07-18T16:06:37.178+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.sun.facelets.compiler.UIInstructionsBeanInfo)|#]
[#|2013-07-18T16:06:37.183+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.sun.facelets.compiler.UILeafBeanInfo)|#]
[#|2013-07-18T16:06:37.190+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(login)|#]
[#|2013-07-18T16:06:37.198+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(login_fr_FR)|#]
[#|2013-07-18T16:06:37.205+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.html.HtmlFormBeanInfo)|#]
[#|2013-07-18T16:06:37.207+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIFormBeanInfo)|#]
[#|2013-07-18T16:06:37.253+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.html.HtmlMessagesBeanInfo)|#]
[#|2013-07-18T16:06:37.255+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIMessagesBeanInfo)|#]
[#|2013-07-18T16:06:37.265+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.richfaces.component.html.HtmlInputTextBeanInfo)|#]
[#|2013-07-18T16:06:37.273+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIInputBeanInfo)|#]
[#|2013-07-18T16:06:37.277+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UIOutputBeanInfo)|#]
[#|2013-07-18T16:06:37.326+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.richfaces.component.html.HtmlInputSecretBeanInfo)|#]
[#|2013-07-18T16:06:37.328+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(org.richfaces.component.html.HtmlInputSecretBeanInfo)|#]
[#|2013-07-18T16:06:37.341+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.html.HtmlCommandButtonBeanInfo)|#]
[#|2013-07-18T16:06:37.344+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.UICommandBeanInfo)|#]
[#|2013-07-18T16:06:37.392+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(javax.faces.component.html.HtmlSelectBooleanCheckboxBeanInfo)|#]
[#|2013-07-18T16:06:37.195+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(login_fr)|#]
[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|loadClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|
Delegating to classloader1
org.glassfish.internal.api.DelegatingClassLoader_at_7a8f9805|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|
Searching local repositories|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
--> Passing on ClassNotFoundException|#]
[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|loadClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|
Delegating to classloader1
org.glassfish.internal.api.DelegatingClassLoader_at_7a8f9805|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|
Searching local repositories|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;|
findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]
-Joe
On 7/18/13 10:13 AM, Anthony Levesque wrote:
> Joe,
>
> Here is the scenario I played :
>
> 1 - Start cluster
> 2 - Deploy web app
> 3 - Connect through load balancer -> instance 2
> 4 - Log on
> 5 - Stop instance 2
> 6 - Refresh browser -> instance 1
> 7 - Session lost + StreamCorruptedException
> 8 - Stop cluster
>
> Logs are attached.
>
> Thanks for help.
>
> Regards,
>
> Anthony
>
> 2013/7/17 Fialli Joe <joe.fialli_at_oracle.com
> <mailto:joe.fialli_at_oracle.com>>
>
> Anthony,
>
> Forgot to include an easy way to collect all log files, see inline
> below.
>
> -Joe
>
>
> On 7/17/13 5:00 PM, Fialli Joe wrote:
>> Anthony,
>>
>> Please turn on logging for enterprise.system.container.web to
>> FINEST. That will provide classname of session being serialized
>> and deserialized.
>> Hopefully, this will give you a starting point to review the
>> classes being serialized.
>>
>> When you send along log files, try to send all of them.
> Forgot to mention to use "asadmin collect-log-files --target
> <your-cluster-name>" ....
>
> Documentation for this asadmin command and all its options are
> here:
> http://docs.oracle.com/cd/E18930_01/html/821-2433/collect-log-files-1.html
>
>
>
>> Start test scenario with clean server logs and provide all of them.
>> You edited out the initialization of the replication stores
>> (which use the class names of the sessions), that would have
>> provided
>> the names of the top level session classes. Try to limit the
>> failure scenario to one stack trace in server log, to keep server
>> log volume down.
>>
>> There is a possibility it is a class loading issue. I did notice
>> that a delegated class loader for the container was getting used
>> sometimes
>> and then when replication was deserializing, it just was using
>> default classloader, not container class loader. Not sure why
>> that is
>> happening and additional web container logging may allow us to
>> diagnose that.
>>
>> -Joe Fialli, Oracle Corporation
>>
>> On 7/17/13 12:38 PM, Anthony Levesque wrote:
>>> Hello,
>>>
>>> Thank you very much for your help.
>>>
>>> However, I have been struggling on the issue all the day without
>>> much progress.
>>>
>>> I would at least need to know which class is being deserialized
>>> and cause the exception since my web app is rather big.
>>>
>>> I have tried to play with the debugger and the transient fields
>>> on some suspected controllers without any success.
>>>
>>> There is no exception in the logs of the other instance.
>>>
>>> I send you attached to this mail my last server log in case of
>>> you can see more things than me.
>>>
>>> Is there a way to see the content of a serialized session ?
>>> Where is it stored ?
>>>
>>> For your information, it happens on the first form submit (login
>>> page in JSF) and then the app is broken.
>>>
>>> My framework is RichFaces 3.3-JSF2.
>>>
>>> Regards,
>>>
>>> Anthony
>>>
>>> 2013/7/16 Fialli Joe <joe.fialli_at_oracle.com
>>> <mailto:joe.fialli_at_oracle.com>>
>>>
>>> Alev50,
>>>
>>> Please see comments inline below.
>>>
>>> -Joe Fialli, Oracle Corporation
>>>
>>>
>>> On 7/16/13 1:41 PM, alev50_at_gmail.com
>>> <mailto:alev50_at_gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I am a Java EE developer working on a web app for
>>> GlassFish 3.1.2 Final
>>> (JSF 2 + Spring 3 + JPA). I am facing a blocking
>>> clustering issue and
>>> would like to get some help or hint to go further.
>>>
>>> I have created a simple cluster with 2 instances. My web
>>> app is
>>> deployed on the cluster with “Availibity” enabled and
>>> “distributable”
>>> property in web.xml. My controllers are serializable.
>>>
>>> The following exception points to a serialize/deserialize
>>> issue with your object.
>>>
>>>
>>> [#|2013-07-16T19:13:36.476+0200|WARNING|glassfish3.1.2|javax.enterprise
>>> .system.container.web.org
>>> <http://system.container.web.org>.glassfish.web.ha.session.management|_ThreadID
>>> =21;_ThreadName=Thread-2;|Exception occurred in getSession
>>> java.io <http://java.io>.StreamCorruptedException: invalid
>>> type code: 00
>>>
>>>
>>> You have sent along the exception produced when reading the
>>> serialized
>>> form of the session. Where there any exception when the
>>> serialized form
>>> was created ? (look in server log of other cluster member
>>> in a 2 instance cluster).
>>> Any failures when creating the serialized form could provide
>>> further assistance
>>> in understanding when the stream became corrupted.
>>>
>>> From the stack trace, there is no evidence that customized
>>> readObject/writeObject
>>> (or readExternal/writeExternal) caused serialized stream
>>> corruption.
>>> But it is an obvious question to ask in this case. If there
>>> is any customization
>>> of your controller object or any classes of the instances
>>> reachable from the controller,
>>> I recommend a junit serialize/deserialize test of the
>>> controller with sample
>>> data that matches the failure case (if that is possible). A
>>> confirmation that serialize/deserialize works for your
>>> controller classes is the first step for researching this issue.
>>>
>>> Sometimes there are fields in a class that should be marked
>>> as transient so their data is not serialized.
>>> (since the value would not make sense outside of the
>>> original JVM. For example, a pointer to a GUI
>>> component should not be serialized outside its original JVM.
>>>
>>> Hope these suggestions help isolate the issue.
>>>
>>>
>>> Here is the stack I get repeatedly :
>>>
>>> [#|2013-07-16T19:13:36.476+0200|FINER|glassfish3.1.2|javax.enterprise.s
>>> ystem.container.web.org
>>> <http://ystem.container.web.org>.glassfish.web.loader|_ThreadID=21;_ThreadName=T
>>> hread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName
>>> =loadClass;|loadClass(com.transat.ga2010.service.TitreServiceImpl)|#]
>>>
>>> [#|2013-07-16T19:13:36.476+0200|FINER|glassfish3.1.2|javax.enterprise.s
>>> ystem.container.web.org
>>> <http://ystem.container.web.org>.glassfish.web.loader|_ThreadID=21;_ThreadName=T
>>> hread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName
>>> =loadClass;| Returning class from cache|#]
>>>
>>> [#|2013-07-16T19:13:36.476+0200|WARNING|glassfish3.1.2|javax.enterprise
>>> .system.container.web.org
>>> <http://system.container.web.org>.glassfish.web.ha.session.management|_ThreadID
>>> =21;_ThreadName=Thread-2;|Exception occurred in getSession
>>> java.io <http://java.io>.StreamCorruptedException:
>>> invalid type code: 00
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1354)
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1969
>>> )
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:177
>>> 5)
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
>>> at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
>>> at
>>> org.apache.catalina.session.StandardSession.deserialize(StandardSession
>>> .java:1144)
>>> at
>>> org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
>>> at
>>> org.glassfish.web.ha.session.management.ReplicationStore.getSession(Rep
>>> licationStore.java:533)
>>> at
>>> org.glassfish.web.ha.session.management.ReplicationStore.getSession(Rep
>>> licationStore.java:485)
>>> at
>>> org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackin
>>> gStore(ReplicationStore.java:404)
>>> at
>>> org.glassfish.web.ha.session.management.ReplicationStore.load(Replicati
>>> onStore.java:387)
>>> at
>>> org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentMa
>>> nagerBase.java:1052)
>>> at
>>> org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentMana
>>> gerBase.java:1010)
>>> at
>>> org.glassfish.web.ha.session.management.ReplicationManagerBase.findSess
>>> ion(ReplicationManagerBase.java:156)
>>> at
>>> org.apache.catalina.connector.Request.isRequestedSessionIdValid(Request
>>> .java:2686)
>>> at
>>> org.apache.catalina.connector.Request.parseSessionCookiesId(Request.jav
>>> a:3711)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdap
>>> ter.java:625)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.jav
>>> a:271)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
>>> 231)
>>> at
>>> com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.cal
>>> l(ContainerMapper.java:317)
>>> at
>>> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMa
>>> pper.java:195)
>>> at
>>> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860
>>> )
>>> at
>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
>>> at
>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
>>> at
>>> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilte
>>> r.java:229)
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProto
>>> colChain.java:137)
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>> 104)
>>> at
>>> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>> 90)
>>> at
>>> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:7
>>> 9)
>>> at
>>> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTas
>>> k.java:54)
>>> at
>>> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.ja
>>> va:59)
>>> at
>>> com.sun.grizzly.ContextTask.run(ContextTask.java:71)
>>> at
>>> com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPoo
>>> l.java:532)
>>> at
>>> com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.j
>>> ava:513)
>>> at java.lang.Thread.run(Thread.java:662)
>>> |#]
>>>
>>> [#|2013-07-16T19:13:36.476+0200|FINE|glassfish3.1.2|javax.enterprise.sy
>>> <http://javax.enterprise.sy>
>>> stem.container.web.org
>>> <http://stem.container.web.org>.glassfish.web.ha.session.management|_ThreadID=21
>>> ;_ThreadName=Thread-2;ClassName=org.glassfish.web.ha.session.management
>>> .ReplicationManagerBase;MethodName=findSession;|in
>>> findSession:
>>> version=1|#]
>>>
>>> [#|2013-07-16T19:13:36.476+0200|FINE|glassfish3.1.2|javax.enterprise.sy
>>> <http://javax.enterprise.sy>
>>> stem.container.web.org
>>> <http://stem.container.web.org>.glassfish.web.ha.session.management|_ThreadID=21
>>> ;_ThreadName=Thread-2;ClassName=org.glassfish.web.ha.session.management
>>> .ReplicationManagerBase;MethodName=findSession;|Required
>>> version 1|#]
>>>
>>> I imagine you have a lot of work with the new version of
>>> GlassFish, but
>>> I wish you could have a look at it. Maybe the bug
>>> problem still exists
>>> in the new version.
>>>
>>>
>>>
>>
>
>