dev@glassfish.java.net

Re: deployment error/thread death

From: Ashish Sahni <Ashish.Sahni_at_Sun.COM>
Date: Thu, 19 Apr 2007 14:14:11 -0700

Hi Tim,
I'm glad you bring serialization up. Please see my comment inline:

Tim Quinn wrote:
> Hi, Jan.
>
> Really great detective work on this. I probably should have followed
> this more closely given that the serialization logic I added was involved.
>
> In your research into this, did you see any place that the
> serialization code could have protected better against something like
> this?
>
> Because the serialization (and deserialization) of descriptors was a
> performance-only enhancement, I wrote the SerializedDescriptorHelper
> class to catch all Throwables that emerge from the attempt to
> serialize. (Although without the serialized descriptor the next
> reload of the app would take longer, an exception during serialization
> would cause no functional problem.)
The exception thrown during redployment does render the webapp unusable.
So in that ir does cause the functional problem. Thoughts ?

Thanks
Ashish


> The JavaDoc for ThreadDeath says that this type of error should be
> rethrown - I guess to actually make sure the thread does stop. But as
> written the logic does not do that - it logs the warning with the
> stack trace and continues.
>
> In this case do you think it would be better if the code caught
> Exception rather than Throwable, which would let ThreadDeath and other
> descendants of Error (as opposed to Exception) bypass the handling and
> "float" to the top?
>
> Thanks.
>
> - Tim
>
> Jan Luehe wrote:
>> Ashish,
>>
>> Ashish Sahni wrote On 04/03/07 04:45 PM,:
>>
>>> Hi Jeanfrancois,
>>>
>>> First to clarify since you mention 'undeployed' - maybe you're
>>> implying implicit undeployment
>>> that might be happening during re-deployment but in any case this is
>>> what I do:
>>> 1. Deploy app
>>> 2. Request app services
>>> 3. Deploy (same) app (again) ie no explicit undeployment
>>>
>>> Secondly, I'm not sure if application is 'holding the classloader'
>>> since the application uses
>>> the jruby runtime libraries. I'm cc'ing Charles and Thomas, the
>>> jruby developers how might
>>> be able to shed light on your question.
>>
>>
>> I've investigated this issue further, and it indeed turns out to be an
>> issue with jruby.
>>
>> The java.lang.ThreadDeath is thrown during redeployment of
>> depot_l2.war, when the deployment code attempts to serialize the
>> webapp's AppDescriptor, and in doing so, needs to acquire a
>> MessageDigest
>> instance for computing a serialization UID:
>>
>> java.lang.ThreadDeath
>> at
>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
>>
>> ...
>> at java.lang.Class.newInstance(Class.java:303)
>> at java.security.Provider$Service.newInstance(Provider.java:1130)
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
>> at java.security.Security.getImpl(Security.java:658)
>> at java.security.MessageDigest.getInstance(MessageDigest.java:122)
>> at
>> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
>>
>> ...
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>> at
>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>
>> at
>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>
>> at
>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>
>> at
>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>
>> ...
>> at
>> com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
>>
>> at
>> com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
>>
>>
>>
>> It turns out that jruby registers its own provider for cryptographic
>> services (named "BC"), but never unregisters it during undeployment,
>> causing the subsequent redeployment code to attempt to grab a
>> MessageDigest instance whose classloader has been deactivated, which
>> in turn has the deactivated classloader (an instance of
>> WebappClassLoader) throw the ThreadDeath with the above stacktrace.
>>
>> I've instrumented java.security.Security.addProvider() and
>> removeProvider() to print the caller's stacktrace. I see
>> Security.addProvider() getting invoked when accessing
>> http://<host>:<port>/depot_l2/store/, see attached stack trace, but I
>> don't see any invocation of Security.removeProvider during
>> undeployment.
>>
>> jruby needs to unregister its provider for cryptographic services
>> during undeployment, by calling
>> java.security.Security.removeProvider("BC").
>>
>> This could be done inside
>> org.jruby.webapp.AbstractRailsServlet.destroy().
>>
>>
>> Jan
>>
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>> Jeanfrancois Arcand wrote:
>>>
>>>> Hi Ashish,
>>>>
>>>> there is two possibilities:
>>>>
>>>> (1) There is a thread pool or object in GlassFish that store the
>>>> ContextClassloader and try to re-use it. The ContextClassLoader of
>>>> a web application is always a WebappClassloader, and that
>>>> classloader is marked dead when the application is undeployed. In
>>>> 9.0, we fixed a bug in the ORB thread pool related to that problem.
>>>> This time I suspect the component that create it is:
>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>>>
>>>>
>>>>
>>>> (2) Your application is holding the classloader. Are you loading
>>>> classes using the ContextClassloader and try to re-use that
>>>> Classloader?
>>>>
>>>> If your application doesn't use or store the ContextClassLoader,
>>>> that means one component in GlassFish is doing it. Now to find the
>>>> component that does it is far from simple. Last time I've
>>>> instrumented the JDK classes to find which component was doing
>>>> that. Just file a bug if you think your application is not storing
>>>> the classloader.
>>>>
>>>> Thanks
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>> Ashish Sahni wrote:
>>>>
>>>>> Hi All,
>>>>> I've been seeing this error(see attached mail) consistently for
>>>>> the last few days.
>>>>> It can easily reproduced on re-deployment. If interested I can
>>>>> send you the
>>>>> webapp used - its quite large. I have not checked if there's a bug
>>>>> filed
>>>>> on this issue already. Let me know if you have any thoughts.
>>>>>
>>>>> Thanks
>>>>> Ashish
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> Subject:
>>>>> deployment error/thread death
>>>>> From:
>>>>> ashish_at_hysteria.SFBay.Sun.COM
>>>>> Date:
>>>>> Tue, 03 Apr 2007 15:11:55 -0700 (PDT)
>>>>> To:
>>>>> Ashish.Sahni_at_sun.com
>>>>>
>>>>> To:
>>>>> Ashish.Sahni_at_sun.com
>>>>>
>>>>>
>>>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>>>> Illegal access: this web application instance has been stopped
>>>>> already (the eventual following stack trace is caused by an error
>>>>> thrown for debugging purposes as well as to attempt to terminate
>>>>> the thread which caused the illegal access, and has no functional
>>>>> impact)|#]
>>>>>
>>>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>>>> Illegal access: this web application instance has been stopped
>>>>> already (the eventual following stack trace is caused by an error
>>>>> thrown for debugging purposes as well as to attempt to terminate
>>>>> the thread which caused the illegal access, and has no functional
>>>>> impact)|#]
>>>>>
>>>>> [#|2007-04-03T15:08:22.628-0700|WARNING|sun-appserver9.1|javax.enterprise.system.core|_ThreadID=17;_ThreadName=Thread-29;_RequestID=b4d48310-63bf-4ab9-95ac-99d72a398625;|CORE5064:
>>>>> Error serializing descriptor for application cookbook-test-1.2.2;
>>>>> continuing
>>>>> java.lang.ThreadDeath
>>>>> at
>>>>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
>>>>>
>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>>> at sun.misc.Unsafe.defineClass(Native Method)
>>>>> at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
>>>>> at
>>>>> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at
>>>>> sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
>>>>>
>>>>> at
>>>>> sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>>
>>>>> at
>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>>>>> at java.lang.Class.newInstance0(Class.java:350)
>>>>> at java.lang.Class.newInstance(Class.java:303)
>>>>> at java.security.Provider$Service.newInstance(Provider.java:1130)
>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
>>>>> at java.security.Security.getImpl(Security.java:658)
>>>>> at
>>>>> java.security.MessageDigest.getInstance(MessageDigest.java:122)
>>>>> at
>>>>> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
>>>>>
>>>>> at
>>>>> java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
>>>>> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:192)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at
>>>>> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:189)
>>>>>
>>>>> at
>>>>> java.io.ObjectStreamClass.writeNonProxy(ObjectStreamClass.java:651)
>>>>> at
>>>>> java.io.ObjectOutputStream.writeClassDescriptor(ObjectOutputStream.java:615)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1169)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1284)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>>> at
>>>>> com.sun.enterprise.deployment.util.ModuleDescriptor.writeObject(ModuleDescriptor.java:242)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>>> at java.util.HashSet.writeObject(HashSet.java:254)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>>> at
>>>>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>>
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>>> at
>>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.EJBCompiler.compile(EJBCompiler.java:217)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.ModuleDeployer.runEJBC(ModuleDeployer.java:801)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.WebModuleDeployer.deploy(WebModuleDeployer.java:168)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:166)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:174)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:95)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java:892)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:266)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:761)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
>>>>>
>>>>> at
>>>>> com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
>>>>>
>>>>> |#]
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>> ------------------------------------------------------------------------
>>
>> at java.lang.Thread.dumpStack(Thread.java:1158)
>> at java.security.Security.insertProviderAt(Security.java:329)
>> at org.jruby.RubyDigest.createDigest(RubyDigest.java:50)
>> at org.jruby.libraries.DigestLibrary.load(DigestLibrary.java:40)
>> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
>> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
>> at org.jruby.RubyDigest.createDigestSHA2(RubyDigest.java:104)
>> at org.jruby.libraries.DigestLibrary$SHA2.load(DigestLibrary.java:63)
>> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
>> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
>> at org.jruby.RubyKernel.require(RubyKernel.java:670)
>> at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
>> at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
>> at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
>> at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
>> at org.jruby.RubyObject.callMethod(RubyObject.java:487)
>> at org.jruby.RubyObject.callMethod(RubyObject.java:395)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
>> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
>> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
>> at org.jruby.RubyObject.eval(RubyObject.java:563)
>> at org.jruby.Ruby.loadScript(Ruby.java:1179)
>> at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
>> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
>> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
>> at org.jruby.RubyKernel.require(RubyKernel.java:670)
>> at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
>> at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
>> at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
>> at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
>> at org.jruby.RubyObject.callMethod(RubyObject.java:487)
>> at org.jruby.RubyObject.callMethod(RubyObject.java:395)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
>> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
>> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
>> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
>> at org.jruby.RubyObject.eval(RubyObject.java:563)
>> at org.jruby.Ruby.loadScript(Ruby.java:1179)
>> at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
>> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
>> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
>> at org.jruby.webapp.RailsFactory.makeObject(RailsFactory.java:109)
>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
>> at org.jruby.webapp.AbstractRailsServlet.serviceRequest(AbstractRailsServlet.java:149)
>> at org.jruby.webapp.AbstractRailsServlet.service(AbstractRailsServlet.java:136)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:818)
>> at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:398)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
>> at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
>> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
>> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
>> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
>> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:138)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
>> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
>> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
>> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
>> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
>> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
>> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
>> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
>> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
>> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
>> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
>> at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:345)
>> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
>> at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>