Ashish,
Ashish Sahni wrote On 04/03/07 04:45 PM,:
> Hi Jeanfrancois,
>
> First to clarify since you mention 'undeployed' - maybe you're
> implying implicit undeployment
> that might be happening during re-deployment but in any case this is
> what I do:
> 1. Deploy app
> 2. Request app services
> 3. Deploy (same) app (again) ie no explicit undeployment
>
> Secondly, I'm not sure if application is 'holding the classloader'
> since the application uses
> the jruby runtime libraries. I'm cc'ing Charles and Thomas, the jruby
> developers how might
> be able to shed light on your question.
I've investigated this issue further, and it indeed turns out to be an
issue with jruby.
The java.lang.ThreadDeath is thrown during redeployment of
depot_l2.war, when the deployment code attempts to serialize the
webapp's AppDescriptor, and in doing so, needs to acquire a MessageDigest
instance for computing a serialization UID:
java.lang.ThreadDeath
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
...
at java.lang.Class.newInstance(Class.java:303)
at java.security.Provider$Service.newInstance(Provider.java:1130)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
at java.security.Security.getImpl(Security.java:658)
at java.security.MessageDigest.getInstance(MessageDigest.java:122)
at
java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
...
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at
com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
at
com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
at
com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
at
com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
...
at
com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
at
com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
It turns out that jruby registers its own provider for cryptographic
services (named "BC"), but never unregisters it during undeployment,
causing the subsequent redeployment code to attempt to grab a
MessageDigest instance whose classloader has been deactivated, which
in turn has the deactivated classloader (an instance of
WebappClassLoader) throw the ThreadDeath with the above stacktrace.
I've instrumented java.security.Security.addProvider() and
removeProvider() to print the caller's stacktrace. I see
Security.addProvider() getting invoked when accessing
http://<host>:<port>/depot_l2/store/, see attached stack trace, but I
don't see any invocation of Security.removeProvider during
undeployment.
jruby needs to unregister its provider for cryptographic services
during undeployment, by calling java.security.Security.removeProvider("BC").
This could be done inside org.jruby.webapp.AbstractRailsServlet.destroy().
Jan
>
> Thanks
> Ashish
>
>
> Jeanfrancois Arcand wrote:
>
>> Hi Ashish,
>>
>> there is two possibilities:
>>
>> (1) There is a thread pool or object in GlassFish that store the
>> ContextClassloader and try to re-use it. The ContextClassLoader of a
>> web application is always a WebappClassloader, and that classloader
>> is marked dead when the application is undeployed. In 9.0, we fixed a
>> bug in the ORB thread pool related to that problem. This time I
>> suspect the component that create it is:
>>
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>
>>> at
>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>
>>
>>
>> (2) Your application is holding the classloader. Are you loading
>> classes using the ContextClassloader and try to re-use that Classloader?
>>
>> If your application doesn't use or store the ContextClassLoader, that
>> means one component in GlassFish is doing it. Now to find the
>> component that does it is far from simple. Last time I've
>> instrumented the JDK classes to find which component was doing that.
>> Just file a bug if you think your application is not storing the
>> classloader.
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>>
>> Ashish Sahni wrote:
>>
>>> Hi All,
>>> I've been seeing this error(see attached mail) consistently for the
>>> last few days.
>>> It can easily reproduced on re-deployment. If interested I can send
>>> you the
>>> webapp used - its quite large. I have not checked if there's a bug
>>> filed
>>> on this issue already. Let me know if you have any thoughts.
>>>
>>> Thanks
>>> Ashish
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> Subject:
>>> deployment error/thread death
>>> From:
>>> ashish_at_hysteria.SFBay.Sun.COM
>>> Date:
>>> Tue, 03 Apr 2007 15:11:55 -0700 (PDT)
>>> To:
>>> Ashish.Sahni_at_sun.com
>>>
>>> To:
>>> Ashish.Sahni_at_sun.com
>>>
>>>
>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>> Illegal access: this web application instance has been stopped
>>> already (the eventual following stack trace is caused by an error
>>> thrown for debugging purposes as well as to attempt to terminate the
>>> thread which caused the illegal access, and has no functional
>>> impact)|#]
>>>
>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>> Illegal access: this web application instance has been stopped
>>> already (the eventual following stack trace is caused by an error
>>> thrown for debugging purposes as well as to attempt to terminate the
>>> thread which caused the illegal access, and has no functional
>>> impact)|#]
>>>
>>> [#|2007-04-03T15:08:22.628-0700|WARNING|sun-appserver9.1|javax.enterprise.system.core|_ThreadID=17;_ThreadName=Thread-29;_RequestID=b4d48310-63bf-4ab9-95ac-99d72a398625;|CORE5064:
>>> Error serializing descriptor for application cookbook-test-1.2.2;
>>> continuing
>>> java.lang.ThreadDeath
>>> at
>>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
>>>
>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>> at sun.misc.Unsafe.defineClass(Native Method)
>>> at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
>>> at
>>> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at
>>> sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
>>>
>>> at
>>> sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
>>>
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
>>>
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>>> at java.lang.Class.newInstance0(Class.java:350)
>>> at java.lang.Class.newInstance(Class.java:303)
>>> at java.security.Provider$Service.newInstance(Provider.java:1130)
>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
>>> at java.security.Security.getImpl(Security.java:658)
>>> at java.security.MessageDigest.getInstance(MessageDigest.java:122)
>>> at
>>> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
>>>
>>> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
>>> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:192)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at
>>> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:189)
>>>
>>> at
>>> java.io.ObjectStreamClass.writeNonProxy(ObjectStreamClass.java:651)
>>> at
>>> java.io.ObjectOutputStream.writeClassDescriptor(ObjectOutputStream.java:615)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1169)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>> at
>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>> at
>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1284)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>> at
>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>> at
>>> com.sun.enterprise.deployment.util.ModuleDescriptor.writeObject(ModuleDescriptor.java:242)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>> at
>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>> at
>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>> at java.util.HashSet.writeObject(HashSet.java:254)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>> at
>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>> at
>>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>
>>> at
>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>> at
>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>
>>> at
>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>
>>> at
>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.EJBCompiler.compile(EJBCompiler.java:217)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.ModuleDeployer.runEJBC(ModuleDeployer.java:801)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.WebModuleDeployer.deploy(WebModuleDeployer.java:168)
>>>
>>> at
>>> com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:166)
>>>
>>> at
>>> com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:174)
>>>
>>> at
>>> com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:95)
>>>
>>> at
>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java:892)
>>>
>>> at
>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:266)
>>>
>>> at
>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:761)
>>>
>>> at
>>> com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
>>>
>>> at
>>> com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
>>>
>>> |#]
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
at java.lang.Thread.dumpStack(Thread.java:1158)
at java.security.Security.insertProviderAt(Security.java:329)
at org.jruby.RubyDigest.createDigest(RubyDigest.java:50)
at org.jruby.libraries.DigestLibrary.load(DigestLibrary.java:40)
at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
at org.jruby.RubyDigest.createDigestSHA2(RubyDigest.java:104)
at org.jruby.libraries.DigestLibrary$SHA2.load(DigestLibrary.java:63)
at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
at org.jruby.RubyKernel.require(RubyKernel.java:670)
at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
at org.jruby.RubyObject.callMethod(RubyObject.java:487)
at org.jruby.RubyObject.callMethod(RubyObject.java:395)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
at org.jruby.RubyObject.eval(RubyObject.java:563)
at org.jruby.Ruby.loadScript(Ruby.java:1179)
at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
at org.jruby.RubyKernel.require(RubyKernel.java:670)
at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
at org.jruby.RubyObject.callMethod(RubyObject.java:487)
at org.jruby.RubyObject.callMethod(RubyObject.java:395)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
at org.jruby.RubyObject.eval(RubyObject.java:563)
at org.jruby.Ruby.loadScript(Ruby.java:1179)
at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
at org.jruby.webapp.RailsFactory.makeObject(RailsFactory.java:109)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
at org.jruby.webapp.AbstractRailsServlet.serviceRequest(AbstractRailsServlet.java:149)
at org.jruby.webapp.AbstractRailsServlet.service(AbstractRailsServlet.java:136)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:818)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:398)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:138)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:345)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)