Hi, Jan.
Really great detective work on this. I probably should have followed
this more closely given that the serialization logic I added was involved.
In your research into this, did you see any place that the serialization
code could have protected better against something like this?
Because the serialization (and deserialization) of descriptors was a
performance-only enhancement, I wrote the SerializedDescriptorHelper
class to catch all Throwables that emerge from the attempt to
serialize. (Although without the serialized descriptor the next reload
of the app would take longer, an exception during serialization would
cause no functional problem.) The JavaDoc for ThreadDeath says that
this type of error should be rethrown - I guess to actually make sure
the thread does stop. But as written the logic does not do that - it
logs the warning with the stack trace and continues.
In this case do you think it would be better if the code caught
Exception rather than Throwable, which would let ThreadDeath and other
descendants of Error (as opposed to Exception) bypass the handling and
"float" to the top?
Thanks.
- Tim
Jan Luehe wrote:
> Ashish,
>
> Ashish Sahni wrote On 04/03/07 04:45 PM,:
>
>> Hi Jeanfrancois,
>>
>> First to clarify since you mention 'undeployed' - maybe you're
>> implying implicit undeployment
>> that might be happening during re-deployment but in any case this is
>> what I do:
>> 1. Deploy app
>> 2. Request app services
>> 3. Deploy (same) app (again) ie no explicit undeployment
>>
>> Secondly, I'm not sure if application is 'holding the classloader'
>> since the application uses
>> the jruby runtime libraries. I'm cc'ing Charles and Thomas, the jruby
>> developers how might
>> be able to shed light on your question.
>
>
> I've investigated this issue further, and it indeed turns out to be an
> issue with jruby.
>
> The java.lang.ThreadDeath is thrown during redeployment of
> depot_l2.war, when the deployment code attempts to serialize the
> webapp's AppDescriptor, and in doing so, needs to acquire a MessageDigest
> instance for computing a serialization UID:
>
> java.lang.ThreadDeath
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
>
> ...
> at java.lang.Class.newInstance(Class.java:303)
> at java.security.Provider$Service.newInstance(Provider.java:1130)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
> at java.security.Security.getImpl(Security.java:658)
> at java.security.MessageDigest.getInstance(MessageDigest.java:122)
> at
> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
> ...
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
> at
> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>
> at
> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>
> at
> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>
> at
> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>
> ...
> at
> com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
>
> at
> com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
>
>
>
> It turns out that jruby registers its own provider for cryptographic
> services (named "BC"), but never unregisters it during undeployment,
> causing the subsequent redeployment code to attempt to grab a
> MessageDigest instance whose classloader has been deactivated, which
> in turn has the deactivated classloader (an instance of
> WebappClassLoader) throw the ThreadDeath with the above stacktrace.
>
> I've instrumented java.security.Security.addProvider() and
> removeProvider() to print the caller's stacktrace. I see
> Security.addProvider() getting invoked when accessing
> http://<host>:<port>/depot_l2/store/, see attached stack trace, but I
> don't see any invocation of Security.removeProvider during
> undeployment.
>
> jruby needs to unregister its provider for cryptographic services
> during undeployment, by calling
> java.security.Security.removeProvider("BC").
>
> This could be done inside
> org.jruby.webapp.AbstractRailsServlet.destroy().
>
>
> Jan
>
>>
>> Thanks
>> Ashish
>>
>>
>> Jeanfrancois Arcand wrote:
>>
>>> Hi Ashish,
>>>
>>> there is two possibilities:
>>>
>>> (1) There is a thread pool or object in GlassFish that store the
>>> ContextClassloader and try to re-use it. The ContextClassLoader of a
>>> web application is always a WebappClassloader, and that classloader
>>> is marked dead when the application is undeployed. In 9.0, we fixed
>>> a bug in the ORB thread pool related to that problem. This time I
>>> suspect the component that create it is:
>>>
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>>
>>>
>>>
>>> (2) Your application is holding the classloader. Are you loading
>>> classes using the ContextClassloader and try to re-use that
>>> Classloader?
>>>
>>> If your application doesn't use or store the ContextClassLoader,
>>> that means one component in GlassFish is doing it. Now to find the
>>> component that does it is far from simple. Last time I've
>>> instrumented the JDK classes to find which component was doing that.
>>> Just file a bug if you think your application is not storing the
>>> classloader.
>>>
>>> Thanks
>>>
>>> -- Jeanfrancois
>>>
>>>
>>> Ashish Sahni wrote:
>>>
>>>> Hi All,
>>>> I've been seeing this error(see attached mail) consistently for the
>>>> last few days.
>>>> It can easily reproduced on re-deployment. If interested I can send
>>>> you the
>>>> webapp used - its quite large. I have not checked if there's a bug
>>>> filed
>>>> on this issue already. Let me know if you have any thoughts.
>>>>
>>>> Thanks
>>>> Ashish
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> Subject:
>>>> deployment error/thread death
>>>> From:
>>>> ashish_at_hysteria.SFBay.Sun.COM
>>>> Date:
>>>> Tue, 03 Apr 2007 15:11:55 -0700 (PDT)
>>>> To:
>>>> Ashish.Sahni_at_sun.com
>>>>
>>>> To:
>>>> Ashish.Sahni_at_sun.com
>>>>
>>>>
>>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>>> Illegal access: this web application instance has been stopped
>>>> already (the eventual following stack trace is caused by an error
>>>> thrown for debugging purposes as well as to attempt to terminate
>>>> the thread which caused the illegal access, and has no functional
>>>> impact)|#]
>>>>
>>>> [#|2007-04-03T15:08:22.618-0700|INFO|sun-appserver9.1|org.apache.catalina.loader.WebappClassLoader|_ThreadID=17;_ThreadName=Thread-29;|PWC1635:
>>>> Illegal access: this web application instance has been stopped
>>>> already (the eventual following stack trace is caused by an error
>>>> thrown for debugging purposes as well as to attempt to terminate
>>>> the thread which caused the illegal access, and has no functional
>>>> impact)|#]
>>>>
>>>> [#|2007-04-03T15:08:22.628-0700|WARNING|sun-appserver9.1|javax.enterprise.system.core|_ThreadID=17;_ThreadName=Thread-29;_RequestID=b4d48310-63bf-4ab9-95ac-99d72a398625;|CORE5064:
>>>> Error serializing descriptor for application cookbook-test-1.2.2;
>>>> continuing
>>>> java.lang.ThreadDeath
>>>> at
>>>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1325)
>>>>
>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>>>> at sun.misc.Unsafe.defineClass(Native Method)
>>>> at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
>>>> at
>>>> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
>>>>
>>>> at
>>>> sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
>>>>
>>>> at
>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
>>>>
>>>> at
>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>>
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
>>>> at java.lang.Class.newInstance0(Class.java:350)
>>>> at java.lang.Class.newInstance(Class.java:303)
>>>> at java.security.Provider$Service.newInstance(Provider.java:1130)
>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:220)
>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:147)
>>>> at java.security.Security.getImpl(Security.java:658)
>>>> at java.security.MessageDigest.getInstance(MessageDigest.java:122)
>>>> at
>>>> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1731)
>>>>
>>>> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
>>>> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:192)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at
>>>> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:189)
>>>>
>>>> at
>>>> java.io.ObjectStreamClass.writeNonProxy(ObjectStreamClass.java:651)
>>>> at
>>>> java.io.ObjectOutputStream.writeClassDescriptor(ObjectOutputStream.java:615)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1169)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1284)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>> at
>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>> at
>>>> com.sun.enterprise.deployment.util.ModuleDescriptor.writeObject(ModuleDescriptor.java:242)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>> at
>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>> at java.util.HashSet.writeObject(HashSet.java:254)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>> at
>>>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
>>>>
>>>> at
>>>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
>>>> at
>>>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.storeSerializedDescriptor(SerializedDescriptorHelper.java:239)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:143)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.SerializedDescriptorHelper.store(SerializedDescriptorHelper.java:123)
>>>>
>>>> at
>>>> com.sun.enterprise.instance.BaseManager.saveAppDescriptor(BaseManager.java:684)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.EJBCompiler.preDeployModule(EJBCompiler.java:352)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.EJBCompiler.compile(EJBCompiler.java:217)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.ModuleDeployer.runEJBC(ModuleDeployer.java:801)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.WebModuleDeployer.deploy(WebModuleDeployer.java:168)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:166)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:174)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:95)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java:892)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:266)
>>>>
>>>> at
>>>> com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:761)
>>>>
>>>> at
>>>> com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:174)
>>>>
>>>> at
>>>> com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:210)
>>>>
>>>> |#]
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
> ------------------------------------------------------------------------
>
> at java.lang.Thread.dumpStack(Thread.java:1158)
> at java.security.Security.insertProviderAt(Security.java:329)
> at org.jruby.RubyDigest.createDigest(RubyDigest.java:50)
> at org.jruby.libraries.DigestLibrary.load(DigestLibrary.java:40)
> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
> at org.jruby.RubyDigest.createDigestSHA2(RubyDigest.java:104)
> at org.jruby.libraries.DigestLibrary$SHA2.load(DigestLibrary.java:63)
> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
> at org.jruby.RubyKernel.require(RubyKernel.java:670)
> at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
> at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
> at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
> at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
> at org.jruby.RubyObject.callMethod(RubyObject.java:487)
> at org.jruby.RubyObject.callMethod(RubyObject.java:395)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
> at org.jruby.RubyObject.eval(RubyObject.java:563)
> at org.jruby.Ruby.loadScript(Ruby.java:1179)
> at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
> at org.jruby.RubyKernel.require(RubyKernel.java:670)
> at org.jruby.RubyKernelInvokerSrequire1.call(Unknown Source)
> at org.jruby.runtime.callback.InvocationCallback.execute(InvocationCallback.java:49)
> at org.jruby.internal.runtime.methods.FullFunctionCallbackMethod.internalCall(FullFunctionCallbackMethod.java:76)
> at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:69)
> at org.jruby.RubyObject.callMethod(RubyObject.java:487)
> at org.jruby.RubyObject.callMethod(RubyObject.java:395)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:770)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:298)
> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
> at org.jruby.evaluator.EvaluationState.evalInternal(EvaluationState.java:1317)
> at org.jruby.evaluator.EvaluationState.eval(EvaluationState.java:163)
> at org.jruby.RubyObject.eval(RubyObject.java:563)
> at org.jruby.Ruby.loadScript(Ruby.java:1179)
> at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:53)
> at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:277)
> at org.jruby.runtime.load.LoadService.require(LoadService.java:299)
> at org.jruby.webapp.RailsFactory.makeObject(RailsFactory.java:109)
> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
> at org.jruby.webapp.AbstractRailsServlet.serviceRequest(AbstractRailsServlet.java:149)
> at org.jruby.webapp.AbstractRailsServlet.service(AbstractRailsServlet.java:136)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:818)
> at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:398)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
> at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:138)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611)
> at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564)
> at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558)
> at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067)
> at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549)
> at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248)
> at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199)
> at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:345)
> at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
> at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:93)
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>