dev@glassfish.java.net

Re: v3 build hanging

From: Jane Young <Jane.Young_at_Sun.COM>
Date: Tue, 14 Jul 2009 12:38:54 -0700

Thank you for verifying. Nandini, Can you commit the fix since it's
also impacting dev build.

Thanks,
Jane


Hong Zhang wrote:
> Hi, Jane
>> Thanks for the stacktrace. As Jerome pointed out in v3 mtg, this
>> maybe related to the change in CreateDomainCommand.java, rev 29043
>>
> I followed Nandini's suggestion to comment out modifyInitialDomainXML
> method on CreateDomainCommand.java (line number 496) and I was able to
> build afterwards.
>
> Thanks,
>
> - Hong
>> Can you checkout the source from rev 29041: "svn checkout -r 29041
>> https://svn.dev.java.net/svn/glassfish-svn/trunk/v3" and do a build?
>> Please let me know if the build succeeds.
>>
>> Thanks,
>> Jane
>>
>>
>> Hong Zhang wrote:
>>> The stack trace I attached earlier was for creating-doman, and I
>>> have now attached the stack trace for starting the domain which is
>>> probably what we need to look at.
>>>
>>> In the meantime, any workaround for me to go further?
>>>
>>>
>>> Hong Zhang wrote:
>>>>
>>>>>> Since I'm not able to reproduce this hang. Is it possible if you
>>>>>> can execute "jstack -l pid" to produce the stack dump?
>>>>>>
>>>>> I have just cleaned my local repository to see if I can get a
>>>>> successful build as I want to move forward to do other work. If I
>>>>> get this error again, I will try jstack -l pid.
>>>>
>>>> Ok, it still hangs for me after I started with a clean repository:
>>>> hzhang_at_nmr:~/.m2$ jstack -l 10965
>>>> 2009-07-14 12:35:30
>>>> Full thread dump Java HotSpot(TM) Server VM (11.0-b16 mixed mode):
>>>>
>>>> "Attach Listener" daemon prio=10 tid=0x081b1800 nid=0x2b6c waiting
>>>> on condition [0x00000000..0x00000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "GlassFish Kernel Main Thread" prio=10 tid=0x7c226c00 nid=0x2b1f in
>>>> Object.wait() [0x7c1ad000..0x7c1ade30]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xafa12e60> (a
>>>> com.sun.enterprise.v3.server.AppServerStartup$1)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at
>>>> com.sun.enterprise.v3.server.AppServerStartup$1.run(AppServerStartup.java:144)
>>>>
>>>> - locked <0xafa12e60> (a
>>>> com.sun.enterprise.v3.server.AppServerStartup$1)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "AutoDeployer" daemon prio=10 tid=0x083a3c00 nid=0x2b1e in
>>>> Object.wait() [0x7c1fe000..0x7c1fedb0]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xaf8501d8> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at java.util.TimerThread.mainLoop(Timer.java:483)
>>>> - locked <0xaf8501d8> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "DynamicReloader" daemon prio=10 tid=0x083a4800 nid=0x2b1d in
>>>> Object.wait() [0x7c451000..0x7c451f30]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xaf741b70> (a java.util.TaskQueue)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at java.util.TimerThread.mainLoop(Timer.java:483)
>>>> - locked <0xaf741b70> (a java.util.TaskQueue)
>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "pool-2-thread-1" daemon prio=10 tid=0x7c21e000 nid=0x2b1b waiting
>>>> on condition [0x7c4a2000..0x7c4a3030]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for <0xaf2293a8> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>>>
>>>> at
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>>>
>>>> at
>>>> org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
>>>>
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>
>>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>> at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Locked ownable synchronizers:
>>>> - <0xaf229490> (a
>>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>>>
>>>> "pool-1-thread-1" daemon prio=10 tid=0x7c216800 nid=0x2b1a waiting
>>>> on condition [0x7c4f3000..0x7c4f3fb0]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for <0xaf1c5968> (a
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>>>
>>>> at
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>>>
>>>> at
>>>> org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:142)
>>>>
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>>>
>>>> at
>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>> at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Locked ownable synchronizers:
>>>> - <0xaf1ca4c8> (a
>>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>>>
>>>> "Low Memory Detector" daemon prio=10 tid=0x080f9400 nid=0x2ae3
>>>> runnable [0x00000000..0x00000000]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "CompilerThread1" daemon prio=10 tid=0x080f7400 nid=0x2ae2 waiting
>>>> on condition [0x00000000..0x7c984428]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "CompilerThread0" daemon prio=10 tid=0x080f4400 nid=0x2ae1 waiting
>>>> on condition [0x00000000..0x7ca05468]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Signal Dispatcher" daemon prio=10 tid=0x080f2c00 nid=0x2ae0
>>>> runnable [0x00000000..0x7ca56e10]
>>>> java.lang.Thread.State: RUNNABLE
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Finalizer" daemon prio=10 tid=0x080dec00 nid=0x2adf in
>>>> Object.wait() [0x7cae9000..0x7caea0b0]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xb021d838> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>>> - locked <0xb021d838> (a java.lang.ref.ReferenceQueue$Lock)
>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>>> at
>>>> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "Reference Handler" daemon prio=10 tid=0x080dd400 nid=0x2ade in
>>>> Object.wait() [0x7cb3a000..0x7cb3ae30]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xb021d900> (a java.lang.ref.Reference$Lock)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at
>>>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>>> - locked <0xb021d900> (a java.lang.ref.Reference$Lock)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "main" prio=10 tid=0x08058800 nid=0x2ada in Object.wait()
>>>> [0xb7d71000..0xb7d721f8]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> - waiting on <0xafa12e60> (a
>>>> com.sun.enterprise.v3.server.AppServerStartup$1)
>>>> at java.lang.Thread.join(Thread.java:1143)
>>>> - locked <0xafa12e60> (a
>>>> com.sun.enterprise.v3.server.AppServerStartup$1)
>>>> at
>>>> com.sun.enterprise.v3.server.AppServerStartup.stop(AppServerStartup.java:348)
>>>>
>>>> at org.glassfish.api.embedded.Server.stop(Server.java:359)
>>>> at
>>>> com.sun.enterprise.admin.cli.optional.CreateDomainCommand.modifyInitialDomainXml(CreateDomainCommand.java:773)
>>>>
>>>> at
>>>> com.sun.enterprise.admin.cli.optional.CreateDomainCommand.createTheDomain(CreateDomainCommand.java:496)
>>>>
>>>> at
>>>> com.sun.enterprise.admin.cli.optional.CreateDomainCommand.runCommand(CreateDomainCommand.java:267)
>>>>
>>>> at
>>>> com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:171)
>>>>
>>>> at
>>>> com.sun.enterprise.cli.framework.CLIMain.invokeCommand(CLIMain.java:105)
>>>>
>>>> at
>>>> com.sun.enterprise.admin.cli.AsadminMain.local(AsadminMain.java:190)
>>>> at
>>>> com.sun.enterprise.admin.cli.AsadminMain.main(AsadminMain.java:84)
>>>>
>>>> Locked ownable synchronizers:
>>>> - None
>>>>
>>>> "VM Thread" prio=10 tid=0x080d9c00 nid=0x2add runnable
>>>>
>>>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x08060000 nid=0x2adb
>>>> runnable
>>>>
>>>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x08061400 nid=0x2adc
>>>> runnable
>>>>
>>>> "VM Periodic Task Thread" prio=10 tid=0x080fb000 nid=0x2ae4 waiting
>>>> on condition
>>>>
>>>> JNI global references: 842
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - Hong
>>>>>
>>>>>>
>>>>>>
>>>>>> Hong Zhang wrote:
>>>>>>
>>>>>>>
>>>>>>>> I just built successfully on the Mac using an existing Maven repo.
>>>>>>>> Looks like this hang is happening sporadically.
>>>>>>>> BTW: are you using a Linux? I recall seeing the same hang in
>>>>>>>> create-domain on the Hudson dev job yesterday. Hudson dev job
>>>>>>>> is also running on Linux.
>>>>>>>>
>>>>>>> Yes, I am on ubuntu.
>>>>>>>
>>>>>>>>
>>>>>>>> Hong Zhang wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> in which module is this ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Most times in core/kernel. There was one time the build was
>>>>>>>>>>> able to go further, and then when it's building the nucleus
>>>>>>>>>>> distribution, it hang with a similar error.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ok let me know if it hangs somewhere else. I need to spawn a
>>>>>>>>>> process to test the embedded as running several embedded
>>>>>>>>>> instances in the same process can have side effects.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jerome: I have updated workspace and got your changes. Now it
>>>>>>>>> hang in building nucleus distribution:
>>>>>>>>>
>>>>>>>>> create.distribution:
>>>>>>>>> [echo] Creating distribution: nucleus
>>>>>>>>> [unzip] Expanding:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/felix-3.0-SNAPSHOT.zip
>>>>>>>>> into
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage
>>>>>>>>>
>>>>>>>>> [unzip] Expanding:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfish-grizzly-3.0-SNAPSHOT.zip
>>>>>>>>> into
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage
>>>>>>>>>
>>>>>>>>> [unzip] Expanding:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfish-hk2-3.0-SNAPSHOT.zip
>>>>>>>>> into
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage
>>>>>>>>>
>>>>>>>>> [unzip] Expanding:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfish-nucleus-3.0-SNAPSHOT.zip
>>>>>>>>> into
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage
>>>>>>>>>
>>>>>>>>> [mkdir] Created dir:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/target
>>>>>>>>> [delete] DEPRECATED - Use of the implicit FileSet is
>>>>>>>>> deprecated. Use a nested fileset element instead.
>>>>>>>>> [delete] Deleting 4 files from
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> fetch-domain-creation-artifacts:
>>>>>>>>> [echo] cli-optional jar:
>>>>>>>>> /home/hzhang/.m2/repository/org/glassfish/admin/cli-optional/3.0-SNAPSHOT/cli-optional-3.0-SNAPSHOT.jar
>>>>>>>>>
>>>>>>>>> [echo] server-mgmt jar:
>>>>>>>>> /home/hzhang/.m2/repository/org/glassfish/admin/server-mgmt/3.0-SNAPSHOT/server-mgmt-3.0-SNAPSHOT.jar
>>>>>>>>>
>>>>>>>>> [echo] admin-util jar:
>>>>>>>>> /home/hzhang/.m2/repository/org/glassfish/admin/admin-util/3.0-SNAPSHOT/admin-util-3.0-SNAPSHOT.jar
>>>>>>>>>
>>>>>>>>> [echo] security jar:
>>>>>>>>> /home/hzhang/.m2/repository/org/glassfish/security/security/3.0-SNAPSHOT/security-3.0-SNAPSHOT.jar
>>>>>>>>>
>>>>>>>>> [copy] Copying 1 file to
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/modules
>>>>>>>>>
>>>>>>>>> [copy] Copying 1 file to
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/modules
>>>>>>>>>
>>>>>>>>> [copy] Copying 1 file to
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/modules
>>>>>>>>>
>>>>>>>>> [copy] Copying 1 file to
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/modules
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> init:
>>>>>>>>>
>>>>>>>>> create-domain-on-windows:
>>>>>>>>>
>>>>>>>>> create-domain-on-unix:
>>>>>>>>>
>>>>>>>>> create-domain:
>>>>>>>>> [echo] Directory to run command from:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/bin
>>>>>>>>>
>>>>>>>>> [echo] Command to run:
>>>>>>>>> /home/hzhang/files/workspaces/v3/distributions/nucleus/target/stage/glassfishv3/glassfish/bin/asadmin
>>>>>>>>>
>>>>>>>>> [exec] Using port 4848 for Admin.
>>>>>>>>> [exec] Using port 8080 for HTTP Instance.
>>>>>>>>> [exec] Using default port 7676 for JMS.
>>>>>>>>> [exec] Using default port 3700 for IIOP.
>>>>>>>>> [exec] Using default port 8181 for HTTP_SSL.
>>>>>>>>> [exec] Using default port 3820 for IIOP_SSL.
>>>>>>>>> [exec] Using default port 3920 for IIOP_MUTUALAUTH.
>>>>>>>>> [exec] Using default port 8686 for JMX_ADMIN.
>>>>>>>>> [exec] Distinguished Name of the self-signed X.509 Server
>>>>>>>>> Certificate is:
>>>>>>>>> [exec] [CN=localhost,OU=GlassFish,O=Sun
>>>>>>>>> Microsystems,L=Santa Clara,ST=California,C=US]
>>>>>>>>> [exec] Jul 14, 2009 11:51:55 AM
>>>>>>>>> com.sun.enterprise.v3.server.AppServerStartup run
>>>>>>>>> [exec] INFO: GlassFish v3 startup time : Embedded(307ms)
>>>>>>>>> startup services(224ms) total(531ms)
>>>>>>>>> [exec] Jul 14, 2009 11:51:55 AM AppServerStartup run
>>>>>>>>> [exec] INFO: [Thread[GlassFish Kernel Main Thread,5,main]]
>>>>>>>>> started
>>>>>>>>> [exec] Received Stop
>>>>>>>>> [exec] Sending stop event to ms
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Hong
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> you updated and did mvn clean install from the root ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes, I updated, and did mvn -U clean install from the root.
>>>>>>>>>>>
>>>>>>>>>>> - Hong
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 14, 2009, at 8:10 AM, Hong Zhang wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> I have updated my workspace this morning and tried to
>>>>>>>>>>>>> build the workspace and the build kept hanging. I have
>>>>>>>>>>>>> restarted a few times and no difference. There is where
>>>>>>>>>>>>> it hangs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------------------------------------------------------
>>>>>>>>>>>>> T E S T S
>>>>>>>>>>>>> -------------------------------------------------------
>>>>>>>>>>>>> Running com.sun.enterprise.v3.admin.AdminAdapterTest
>>>>>>>>>>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
>>>>>>>>>>>>> elapsed: 0.019 sec
>>>>>>>>>>>>> Running com.sun.enterprise.v3.admin.CreateProfilerTest
>>>>>>>>>>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time
>>>>>>>>>>>>> elapsed: 0.001 sec
>>>>>>>>>>>>> Running org.glassfish.tests.kernel.embedded.EmbeddedTest
>>>>>>>>>>>>> Jul 14, 2009 10:55:43 AM
>>>>>>>>>>>>> com.sun.enterprise.v3.server.AppServerStartup run
>>>>>>>>>>>>> INFO: GlassFish v3 startup time : Embedded(219ms)
>>>>>>>>>>>>> startup services(170ms) total(389ms)
>>>>>>>>>>>>> Jul 14, 2009 10:55:43 AM AppServerStartup run
>>>>>>>>>>>>> INFO: [Thread[GlassFish Kernel Main Thread,5,main]] started
>>>>>>>>>>>>> Jul 14, 2009 10:55:43 AM
>>>>>>>>>>>>> com.sun.enterprise.v3.admin.CommandRunnerImpl getCommand
>>>>>>>>>>>>> INFO: Command create-http-listener not found
>>>>>>>>>>>>> Received Stop
>>>>>>>>>>>>> Sending stop event to ms
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone seeing similar thing? I have svn revision 29132.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Hong
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> dev-help_at_glassfish.dev.java.net
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> dev-help_at_glassfish.dev.java.net
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> dev-help_at_glassfish.dev.java.net
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>