users@glassfish.java.net

RE: Re: Interrupted idle thread

From: Martin, Ray <armart3_at_tycho.ncsc.mil>
Date: Wed, 22 Sep 2010 16:55:00 -0400

Issue number: 13499

https://glassfish.dev.java.net/servlets/ReadMsg?listName=issues&msgNo=74
925

don't know that I gave it the greatest title - but, it was the best that
I could do...

I am interested if others have the same problem.

-----Original Message-----
From: Ryan Lubke [mailto:ryan.lubke_at_oracle.com]
Sent: Wednesday, September 22, 2010 1:08 PM
To: users_at_glassfish.dev.java.net
Subject: Re: Interrupted idle thread

  On 9/22/10 9:25 AM, Martin, Ray wrote:
> After opening an issue, what should be expected next?
Could you please share the issue number here?

As far as what's next, one of the Grizzly developers will take a look.
> -----Original Message-----
> From: Ryan Lubke [mailto:ryan.lubke_at_oracle.com]
> Sent: Thursday, September 16, 2010 11:52 AM
> To: users_at_glassfish.dev.java.net
> Subject: Re: Interrupted idle thread
>
> On 9/16/10 4:07 AM, Martin, Ray wrote:
>> I have no listener changes. I did a search for " http
>> request-timeout-seconds" to do as you had suggested earlier - but,
did
>> not find it in 3.0.1 domain.xml.
>>
>> If the test case is not repeatable by anyone else, then filing an
> issue
>> would be pointless - though I would be glad to do so if there is
> value.
>> Where does one go to open an issue?
> I believe there to be value in logging an issue with the test case
> (wouldn't have
> asked otherwise :) ).
>
> You can log an issue here:
> https://glassfish.dev.java.net/servlets/ProjectIssues
>> Thanx.
>>
>> -----Original Message-----
>> From: Ryan Lubke [mailto:ryan.lubke_at_oracle.com]
>> Sent: Wednesday, September 15, 2010 5:50 PM
>> To: users_at_glassfish.dev.java.net
>> Subject: Re: Interrupted idle thread
>>
>> On 9/15/10 1:16 PM, Martin, Ray wrote:
>>> The test is very repeatable.
>>>
>>> I have two cases running right now - one on unpatched 3.0.1 and the
>>> other on patched 3.0.1.
>>>
>>> Both cases show that a web service that runs over 15 minutes,
>> glassfish
>>> will wizz all over the parade.
>>>
>>> Both cases, right on 15 minutes every time, rip out and do very bad
>>> things. Who ever heard of starting a second process because
> something
>>> went beyond 15 minutes. If one had a very sensitive process going -
>>> like calculating and writing to a database - oop, yur database is
all
>>> screwed up - how nice.
>>>
>>> Patched 3.0.1 blows out a WebServiceException and starts a second
>>> instance of the operation. This is the worst that could possibly
>> occur.
>>> Unpatched 3.0.1 only warns of "interrupted idle thread:...", starts
>>> second instance of the operation, but quits the second instance
>>> immediately. On an asynchronous operation, the client believes the
>>> process has completed - this is certainly not a good thing, but at
>> least
>>> it did not corrupt my database.
>> Hi Martin,
>>
>> At this point, I'd like to request you open an issue and attach your
>> test case.
>> Please include any relevant (or what you may think is relevant)
> listener
>> configuration
>> changes you have made in the report.
>>
>> Thanks,
>> -rl
>>> -----Original Message-----
>>> From: Martin, Ray
>>> Sent: Wednesday, September 15, 2010 2:54 PM
>>> To: users_at_glassfish.dev.java.net
>>> Subject: RE: RE: Re: Interrupted idle thread
>>>
>>> I ran the same test on a unpatched version of 3.0.1.
>>> It also ran very badly - but, very differently. It kicked the
> process
>>> the second time, but immediately quits on the second process.
>>> Also, it appears that the connection is still available between the
>>> client and server - unlike previous version. Before, when getting a
>>> ClientAbortException, the connection was no longer useful.
>>>
>>> I think I should go back to an unpatched version of 3.0.1 at a
>> minimum.
>>> -----Original Message-----
>>> From: Martin, Ray
>>> Sent: Wednesday, September 15, 2010 12:58 PM
>>> To: users_at_glassfish.dev.java.net
>>> Subject: RE: Re: Interrupted idle thread
>>>
>>> Now using glassfish 3.0.1
>>>
>>> Problem remains.
>>>
>>> Placed patch jars into glassfish directory.
>>>
>>> Still get the "interrupting idle thread...".
>>>
>>> Was always using a synchronous client up to this point.
>>>
>>> Created an asynchronous client - problem remains.
>>>
>>> I needed a more exact replication of the problem.
>>>
>>> I used Netbeans 6.9.1.
>>>
>>> Created a web service using session bean - the operations are setup
>> and
>>> activate.
>>>
>>> In the session bean, activate method, I put the following"
>>>
>>> System.out.println("Hello");
>>> int min = 30;
>>> long now = System.currentTimeMillis();
>>> long wait = min*60*1000;
>>> long go = now+wait;
>>> min--;
>>> while(now<go) {
>>> try {
>>> Thread.sleep(1000);
>>> } catch ...
>>> }
>>> now = System.currentTimeMillis();
>>> if(cnt>59) {
>>> System.out.println(min--);
>>> cnt = 0;
>>> }
>>> cnt++;
>>> }
>>>
>>> The sys out is written to the glassfish log, of course. When the
>>> Activate SOAP message arrives, Hello appears, one minute later a 29
>>> appears. The sys out keeps counting down until a 16 appears.
>>>
>>> Then, WARNING: Interrupting idle Thread: http-thread-pool-8080-(2),
>>> appears.
>>>
>>> Then Hello appears again, then a 15, then a 29, 14, 28
>>>
>>> Finally, the first minute count goes to 1 (the second count was 16)
>>> Then the log shows:
>>> WARNING: invocation error on ejb endpoint ...
>>>
>>> The second count continues to wind down to 1, then the error:
>>> WARNING: invocation error on ejb endpoint ...
>>>
>>>
>>> Note: I cannot find the " http request-timeout-seconds" in
domain.xml
>>>
>>> Note: My brother says there is a mechanism to tell the client not to
>>> timeout - he could not remember what it was. Anyone?
>>>
>>> Please help!
>>>
>>> Thanx,
>>> Ray
>>>
>>>
>>> -----Original Message-----
>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>> Sent: Thursday, September 02, 2010 12:54 PM
>>> To: users_at_glassfish.dev.java.net
>>> Subject: Re: Interrupted idle thread
>>>
>>> Martin,
>>>
>>> here is one more issue been fixed after 3.0.1 release, might be
>>> related [1].
>>> At the bottom you may find the proposed patch.
>>>
>>> WBR,
>>> Alexey.
>>>
>>> [1] https://grizzly.dev.java.net/issues/show_bug.cgi?id=862
>>>
>>> On Sep 2, 2010, at 14:06 , Martin, Ray wrote:
>>>
>>>> Grizzly Framework 1.9.18-o
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>> Sent: Wednesday, September 01, 2010 5:08 PM
>>>> To: users_at_glassfish.dev.java.net
>>>> Subject: Re: Interrupted idle thread
>>>>
>>>> Can you pls. check the server.log file, there should be the line
> like
>>>> "Starting Grizzly Framework 1.9......."
>>>>
>>>> which exactly version you see there?
>>>>
>>>> Thanks.
>>>> Alexey.
>>>>
>>>> On Sep 1, 2010, at 22:53 , Martin, Ray wrote:
>>>>
>>>>> Glassfish V3 also does another querky thing.
>>>>>
>>>>> When the process finishes running - in v2 it would send back the
>> SOAP
>>>>> response.
>>>>> In v3, I get "WARNING: invocation error on ejb endpoint:
>>>>> ConductorWebService at ConductorWebServiceSerice/
>>>>> ConductorWebService :
>>>>> com.ctc.wstx.exc.WstxIOException: null
>>>>>
>>>>> Ya gotta luv v2 - no such hassle :)
>>>>>
>>>>> -----Original Message-----
>>>>> From: Martin, Ray
>>>>> Sent: Wednesday, September 01, 2010 2:59 PM
>>>>> To: users_at_glassfish.dev.java.net
>>>>> Subject: RE: Re: Interrupted idle thread
>>>>>
>>>>> I would be glad to do so - I just have not figured out how just
> yet.
>>>>> -----Original Message-----
>>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>>> Sent: Wednesday, September 01, 2010 2:57 PM
>>>>> To: users_at_glassfish.dev.java.net
>>>>> Subject: Re: Interrupted idle thread
>>>>>
>>>>> I see.
>>>>> If you can provide some simple testcase to reproduce the issue -
>> I'll
>>>>> try to figure out what's going wrong.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>> On Sep 1, 2010, at 20:30 , Martin, Ray wrote:
>>>>>
>>>>>> Yes, the netbeans auto generation code now uses an AbstractFacade
> -
>>>>>> it
>>>>>> is gonna take some new understanding.
>>>>>>
>>>>>> V3.0.1 is still broken with respect to JAX-WS. I didn't measure
>> the
>>>>>> time just yet. But, when a process is activated via a JAX-WS
>>>>>> command,
>>>>>> V3 eventually does the "WARNING: Interrupting idle Thread:
>>>>>> http-thread-pool-8080-(2)" - which does not mean a whole lot to
> me.
>>>>>> BUT, then V3 does a very bad thing - it acts as though it had
>>>>>> received a
>>>>>> second JAX-WS message and my process is activated the second time
> -
>>>>>> now
>>>>>> I have two processes pouring junk into my database.
>>>>>>
>>>>>> I MUST go back to v2 - it worked - I have been running and
testing
>>>>>> these
>>>>>> routines for months. Switching to V3 has crushed me.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>>>> Sent: Wednesday, September 01, 2010 4:47 AM
>>>>>> To: users_at_glassfish.dev.java.net
>>>>>> Subject: Re: Interrupted idle thread
>>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> didn't expect NB 6.9.1 makes something different than 6.8. Hope
>>>>>> you'll
>>>>>> finally make things work with it.
>>>>>>
>>>>>> Alexey.
>>>>>>
>>>>>> On Aug 31, 2010, at 18:29 , Martin, Ray wrote:
>>>>>>
>>>>>>> The javaee-api-6.0.jar answered the redlines in my prior code.
>>>>>>>
>>>>>>> I retested the code generators in netbeans - wow, life is
>> different
>>>>>>> than
>>>>>>> the previous version - looks like I have lots of new learning to
>>>>>>> do. I
>>>>>>> guess this is all called progress.
>>>>>>>
>>>>>>> Tore the databases down in preparation for rebuilding.
>>>>>>>
>>>>>>> Deployed and ran - got an ORA-02049 - this looks like a problem
> in
>>>>>>> developer space!!! Ball is back in my court.
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Martin, Ray
>>>>>>> Sent: Tuesday, August 31, 2010 12:19 PM
>>>>>>> To: users_at_glassfish.dev.java.net
>>>>>>> Subject: RE: RE: Re: Interrupted idle thread
>>>>>>>
>>>>>>> I figured those wonderful code generators in netbeans would tell
>> me
>>>>>>> what/how to do annotations. Wrong - the code generators did not
>>>>>>> write
>>>>>>> code like they use to. Don't know if it is because of the
missing
>>>>>>> libraries or if they just don't write code no more???
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Martin, Ray
>>>>>>> Sent: Tuesday, August 31, 2010 12:06 PM
>>>>>>> To: users_at_glassfish.dev.java.net
>>>>>>> Subject: RE: Re: Interrupted idle thread
>>>>>>>
>>>>>>> I lucked out - we have netbeans 6.9.1
>>>>>>>
>>>>>>> I have installed.
>>>>>>>
>>>>>>> At admin console, I again setup all my connections pools and
jdbc
>>>>>>> resource. Did not yet do my security realm.
>>>>>>>
>>>>>>> Had to reset the server on my projects in netbeans.
>>>>>>>
>>>>>>> Everything is good so far - then oops - where are the EJB
>>>>>>> libraries?
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>>>>> Sent: Tuesday, August 31, 2010 10:46 AM
>>>>>>> To: users_at_glassfish.dev.java.net
>>>>>>> Subject: Re: Interrupted idle thread
>>>>>>>
>>>>>>> It should work, but in general 3.0.1 is more stable, that's why
>> IMO
>>>>>>> it
>>>>>>> could make sense to try it.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> Alexey.
>>>>>>>
>>>>>>> On Aug 31, 2010, at 16:21 , Martin, Ray wrote:
>>>>>>>
>>>>>>>> Okay - I will make the switch to a newer version ASAP.
>>>>>>>>
>>>>>>>> BUT - to get me back up and running, could I not set the other
>>>>>>>> variable???
>>>>>>>>
>>>>>>>>
>>>>>>>> The link that you provided says:
>>>>>>>> Additional comments
>>>>>>>> "We misinterpret thread-pool configuration elements in
>> domain.xml,
>>>>>>>> we
>>>>>>>> use thread-pool->idle-thread-
>>>>>>>> timeout-seconds to set transaction timeout and http->timeout-
>>>>>>>> seconds
>>>>>>>> to
>>>>>>>> set thread's idle-timeout."
>>>>>>>>
>>>>>>>>
>>>>>>>> So, if I knew the syntax, I could set
>>>>>>>> thread-pool-idle-thread-timeout-seconds
>>>>>>>> Just for enough to get me going - I am in a really bad position
>> at
>>>>>>>> the
>>>>>>>> moment.
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>>>>>> Sent: Tuesday, August 31, 2010 9:58 AM
>>>>>>>> To: users_at_glassfish.dev.java.net
>>>>>>>> Subject: Re: Interrupted idle thread
>>>>>>>>
>>>>>>>> Now I see, most probably you're using GF v3, where we had issue
>>>>>>>> [1].
>>>>>>>> Can you pls. switch to GF v3.0.1?
>>>>>>>>
>>>>>>>> I don't remember exactly if NB 6.8 supports GF v3 server out of
>>>>>>>> the
>>>>>>>> box, or you need additional plugin to be installed, but with NB
>>>>>>>> 6.9.1
>>>>>>>> you can use GF 3.0.1 (AFAIR it's one you have out of the box),
>>>>>>>> which
>>>>>>>> should have that issue resolved.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> WBR,
>>>>>>>> Alexey.
>>>>>>>>
>>>>>>>> [1] https://glassfish.dev.java.net/issues/show_bug.cgi?id=11490
>>>>>>>>
>>>>>>>>
>>>>>>>> On Aug 31, 2010, at 14:21 , Martin, Ray wrote:
>>>>>>>>
>>>>>>>>> I am using netbeans6.8 and whatever glassfish comes with it.
>>>>>>>>>
>>>>>>>>> At the glassfish admin console, I press the version button, I
>> get
>>>>>>>>> v3
>>>>>>>>> (no
>>>>>>>>> numbers)
>>>>>>>>>
>>>>>>>>> I am very sorry to say, that I cannot get the descriptor onto
>> the
>>>>>>>>> Internet.
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
>>>>>>>>> Sent: Tuesday, August 31, 2010 7:35 AM
>>>>>>>>> To: users_at_glassfish.dev.java.net
>>>>>>>>> Subject: Re: Interrupted idle thread
>>>>>>>>>
>>>>>>>>> Martin,
>>>>>>>>>
>>>>>>>>> just to make sure, are you using the Glassfish 3.0.1 release?
>>>>>>>>> Can you pls. share the domain.xml descriptor?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Alexey.
>>>>>>>>>
>>>>>>>>> On Aug 31, 2010, at 13:28 , Martin, Ray wrote:
>>>>>>>>>
>>>>>>>>>> Mr. Lubke,
>>>>>>>>>>
>>>>>>>>>> I did the following:
>>>>>>>>>> ...
>>>>>>>>>> <http request-timeout-seconds="-1" ...
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> I assume that -1 means to disable.
>>>>>>>>>>
>>>>>>>>>> After restarting glassfish and deploying the EJBModule in an
>>>>>>>>>> Enterprise
>>>>>>>>>> app (so it will see the third party jars) - still got the
>>>>>>>>>> There are no highlighted small blocks.>>>>>>>
>>>>>>>>>> But, worse than that is the fact that something causes the
> SOAP
>>>>>>>>>> message
>>>>>>>>>> to be repeated - then my process is not protected against
that
>>>>>>>>>> repeat
>>>>>>>>>> SOAP message again kicks off and stomps all over the database
>>>>>>>>>> records
>>>>>>>>>> that it just wrote before the timeout.
>>>>>>>>>>
>>>>>>>>>> I am going to try an asynchronous client to see if that will
>>>>>>>>>> take
>>>>>>>>>> care
>>>>>>>>>> of the problem.
>>>>>>>>>>
>>>>>>>>>> Thanx for your help.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Ryan Lubke [mailto:ryan.lubke_at_oracle.com]
>>>>>>>>>> Sent: Monday, August 30, 2010 8:51 PM
>>>>>>>>>> To: Martin, Ray
>>>>>>>>>> Cc: users_at_glassfish.dev.java.net
>>>>>>>>>> Subject: RE: Interrupted idle thread
>>>>>>>>>>
>>>>>>>>>> Hello Martin,
>>>>>>>>>>
>>>>>>>>>> I believe you'll be able to resolve this issue by either
>>>>>>>>>> increasing
>>>>>>>>>> the
>>>>>>>>>> request-timeout-seconds property or disabling
>>>>>>>>>> the feature altogether.
>>>>>>>>>>
>>>>>>>>>> This can be done by backing up your
>>>>>>>>>> GF_HOME/domains/domain1/config/domain.xml and then modify the
>>>>>>>>>> http
>>>>>>>>>> element
>>>>>>>>>> associated with the protocol your listener is using.
>>>>>>>>>>
>>>>>>>>>> For example, if you're using the out-of-the-box
configuration,
>>>>>>>>>> you
>>>>>>>>>> can
>>>>>>>>>> change the timeout for requests made to port 8080 like
>>>>>>>>>> so:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <protocol name="http-listener-1">
>>>>>>>>>> <http request-timeout-seconds="60" default-virtual-
>>>>>>>>>> server="server"
>>>>>>>>>> max-connections="250" />
>>>>>>>>>> </protocol>
>>>>>>>>>>
>>>>>>>>>> It doesn't appear that this value can currently be set to -1
>> via
>>>>>>>>>> asadmin
>>>>>>>>>>
>>>>>>>>>> command line, so I'll be looking into resolving
>>>>>>>>>> that shortly.
>>>>>>>>>>
>>>>>>>>>> Could you please try updating the domain.xml using the above
>> xml
>>>>>>>>>> snippet
>>>>>>>>>>
>>>>>>>>>> as a guide and let us know if it resolves your problem?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> -rl
>>>>>>>>>>
>>>>>>>>>>
> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail:
>> users-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>>> help_at_glassfish.dev.java.net
>>>>>>>>>>
>>>
---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail:
> users-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>> help_at_glassfish.dev.java.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>
---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail:
> users-unsubscribe_at_glassfish.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>> help_at_glassfish.dev.java.net
>>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail:
users-unsubscribe_at_glassfish.dev.java.net
>>>>>>>> For additional commands, e-mail:
>> users-help_at_glassfish.dev.java.net
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail:
users-unsubscribe_at_glassfish.dev.java.net
>>>>>>>> For additional commands, e-mail:
>> users-help_at_glassfish.dev.java.net
> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail:
> users-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>>
> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail:
> users-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>>
> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail:
> users-help_at_glassfish.dev.java.net
>>>>>>>
>>>>>>>
> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>>> For additional commands, e-mail:
> users-help_at_glassfish.dev.java.net
>>>
---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail:
users-help_at_glassfish.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>
---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail:
users-help_at_glassfish.dev.java.net
>>>>>>
>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>>
>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>>
>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>>
> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>
>>>>
>>>>
> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>
>>>
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>
>>>
>>>
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>
>>>
>>>
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>
>>>
>>>
---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
For additional commands, e-mail: users-help_at_glassfish.dev.java.net