dev@grizzly.java.net

Re: Buildinga Proxy/Load Balacer (was Re: [ANN] Port unification now available in Project Grizzly)

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Tue, 02 Oct 2007 10:48:19 -0400

Hi Alexey,

Oleksiy Stashok wrote:
> Hello guys,
>
> Ramesh, think it makes sense to extend current PortUnificationPipeline
> by some SailfinPortUnificationPipeline and add functionality you need
> there.
> Next question is "how to plug custom port unification pipeline". From
> code I see now, it's not possible in Grizzly 1.0, but making small
> change to GrizzlyHttpProtocol class makes this possible.
>
> Jeanfrancois, I know you're pretty conservative in making changes to
> Grizzly 1.0 codebase, but think making port unification pipeline
> pluggable is quite safe change, which will make Grizzly more extendable.
>
> What do you think?

-1.

Any changes to 1.0 is not allowed because it can break too many existing
product (only bug fixes). The Sailfin team *cannot* make any changes to
the 1.0 code base (this is an internal politic as well).

Thanks

-- Jeanfrancois

>
> Thanks.
>
> WBR,
> Alexey.
>
> Ramesh Parthasarathy wrote:
>> Hi JeanFrancois,
>> I tried to extend the PU mechanism in Grizzly 1.0 ([1]) and
>> implement a
>> way by which we can park the request (release the resources and keep the
>> channel alive). Could you please review the changes, i have attached
>> them.
>> Briefly, i have just propagated the selectorthread through the
>> protocolinfo object so that the callback handler can take the decision
>> of canceling/registering the key back on the selector thread once the
>> response is complete. Also i have refactored the DefaultReadTask so that
>> the resources can be released appropriately.
>>
>>
>> Thanks
>> -Ramesh
>>
>>
>> Jeanfrancois Arcand wrote On 09/25/07 17:45,:
>>
>>> Hi Ramesh,
>>>
>>> thanks for not having private discussions :-)
>>>
>>> Ramesh wrote:
>>>
>>>
>>>> Hi JeanFrancois,
>>>> please find responses inline
>>>> (ccied dev alias)
>>>>
>>>>
>>>> Jeanfrancois Arcand wrote:
>>>>
>>>>
>>>>> Hi Ramesh,
>>>>>
>>>>> removing mridul, adding Alexey. Can we have this discussion on the
>>>>> dev alias? I'm sure a lot of peoples can collaborate :-)
>>>>>
>>>>> Ramesh wrote:
>>>>>
>>>>>
>>>>>> Hi JeanFrancois,
>>>>>> I looked at ARP code flow. The DefaultProcessorTask is always
>>>>>> initialized (in AsynReadTask) and the AsyncProcessorTask is just a
>>>>>> wrapper around the DefaultProcessorTask I guess it would be
>>>>>> inefficient to initialize a processor task even before we know if
>>>>>> we will actually be processing the request in the local container.
>>>>>>
>>>>> Yes, but I'm not longer following what you are trying to achieve ;-)
>>>>>
>>>>>
>>>>>
>>>> Our intention has always been to intercept a http request at the
>>>> earliest possible oppurtunity.. If we had our implementation of
>>>> Asynchandler and AsyncFilter, the only concern was that we will be
>>>> doing the interception after some of the artifacts have been created
>>>> (or initialized), which actually may not be required if this request
>>>> was going to be forwarded and not processed locally.
>>>>
>>>>
>>>>
>>>>>> If we intercept the request at the AsyncFilter, would it be not
>>>>>> expensive because we would be creating Processor tasks (which in
>>>>>> turn intializes request, response...) for every request.
>>>>>>
>>>>> Ouf..and good you care about performance :-) I recommended PU at
>>>>> the beginning of the project and that still applies :-) But if you
>>>>> need to park connection, you just need to write an AsyncFilter and
>>>>> enable ARP (only in that case) from your PU Handler. So you take
>>>>> the ARP way only when required.
>>>>>
>>>>>
>>>> Iam sorry , i did not understand this (still learning ARP :-)). Are
>>>> you suggesting we still use PU pipeline along with our
>>>> implementation of the ARP classes (asynchandler and asyncfilter).
>>>> Would this enable the handler to control the task execution
>>>> (park/resume).
>>>>
>>> ARP can be combined with PU. To enable ARP, you just call
>>> SelectorThread.setAsyncExecution(true). Now a possible solution for
>>> you would consist of writing your own AsyncHandler to decide if the
>>> request needs to be parked or not. To recap, if you need to park
>>> requests, you have four solutions:
>>>
>>> [1] Extends Port Unification mechanism (like you already did) and
>>> write your own way of parking the request.
>>> [2] Extends Port Unification, programmatically configure ARP (You can
>>> take a look at how openESB is doing it) to park only the requests you
>>> are interested
>>> [3] Extends ARP's AsyncHandler (you can replace the default
>>> programmatically using the SelectorThread API)
>>> [4] Write your own Pipeline Grizzly 1.0 (similar to the Port
>>> Unification Pipeline). Inside that Pipeline, first detect if the
>>> request has to be serviced locally or remotely. If locally, just call
>>> super.addTask(). If remotely, get an AsyncReadTask and execute it
>>> assuming you have written your AsyncHandler).
>>>
>>>
>>>
>>>> Also right now ARP can be enabled by setting system properties, is
>>>> there a programmatic way (API) for setting the ARP implementation
>>>> classes on the selector thread.
>>>>
>>> Yes, you can configure the AsyncHandler, AsyncExecutor and
>>> AsyncFilter from the SelectorThread.
>>>
>>> Hope that help.
>>>
>>> -- Jeanfrancois
>>>
>>>
>>>
>>>
>>>> -Ramesh
>>>>
>>>>
>>>>> Hope that help....
>>>>>
>>>>> Thanks
>>>>>
>>>>> -- Jeanfrancois
>>>>>
>>>>>
>>>>>
>>>>>> Thanks
>>>>>> -Ramesh
>>>>>>
>>>>>> Mridul Muralidharan wrote:
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I had plugged it into grizzly as part of the http processing ...
>>>>>>> after the request has been read, and before invoking the servlet
>>>>>>> container.
>>>>>>>
>>>>>>> Also, I was using what was exposed through grizzly (actually,
>>>>>>> whatever was in daily builds then ... glassfish2 b54 or
>>>>>>> thereabouts - grizzly 1.6 I guess).
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mridul
>>>>>>>
>>>>>>> Ramesh wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi JeanFrancois,
>>>>>>>> The requirements we have from the http proxy/clb in sailfin are
>>>>>>>>
>>>>>>>> 1. Intercept every Http request at the earliest possible time,
>>>>>>>> this would be using Grizzly 1.0, i don't think we are planning
>>>>>>>> to move to 1.6 on the server side.
>>>>>>>> 2. Parse the request line and headers..
>>>>>>>> 3. Support a pluggable invocation model so that interceptors can
>>>>>>>> be invoked on the request.
>>>>>>>> 4. The HA CLB (being one of the interceptors) takes a routing
>>>>>>>> decision based on the headers and request line. It identifies
>>>>>>>> the instance that needs to process this request. And the action
>>>>>>>> taken could be one of (5) or (6)
>>>>>>>> 5. This instance identified could be the local instance (the
>>>>>>>> container in the same instance) in which case it has to be
>>>>>>>> treated like a normal http request to a glassfish instance...
>>>>>>>> (nothing fancy needs to be done here))
>>>>>>>> 6. The instance is a remote instance in which case it has to be
>>>>>>>> forwarded to a remote instance, this would be using Grizzly 1.6
>>>>>>>> client APIs, the client channel has to be parked and a new
>>>>>>>> connection opened with the remote instance, the remote instance
>>>>>>>> might respond immediately or after some time, so we need to
>>>>>>>> asynchronously write data to the client channel whenever we
>>>>>>>> receive the response.
>>>>>>>> 7. We also have to support SSL from the http client to the front
>>>>>>>> end server, from the clb/proxy to any of the target instances
>>>>>>>> internally , we would not be using SSL. Basically the ssl off
>>>>>>>> loading takes place at the clb and we only propagate the cert
>>>>>>>> and auth information through http headers (just like we do
>>>>>>>> currently in the native-lb)
>>>>>>>> 8. Needless to say , the solution has to be scalable and
>>>>>>>> performant :-).
>>>>>>>>
>>>>>>>> Could you please advice if using ARP is the best way to satisfy
>>>>>>>> these requirements, and not PU.
>>>>>>>> Especially can (5) and (7), be satisfied. with ARP. Also the
>>>>>>>> clb/proxy feature can be configuratively turned off (in
>>>>>>>> domain.xml). Would it be possible to turn off ARP in such a case..
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -Ramesh
>>>>>>>>
>>>>>>>> Jeanfrancois Arcand wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi Ramesh, Mridul
>>>>>>>>>
>>>>>>>>> Mridul, do you still have the code you did when you extended
>>>>>>>>> Grizzly ARP. I know your final version is using Comet, but
>>>>>>>>> Ramesh is intersted about ARP and would like to see your code,
>>>>>>>>> if possible.
>>>>>>>>>
>>>>>>>>> Ramesh, checkout the JRuby module under Grizzly which contains
>>>>>>>>> a very simple example of an ARP AsyncFilter implementation.
>>>>>>>>>
>>>>>>>>> BTW I'm in Santa Clara this week for AjaxWorld and I will be in
>>>>>>>>> an out.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> -- Jeanfrancois
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ramesh wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi JeanFrancois,
>>>>>>>>>>
>>>>>>>>>> Could you please point me to the code (Sun Instant
>>>>>>>>>> Messenger) that uses ARP.
>>>>>>>>>>
>>>>>>>>>> Pls note: I have not ccied dev_at_grizzly, because i though the
>>>>>>>>>> Sun Messenger code is not open source
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> -Ramesh
>>>>>>>>>>
>>>>>>>>>> Jeanfrancois Arcand wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi Ramesh,
>>>>>>>>>>>
>>>>>>>>>>> Ramesh wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi Alexey,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Currently you can propogate a custom data using Context
>>>>>>>>>>>>>
>>>>>>>>>>>> attributes. Is it enough?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, should be enough in 1.6.1... ,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't think about possibility to dynamically switch the
>>>>>>>>>>>>>
>>>>>>>>>>>> protocol during connection lifecycle. But think it's not a
>>>>>>>>>>>> problem to add this feature.
>>>>>>>>>>>>
>>>>>>>>>>>> It depends on which life-cycle do we choose to associate a
>>>>>>>>>>>> handler, it could be for the lifetime of a selectionkey or
>>>>>>>>>>>> it could be for the lifetime of a request. Thats why i
>>>>>>>>>>>> propose to leave this to the handler itself so that the
>>>>>>>>>>>> design is flexible. We could add another attribute in the
>>>>>>>>>>>> PUProtocolRequest through which we can propagate whether or
>>>>>>>>>>>> not to map the handler back to the key, and this can be
>>>>>>>>>>>> set/reset by the handler itself.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Do you have any specific usecase?
>>>>>>>>>>>>>
>>>>>>>>>>>> Yes , if we are implementing a http proxy behavior in the
>>>>>>>>>>>> handler, where we are going to forward the received bytes to
>>>>>>>>>>>> another server, then after we have sent the entire data, we
>>>>>>>>>>>> just have to wait for the response from the server, IMO it
>>>>>>>>>>>> is inefficient to just wait and block at the handler until
>>>>>>>>>>>> the response is received from the server, we could register
>>>>>>>>>>>> a callback to handle the response when it arrives, but if we
>>>>>>>>>>>> returned from the handler we have to ensure the channel is
>>>>>>>>>>>> kept alive so that the response can be sent back on it.
>>>>>>>>>>>>
>>>>>>>>>>> In that case you should explore Grizzlly ARP (Asynchronous
>>>>>>>>>>> Request Processing) [1]. This extension was done exactly to
>>>>>>>>>>> support that use case. Port unification is not the place to
>>>>>>>>>>> implement such feature IMO, as it role is to:
>>>>>>>>>>>
>>>>>>>>>>> 1. Find the protocol
>>>>>>>>>>> 2. Redirect the request to the proper endpoind.
>>>>>>>>>>>
>>>>>>>>>>> In you case, you should 'leave' the Handler and enter the
>>>>>>>>>>> Grizzly ARP mechanism (like Comet is doing. You can taste it
>>>>>>>>>>> by playing with the Sun Instant Messenger tool internally. It
>>>>>>>>>>> use ARP/Comet to park requests).
>>>>>>>>>>>
>>>>>>>>>>> That is why it would be good if we can
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> separate out the readkey registering from the keepalive
>>>>>>>>>>>> logic so that we can just keep the channel alive without
>>>>>>>>>>>> registering a OP_READ. We should be able to register a
>>>>>>>>>>>> OP_READ once the response has been written back completely.
>>>>>>>>>>>>
>>>>>>>>>>> Comet[2] is the solution, not Port Unification IMO.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> -- Jeanfrancois
>>>>>>>>>>>
>>>>>>>>>>> [1]http://weblogs.java.net/blog/jfarcand/archive/2006/02/grizzly_part_ii.html
>>>>>>>>>>>
>>>>>>>>>>> [2]http://weblogs.java.net/blog/jfarcand/archive/2006/07/the_grizzly_com.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> -Ramesh
>>>>>>>>>>>>
>>>>>>>>>>>> Oleksiy Stashok wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Ramesh,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ramesh Parthasarathy wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will try to write a doc on how i have used PU, but it
>>>>>>>>>>>>>> might take some
>>>>>>>>>>>>>> time , so here are some initial thoughts
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> It could be very useful! Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Though the Finder and the Handler are meant for
>>>>>>>>>>>>>> specific purposes,
>>>>>>>>>>>>>> the ProtocoInfo/PUProtocolRequest (or the Context in
>>>>>>>>>>>>>> Grizzly 1.6.1)
>>>>>>>>>>>>>> object is the only standard means of sharing data between
>>>>>>>>>>>>>> them, it would
>>>>>>>>>>>>>> be good if we could propagate custom data from the Finder
>>>>>>>>>>>>>> to the Handler
>>>>>>>>>>>>>> so that we dont have to redo work we might have already
>>>>>>>>>>>>>> done in the
>>>>>>>>>>>>>> Finder. This custom object could be propagated through the
>>>>>>>>>>>>>> ProtocolInfo
>>>>>>>>>>>>>> or any other means.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently you can propogate a custom data using Context
>>>>>>>>>>>>> attributes. Is it enough?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. The logic in the PUpipeline/PUFilter that determines
>>>>>>>>>>>>>> whether the
>>>>>>>>>>>>>> Finder has to be invoked. If i understand correctly for a
>>>>>>>>>>>>>> selection Key
>>>>>>>>>>>>>> oncer the Finder returns with a protocol, we create a map
>>>>>>>>>>>>>> of the key and
>>>>>>>>>>>>>> the handler and this holds good until the key is
>>>>>>>>>>>>>> valid/channel is alive.
>>>>>>>>>>>>>> So, any further data that is read on that channel will
>>>>>>>>>>>>>> result in the
>>>>>>>>>>>>>> invocation of the same handler. For e.g the data could be
>>>>>>>>>>>>>> more data for
>>>>>>>>>>>>>> the same http request or it could be a new http request on
>>>>>>>>>>>>>> the same
>>>>>>>>>>>>>> channel (if keep-alive). I guess it would be good if we
>>>>>>>>>>>>>> allowed the
>>>>>>>>>>>>>> handler to control the association of the key and the
>>>>>>>>>>>>>> handler, so that
>>>>>>>>>>>>>> the logic of determining whether the finder is invoked on
>>>>>>>>>>>>>> the next chunk
>>>>>>>>>>>>>> of data can be controlled by user logic.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Hmm. I was looking at port unification as mechanism how
>>>>>>>>>>>>> newly accepted connection could be classified and in future
>>>>>>>>>>>>> it will be processed by certain (single) ProtocolHandler.
>>>>>>>>>>>>> I didn't think about possibility to dynamically switch the
>>>>>>>>>>>>> protocol during connection lifecycle. But think it's not a
>>>>>>>>>>>>> problem to add this feature.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3. The handler basically has to keep the channel alive
>>>>>>>>>>>>>> (return true),
>>>>>>>>>>>>>> until it is done using the channel. There may be
>>>>>>>>>>>>>> circumstances where the
>>>>>>>>>>>>>> handler is finished reading the entire data from the
>>>>>>>>>>>>>> channel, but it is
>>>>>>>>>>>>>> waiting (for some async event) to write more data into the
>>>>>>>>>>>>>> channel. In
>>>>>>>>>>>>>> such a case the channel has to kept alive (else the
>>>>>>>>>>>>>> cancelkey closed the
>>>>>>>>>>>>>> channel) . And if the channel is kept alive the key is
>>>>>>>>>>>>>> registered for
>>>>>>>>>>>>>> OP_READ ? Is there a way we can keep the channel alive and
>>>>>>>>>>>>>> indicate
>>>>>>>>>>>>>> somehow to the selector thread we are not expecting any
>>>>>>>>>>>>>> data to be read
>>>>>>>>>>>>>> as of now... Iam not sure if this is possible in 1.6.1
>>>>>>>>>>>>>> (sorry, i havent
>>>>>>>>>>>>>> gone through the 1.6.1 code in detail) , but it was
>>>>>>>>>>>>>> certainly difficult
>>>>>>>>>>>>>> in 1.0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> You're right, in 1.6.1 PU works the same way. PUFilter
>>>>>>>>>>>>> should be usually added after the ReadFilter, and
>>>>>>>>>>>>> ReadFilter, by default, reregisters connection for reading
>>>>>>>>>>>>> when postProcess is called.
>>>>>>>>>>>>> But again if it's required - we can think about possibility
>>>>>>>>>>>>> to avoid reregistering on OP_READ.
>>>>>>>>>>>>> Do you have any specific usecase?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you!!!
>>>>>>>>>>>>>
>>>>>>>>>>>>> WBR,
>>>>>>>>>>>>> Alexey.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please correct me if any of my observation is not correct.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> -Ramesh
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jeanfrancois Arcand wrote On 09/19/07 18:02,:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kudo to Alexey, the Grizzly's port unification[1] used by
>>>>>>>>>>>>>>> GlassFish v2 (Grizzly 1.0) is now available as a sub
>>>>>>>>>>>>>>> module in Project Grizzly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Port unification allow any server base application to
>>>>>>>>>>>>>>> support more than one protocol using a single tcp/udp/tls
>>>>>>>>>>>>>>> port. As as example, GlassFish v2 by default listen on
>>>>>>>>>>>>>>> port 8080 for http, https and Soap over tcp requests.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any volunteer to writes docs or tutorial? :-) :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Port Unification is currently available with version
>>>>>>>>>>>>>>> 1.6-SNAPSHOT and soon with 1.6.1.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- Jeanfrancois
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://weblogs.java.net/blog/jfarcand/archive/2006/11/one_port_to_rul.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>> dev-help_at_grizzly.dev.java.net
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> dev-help_at_grizzly.dev.java.net
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>>>>>>>>
>>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: dev-help_at_grizzly.dev.java.net
>