users@grizzly.java.net

grizzly-sendfile (was Re: High latency of ARP calls in Glassfish v2)

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 26 Nov 2008 15:31:30 -0500

Salut,

Igor Minar wrote:
>
>
> On Nov 26, 2008, at 11:33 AM, Jeanfrancois Arcand wrote:
>
>> Thanks Igor for the feedback.
>>
>> BTW I've fixed:
>>
>> https://grizzly.dev.java.net/issues/show_bug.cgi?id=289
>
> I saw that, thanks!
>
>> You can update GF with the 1.9.0 SNAPSHOT:
>>
>> http://weblogs.java.net/blog/jfarcand/archive/2008/11/updating_grizzl.html
>>
>>
>> and see if that works.
>
> I'll give it a shot. I ran into some class loading issues in GFv3, which
> I didn't have time to troubleshoot yet. I think that I'll need to
> rewrite my extension as on OSGi module in order to get it to work
> properly in GFv3.

That sucks. Let me see if I can find something on my side.


>
> I'm currently targeting GFv2 because that's the version of the app
> server we use to run our application for which I developed this
> extension. But I'm interested in supporting GFv3 in the (near?) future
> as well. It really depends only on how much free time I have.
>
>> BTW would you be interested to contribute your extension (if it is
>> generic enough?)
>
> All the extension does is that sending (usually large) files to a client
> based on instructions from an app. It is very generic and reusable by
> any kind of app (JavaEE, Rails, Grails, ..) running on glassfish and in
> fact any app using grizzly (currently only the old grizzly version
> bundled with GFv2).

OK, for people reading it is 1.0.22 :-)


This is basically a functionality that has been
> supported by apache and many other web servers for a long time, but
> never made it to glassfish or any other JaveEE server AFAIK.
>
> To keep it as generic as possible, I actually went as far as
> implementing my own extension points, so that I can write plugins for my
> grizzly extension.

This is quite interesting. I know you were working in it but wasn't
aware it was public.

>
> It is licensed as GPL and used in production at mediacast.sun.com.
>
> I'll try to set aside some time during the upcoming long weekend to
> write some user docs and explain all the benefits. Once that is done,
> I'll be sure to send out an announcement to this mailing list.

Great thanks!

>
> In the meantime, you can check out the source code at
> http://kenai.com/projects/grizzly-sendfile
>
> While I have you attention, there is one issue that I was not able to
> resolve, and thus had to resort to patching grizzly. My extension
> deregisters a channel from the grizzly's selector and uses it's own
> selector and worker thread pool (pipeline) to execute the request in
> order to have a better control over how the file download is being
> handled (I support several file sending algorithms).
>
> The problem I found is that without patching grizzly, my extension works
> properly only for HTTP keep-alive connections. If the connection is not
> keep-alive, grizzly will close the connection when I deregister the
> channel and my filter exists. This means that when I finally get around
> to start pumping the data to the client, the connection is already closed.
>
> What I did to resolve this was to patch DefaultProcessorTask to expose a
> new forceKeepAlive() method:
> http://kenai.com/projects/grizzly-sendfile/sources/605-Mercurial-Source-Code-Repository/revision/44
>
>
> Pretty ugly, but with the API of grizzly 1.0.x, I don't think that there
> is a way around it. Is there any chance that you could give me a way to
> handle this via an official grizzly api (ideally something that would
> ship with glassfish v2.1). All I need is to prevent grizzly from closing
> the socketchannel when I deregister it from the main selector (and thus
> take over the responsibility for closing it).


Just invoke:

selectorThread.addBannedSelectionKey(..);

> /**
> * Add a <code>SelectionKey</code> to the banned list of
SelectionKeys.
> * A SelectionKey is banned when new registration aren't allowed
on the
> * Selector.
> */
> public void addBannedSelectionKey(SelectionKey key){
> bannedKeys.offer(key);
> }

I'm using this trick for Comet. Which version of GF are you using? I
suspect the API is there.

A+

-- Jeanfrancois






>
> cheers,
> Igor
>
>
>
>
>
>>
>>
>> A+
>>
>> -- Jeanfrancois
>>
>> Igor Minar wrote:
>>> Alexey helped me out on this one offline and I just want to share the
>>> resolution.
>>> In the end we figured out that the problem was not caused by grizzly,
>>> but by bugs in my code and in the benchmarking code, combination of
>>> which made it pretty difficult to spot the real problem.
>>> When it comes to issue in my code, the delay was caused by calling
>>> SocketChannel#register before Selector#wakeup, which always resulted
>>> in a nasty blocking == delay. Alexey recommended to rewrite the code
>>> so that the registration is executed by the selector thread, which
>>> processes a registration queue (similarly to how grizzly does this
>>> already). I did that and the delay is now gone.
>>> Thanks Alexey!
>>> /i
>>> On Oct 23, 2008, at 2:10 AM, Oleksiy Stashok wrote:
>>>>>>
>>>>>> Hello Igor,
>>>>>>
>>>>>>> I'm building an ARP filter for grizzly that I use in
>>>>>>> glassfish-v2ur2 and I'm seeing huge request latency under
>>>>>>> moderate load (20 users). When I run tests with a single user,
>>>>>>> the delay is usually insignificant.
>>>>>>>
>>>>>>> I wrote a faban benchmark for my filter and also added some
>>>>>>> debugging code that tracks processing time in the filter and I
>>>>>>> came to the conclusion that the delay occurs before the request
>>>>>>> hits the doFilter method of my filter, which means that the delay
>>>>>>> is occurring in grizzly or glassfish and not in my filter.
>>>>>>>
>>>>>>> I see the problem repeatedly. The 90th percentile of the delay is
>>>>>>> around 10 seconds (the average is 4-5 seconds, max is 30+ seconds).
>>>>>>>
>>>>>>> Are there any known issues with grizzly used in glassfish that
>>>>>>> could cause this behavior?
>>>>>> No, it's not known, at least for me. It could be related to
>>>>>> situation, when some thread (other than Selector thread) tries to
>>>>>> operate with a selector directly.
>>>>>> Please make sure your Filter doesn't do something like that.
>>>>>>
>>>>>> For sure it could be good if you can provide some testcase, which
>>>>>> reproduces the problem, so it's possible to run it locally and debug.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> WBR,
>>>>>> Alexey.
>>>>>
>>>>> Since I work for Sun, I could set up a box on SWAN and give you
>>>>> access to it. Another alternative is that I'll send you all the
>>>>> code and instructions on how to set it up. The code is open source
>>>>> so either way will work for me.
>>>> I prefer 1st option :) Please let me know the coordinates of the
>>>> machine.
>>>>
>>>> Thank you.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>
>>>>>
>>>>>
>>>>> thanks,
>>>>> Igor
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> cheers,
>>>>>>> Igor
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here are relevant parts of the domain.xml:
>>>>>>>
>>>>>>> <jvm-options>-Dcom.sun.enterprise.web.connector.grizzly.asyncHandlerClass=com.sun.enterprise.web.connector.grizzly.async.DefaultAsyncHandler</jvm-options>
>>>>>>>
>>>>>>> <jvm-options>-Dcom.sun.enterprise.web.connector.grizzly.asyncHandler.ports=8080</jvm-options>
>>>>>>>
>>>>>>> <jvm-options>-Dcom.sun.enterprise.web.connector.grizzly.asyncFilters=com.igorminar.grizzlysendfile.SendfileFilter</jvm-options>
>>>>>>>
>>>>>>>
>>>>>>> <http-listener acceptor-threads="5" address="0.0.0.0"
>>>>>>> blocking-enabled="false" default-virtual-server="server"
>>>>>>> enabled="true" family="inet" id="http-listener-1" port="8080"
>>>>>>> security-enabled="false" server-name="" xpowered-by="true">
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>