Hi Devaraj,
Devaraj Das wrote:
> Hi Jeanfrancois,
> Thanks for filing the bug! I would request you to fix this ASAP since it's
> critical for us.
Sure. We should have something this week if all goes well. Now if you
can't wait, you can always fallback to Grizzly 1.0. Not ideal, but a
possible workaround :-)
Thanks!
-- Jeanfrancois
> Regards,
> Devaraj.
>
> -----Original Message-----
> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
> Sent: Tuesday, June 12, 2007 10:14 PM
> To: users_at_grizzly.dev.java.net
> Cc: 'Owen O'Malley'; 'Sameer Paranjpye'
> Subject: Re: Grizzly in Hadoop
>
> Hi,
>
> Jeanfrancois Arcand wrote:
>> Hi Devaraj,
>>
>> Devaraj Das wrote:
>>> Hi Jeanfrancois,
>>> I will get back to you in detail in some time from now. But for now,
>>> I just want to update you that setting the keepalivetimeout to 0
>>> seemed to have improved performance. Thanks for that tip. Now, do you
>>> think, if we start using the persistent connections (transfer a batch
>>> of files at a time) feature, it would help us significantly?
>> Yes it would as you will save the network overhead of opening a
> connection.
>>> Also, will using multiple selectorReadThreads help the performance
>>> (since we have multiple CPUs on our machines)?
>> Right now it is not supported with the http module (Just wakes up I
>> didn't port it yet). Hence that's why you aren't seeing any
>> performance difference.
>
> I've filled:
>
> https://grizzly.dev.java.net/issues/show_bug.cgi?id=4
>
> to track the issue.
>
> -- Jeanfrancois
>
>> Taking about your benchmark, It would be interesting to understand how
>> you configure Jetty. Your connections seem to be data intensive as
>> opposed to connection intensive. NIO is good at handling large amount
>> of connections without spending idle threads but if the threads are
>> not idle anyway (as in this case), the place to make this faster is
>> only to reduce the number of bcopy/memcopy calls. I'm just thinking
>> loud here :-)
>>
>>
>>> I will send the source code of the adpater shortly...
>>> Thanks,
>> Thanks!
>>
>> -- Jeanfrancois
>>
>>> Devaraj.
>>>
>>> -----Original Message-----
>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>> Sent: Tuesday, June 12, 2007 6:25 PM
>>> To: users_at_grizzly.dev.java.net; Scott Oaks
>>> Subject: Re: Grizzly in Hadoop
>>>
>>> Hi,
>>>
>>> Devaraj Das wrote:
>>>> Hi,
>>>>
>>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>>> framework that has the MapReduce and Distributed File System
>>>> implementations). The main reason for using it is to optimize a
>>>> framework
>>> phase called "shuffle".
>>>> In this phase we move lots of data across the network.
>>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering
>>> using it
>>> :-)
>>>
>>>> We are currently using HTTP for moving the data (actually files) and
>>>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>>>> and all its niceness). But initial experiments with our benchmark
>>>> showed that with Grizzly the performance of the shuffle phase is
>>>> nearly the same as we have with Jetty5. This is not what we
>>>> initially expected and hence would like to get feedback on where we
>>>> might be going
>>> wrong.
>>>> Hadoop is designed to run on large clusters of 100s of nodes
>>>> (currently it can run stable/reliably in a 1K node cluster). From
>>>> the Grizzly point of view, what needs to be known is that each node
>>>> has a HTTP server. Both
>>>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>>>> service the incoming requests.
>>>>
>>>> There are 2 clients on each node, and each client has a configurable
>>>> number of fetcher threads. The fetcher code is written using the
>>>> java.net.URLConnection API.
>>>> Every node has both the server and the clients. The threads
>>>> basically hit the HTTP server asking for specific files. They all do
>>>> this at once (with some randomness in the order of hosts, maybe).
>>>>
>>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>>> cluster of 500 nodes. On the hardware side, I used a cluster that
>>>> has
>>>> 4 dualcore processors in each machine. The machines are partitioned
>>>> into racks with a gigabit ethernet within the rack and 100Mbps
>>>> across the racks. There are roughly 78000 independent files spread
>>>> across these 500 nodes each of size ~60KB that the client pulls (and
>>>> again we
>>> have two such clients per node).
>>>> So you can imagine you have a massive all-all communication
>>>> happening. The configuration for the server and client is as follows:
>>>> Grizzly configuration for port 9999
>>>> maxThreads: 100
>>>> minThreads: 1
>>>> ByteBuffer size: 8192
>>>> useDirectByteBuffer: false
>>>> useByteBufferView: false
>>>> maxHttpHeaderSize: 8192
>>>> maxKeepAliveRequests: 256
>>>> keepAliveTimeoutInSeconds: 10
>>>> Static File Cache enabled: true
>>>> Stream Algorithm :
>>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>>> Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>>> Round Robin Selector Algorithm enabled: false
>>>> Round Robin Selector pool size: 0
>>>> recycleTasks: true
>>>> Asynchronous Request Processing enabled: false I also tried
>>>> some configs with multiple selectorReadThreads but didn't make much
>>>> difference.
>>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
>>> reason to have such a low number? Closing faster idle connection?
>>>
>>>
>>>> The client has 30 fetcher threads and the way it is designed is that
>>>> only
>>>> one fetch from any given host would happen at any point of time. So if a
>>>> server host, h1, has 'n' files that we should pull, we do that one at a
>>> time
>>>> (as opposed to multiple threads hitting that server to fetch multiple
>>> files
>>>> in parallel).
>>>>
>>>> Also, we don't use the features of HTTP1.1 persistent connections or
>>>> pipelining. We fetch exactly one file and close the connection to the
>>>> server.
>>> That's answer my previous question :-) I would recommend setting the
>>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>>> improve (no call to the keep-alive subsystem).
>>>
>>>> With the above setup, the performance I see is not different from what I
>>> see
>>>> with Jetty5.
>>> Can it be the benchmark itself that it not able to load more?
>>>
>>>
>>> I see a lot of Read timeouts on the client side (and we have a
>>>> backoff (for the server host) on the client implementation whenever we
>>> fail
>>>> to fetch a file). I also saw some exceptions of the form on the server:
>>> You seem to hit the epoll problem on Linux. I know there is a way to
>>> avoid using epoll (a property). I will ping the NIO team and let you
>>> know.
>>>
>>> Also, what exactly your Adapter implementation is doing? Can you share
>>> the code? If I can have access to your setup, I would like to see if
>>> using Grizzly 1.0.15 makes a difference (just to make sure we don't
>>> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
>>> benchmark).
>>>
>>> Thanks,
>>>
>>> --Jeanfrancois
>>>
>>>
>>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>>> SEVERE: doSelect exception
>>>> java.io.IOException: Operation not permitted
>>>> at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>>> at
>>>>
> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>>> at
>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>> at
>>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>>> at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>>> at com.sun.grizzly.Controller.start(Controller.java:451)
>>>> at
>>>>
> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>>> at
>>>>
> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>>> at
>>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>>> Are we missing something in the configuration or something else?
>>>>
>>>> Thanks for the help.
>>>>
>>>> Regards,
>>>> Devaraj.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>