Hi,
Jeanfrancois Arcand wrote:
> Hi Devaraj,
>
> Devaraj Das wrote:
>> Hi Jeanfrancois,
>> I will get back to you in detail in some time from now. But for now, I
>> just
>> want to update you that setting the keepalivetimeout to 0 seemed to have
>> improved performance. Thanks for that tip. Now, do you think, if we start
>> using the persistent connections (transfer a batch of files at a time)
>> feature, it would help us significantly?
>
> Yes it would as you will save the network overhead of opening a connection.
>
>> Also, will using multiple selectorReadThreads help the performance
>> (since we
>> have multiple CPUs on our machines)?
>
> Right now it is not supported with the http module (Just wakes up I
> didn't port it yet). Hence that's why you aren't seeing any performance
> difference.
I've filled:
https://grizzly.dev.java.net/issues/show_bug.cgi?id=4
to track the issue.
-- Jeanfrancois
>
> Taking about your benchmark, It would be interesting to understand how
> you configure Jetty. Your connections seem to be data intensive as
> opposed to connection intensive. NIO is good at handling large amount of
> connections without spending idle threads but if the threads are not
> idle anyway (as in this case), the place to make this faster is only to
> reduce the number of bcopy/memcopy calls. I'm just thinking loud here :-)
>
>
>> I will send the source code of the adpater shortly...
>> Thanks,
>
> Thanks!
>
> -- Jeanfrancois
>
>> Devaraj.
>>
>> -----Original Message-----
>> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>> Sent: Tuesday, June 12, 2007 6:25 PM
>> To: users_at_grizzly.dev.java.net; Scott Oaks
>> Subject: Re: Grizzly in Hadoop
>>
>> Hi,
>>
>> Devaraj Das wrote:
>>> Hi,
>>>
>>> We are considering using Grizzly (1.5) in Hadoop (an open source
>>> framework that has the MapReduce and Distributed File System
>>> implementations). The main reason for using it is to optimize a
>>> framework
>> phase called "shuffle".
>>> In this phase we move lots of data across the network.
>>
>> Cool :-) I know Hadoop as your Web 2.0 stack is also considering using it
>> :-)
>>
>>> We are currently using HTTP for moving the data (actually files) and
>>> we use Jetty5. Now we are thinking of moving to Grizzly (to have NIO
>>> and all its niceness). But initial experiments with our benchmark
>>> showed that with Grizzly the performance of the shuffle phase is
>>> nearly the same as we have with Jetty5. This is not what we initially
>>> expected and hence would like to get feedback on where we might be going
>> wrong.
>>> Hadoop is designed to run on large clusters of 100s of nodes
>>> (currently it can run stable/reliably in a 1K node cluster). From the
>>> Grizzly point of view, what needs to be known is that each node has a
>>> HTTP server. Both
>>> Jetty5 and Grizzly provides the ability to have multiple handlers to
>>> service the incoming requests.
>>>
>>> There are 2 clients on each node, and each client has a configurable
>>> number of fetcher threads. The fetcher code is written using the
>>> java.net.URLConnection API.
>>> Every node has both the server and the clients. The threads basically
>>> hit the HTTP server asking for specific files. They all do this at
>>> once (with some randomness in the order of hosts, maybe).
>>>
>>> The benchmark that I tested with is a sort for ~5TB of data with a
>>> cluster of 500 nodes. On the hardware side, I used a cluster that has
>>> 4 dualcore processors in each machine. The machines are partitioned
>>> into racks with a gigabit ethernet within the rack and 100Mbps across
>>> the racks. There are roughly 78000 independent files spread across
>>> these 500 nodes each of size ~60KB that the client pulls (and again we
>> have two such clients per node).
>>> So you can imagine you have a massive all-all communication
>>> happening. The configuration for the server and client is as follows:
>>> Grizzly configuration for port 9999
>>> maxThreads: 100
>>> minThreads: 1
>>> ByteBuffer size: 8192
>>> useDirectByteBuffer: false
>>> useByteBufferView: false
>>> maxHttpHeaderSize: 8192
>>> maxKeepAliveRequests: 256
>>> keepAliveTimeoutInSeconds: 10
>>> Static File Cache enabled: true
>>> Stream Algorithm :
>>> com.sun.grizzly.http.algorithms.NoParsingAlgorithm
>>> Pipeline : com.sun.grizzly.http.LinkedListPipeline
>>> Round Robin Selector Algorithm enabled: false
>>> Round Robin Selector pool size: 0
>>> recycleTasks: true
>>> Asynchronous Request Processing enabled: false I also tried
>>> some configs with multiple selectorReadThreads but didn't make much
>>> difference.
>>
>> keepAliveTimeoutInSeconds: 10 seems for me a little low...what is the
>> reason
>> to have such a low number? Closing faster idle connection?
>>
>>
>>> The client has 30 fetcher threads and the way it is designed is that
>>> only
>>> one fetch from any given host would happen at any point of time. So if a
>>> server host, h1, has 'n' files that we should pull, we do that one at a
>> time
>>> (as opposed to multiple threads hitting that server to fetch multiple
>> files
>>> in parallel).
>>>
>>> Also, we don't use the features of HTTP1.1 persistent connections or
>>> pipelining. We fetch exactly one file and close the connection to the
>>> server.
>>
>> That's answer my previous question :-) I would recommend setting the
>> keepAliveTimeoutInSeconds=0 then as I'm sure the performance will
>> improve (no call to the keep-alive subsystem).
>>
>>> With the above setup, the performance I see is not different from what I
>> see
>>> with Jetty5.
>>
>> Can it be the benchmark itself that it not able to load more?
>>
>>
>> I see a lot of Read timeouts on the client side (and we have a
>>> backoff (for the server host) on the client implementation whenever we
>> fail
>>> to fetch a file). I also saw some exceptions of the form on the server:
>>
>> You seem to hit the epoll problem on Linux. I know there is a way to
>> avoid using epoll (a property). I will ping the NIO team and let you
>> know.
>>
>> Also, what exactly your Adapter implementation is doing? Can you share
>> the code? If I can have access to your setup, I would like to see if
>> using Grizzly 1.0.15 makes a difference (just to make sure we don't
>> have a bug in 1.5....as far as I can tell, I.5 is as fast as 1.0 on my
>> benchmark).
>>
>> Thanks,
>>
>> --Jeanfrancois
>>
>>
>>> Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
>>> SEVERE: doSelect exception
>>> java.io.IOException: Operation not permitted
>>> at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
>>> at
>>>
>> sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
>>
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
>>> at
>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
>>> at com.sun.grizzly.Controller.doSelect(Controller.java:218)
>>> at com.sun.grizzly.Controller.start(Controller.java:451)
>>> at
>>>
>> com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
>>
>>> at
>>>
>> com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
>>
>>> at
>> com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
>>> Are we missing something in the configuration or something else?
>>>
>>> Thanks for the help.
>>>
>>> Regards,
>>> Devaraj.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>