Forgot to mention one detail .. the file serving happens through the service
method in an Adapter that we implemented. It happens this way since we parse
a few arguments in the http request url to figure out which file to serve,
etc.
-----Original Message-----
From: Devaraj Das [mailto:ddas_at_yahoo-inc.com]
Sent: Tuesday, June 12, 2007 11:57 AM
To: 'users_at_grizzly.dev.java.net'
Subject: Grizzly in Hadoop
Hi,
We are considering using Grizzly (1.5) in Hadoop (an open source framework
that has the MapReduce and Distributed File System implementations). The
main reason for using it is to optimize a framework phase called "shuffle".
In this phase we move lots of data across the network.
We are currently using HTTP for moving the data (actually files) and we use
Jetty5. Now we are thinking of moving to Grizzly (to have NIO and all its
niceness). But initial experiments with our benchmark showed that with
Grizzly the performance of the shuffle phase is nearly the same as we have
with Jetty5. This is not what we initially expected and hence would like to
get feedback on where we might be going wrong.
Hadoop is designed to run on large clusters of 100s of nodes (currently it
can run stable/reliably in a 1K node cluster). From the Grizzly point of
view, what needs to be known is that each node has a HTTP server. Both
Jetty5 and Grizzly provides the ability to have multiple handlers to service
the incoming requests.
There are 2 clients on each node, and each client has a configurable number
of fetcher threads. The fetcher code is written using the
java.net.URLConnection API.
Every node has both the server and the clients. The threads basically hit
the HTTP server asking for specific files. They all do this at once (with
some randomness in the order of hosts, maybe).
The benchmark that I tested with is a sort for ~5TB of data with a cluster
of 500 nodes. On the hardware side, I used a cluster that has 4 dualcore
processors in each machine. The machines are partitioned into racks with a
gigabit ethernet within the rack and 100Mbps across the racks. There are
roughly 78000 independent files spread across these 500 nodes each of size
~60KB that the client pulls (and again we have two such clients per node).
So you can imagine you have a massive all-all communication happening. The
configuration for the server and client is as follows:
Grizzly configuration for port 9999
maxThreads: 100
minThreads: 1
ByteBuffer size: 8192
useDirectByteBuffer: false
useByteBufferView: false
maxHttpHeaderSize: 8192
maxKeepAliveRequests: 256
keepAliveTimeoutInSeconds: 10
Static File Cache enabled: true
Stream Algorithm :
com.sun.grizzly.http.algorithms.NoParsingAlgorithm
Pipeline : com.sun.grizzly.http.LinkedListPipeline
Round Robin Selector Algorithm enabled: false
Round Robin Selector pool size: 0
recycleTasks: true
Asynchronous Request Processing enabled: false I also tried some
configs with multiple selectorReadThreads but didn't make much difference.
The client has 30 fetcher threads and the way it is designed is that only
one fetch from any given host would happen at any point of time. So if a
server host, h1, has 'n' files that we should pull, we do that one at a time
(as opposed to multiple threads hitting that server to fetch multiple files
in parallel).
Also, we don't use the features of HTTP1.1 persistent connections or
pipelining. We fetch exactly one file and close the connection to the
server.
With the above setup, the performance I see is not different from what I see
with Jetty5. I see a lot of Read timeouts on the client side (and we have a
backoff (for the server host) on the client implementation whenever we fail
to fetch a file). I also saw some exceptions of the form on the server:
Jun 11, 2007 5:04:51 PM com.sun.grizzly.Controller doSelect
SEVERE: doSelect exception
java.io.IOException: Operation not permitted
at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
at
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:202)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:183)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:277)
at com.sun.grizzly.Controller.doSelect(Controller.java:218)
at com.sun.grizzly.Controller.start(Controller.java:451)
at
com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1158)
at
com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1121)
at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1099)
Are we missing something in the configuration or something else?
Thanks for the help.
Regards,
Devaraj.