Hi Alexey,
On May 11, 2009, at 9:45 AM, Oleksiy Stashok wrote:
> Hi Igor,
>
>>
>> grizzly-sendfile-0.3-SNAPSHOT (the one used for tests) uses grizzly
>> 1.9.15 not 1.0.x.
> oh, didn't know that. Shame on me :)
:-P
>
>> Is there more info/docs on aynchronous write queues?
> http://blogs.sun.com/oleksiys/entry/grizzly_1_7_0_presents
> we've changed API a little bit since that, but not much.
That looks like an overkill for grizzly-sendfile since it's dealing
with static data. However, I can imagine that something like this
would be useful for dynamically generated responses (webapps). Also
the results of my benchmarks of this feature in grizzly webserver are
not significantly different than when it is turned off.. see below.
>
>
>> btw I tried to run the same tests against glassfish v3b47 and it
>> kept on getting into a state where after ~9min of testing it would
>> start sending only partial responses or would refuse sending static
>> files all together. I didn't have a lot of time to investigate (it
>> took freaking forever to run all the tests I did), but I should run
>> my tests against grizzly webserver to see if the problem repeats
>> there.
>
>> There were two or three glassfish tests that passed and with
>> threadcount of 50 and buffer size 81k (10x the default), I still
>> saw that gf could achieve only 40-60% of grizzly-sendfile's
>> throughput.
> hmm. Ok, if you'll have any further results, please let us know.
I built a simple grizzly webserver:
GrizzlyWebServer ws = new GrizzlyWebServer(port, canonicalPath);
ws.setMaxThreads(threadCount); //set to 50
ws.getSelectorThread().setBufferSize(bufferSize); //set to 81000
//optionally: ws.getSelectorThread().setAsyncHttpWriteEnabled(true);
ws.start();
and tested it with the same benchmark as grizzly-sendfile and this is
what I got:
200 clients, not throttled:
Type Success Failure
Count Count
1KB Download 446 3
200KB Download 493 1
500KB Download 479 1
1MB Download 468 1
20MB Download 502 6
100MB Download 496 2
total throughput: 104.200 MB/s (grizzly-sendfile's best: 208.59 MB/s
&& 0 failures)
200 clients, throttled at 1MB/s:
Type Success Failure
Count Count
1KB Download 20 158
200KB Download 20 185
500KB Download 13 163
1MB Download 21 170
20MB Download 17 167
100MB Download 16 178
total throughput: 3.442 MB/s (grizzly-sendfile's best: 156.59 MB/s &&
0 failures)
Most if not all of the failures were connection timeout exceptions
(despite connection.setConnectTimeout(0); and
connection.setReadTimeout(0); in my benchmark code) or connection
reset by peer exceptions.
I didn't notice any significant difference between
AsyncHttpWriteEnabled on or off, which is quite surprising.
The load test also uncovered a bunch of concurrency bugs - the stack
traces are below.
Am I doing something wrong?
cheers,
Igor
May 11, 2009 10:37:24 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69)
at
sun
.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:
105)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:74)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:
492)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
183)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:37:24 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:69)
at
sun
.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:
105)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:74)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:
492)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
183)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:47:37 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:47:37 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:51:56 PM com.sun.grizzly.http.ProcessorTask
invokeAdapter
SEVERE: HTTP Processing error
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:501)
at
com
.sun
.grizzly
.http
.SocketChannelOutputBuffer.sendFile(SocketChannelOutputBuffer.java:357)
at
com
.sun
.grizzly
.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:217)
at
com
.sun
.grizzly
.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:139)
at
com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
123)
at
com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:655)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:905)
at
com
.sun
.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:
161)
at
com
.sun
.grizzly
.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:
136)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
103)
at
com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
89)
at
com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:
76)
at
com
.sun
.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:
53)
at
com
.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:
57)
at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
at
com
.sun
.grizzly
.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:356)
at
com
.sun
.grizzly
.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:
256)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
193)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:54:19 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:54:19 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:56:10 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:56:10 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:56:10 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:56:10 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:57:06 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:57:06 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:59:18 PM com.sun.grizzly.http.SelectorThread$3
onException
SEVERE: Exception during controller processing
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
May 11, 2009 10:59:18 PM com.sun.grizzly.SelectorHandlerRunner
handleSelectException
SEVERE: doSelect exception
java.lang.NullPointerException
at
com
.sun
.grizzly
.http
.SelectorThreadKeyHandler.doRegisterKey(SelectorThreadKeyHandler.java:
86)
at
com
.sun
.grizzly
.DefaultSelectionKeyHandler.register(DefaultSelectionKeyHandler.java:
133)
at
com
.sun
.grizzly
.TCPSelectorHandler.processPendingOperations(TCPSelectorHandler.java:
445)
at
com.sun.grizzly.TCPSelectorHandler.preSelect(TCPSelectorHandler.java:
370)
at
com
.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:
181)
at
com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:
128)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
>
>
> Thank you.
>
> WBR,
> Alexey.
>
>>
>> On May 11, 2009, at 2:02 AM, Oleksiy Stashok wrote:
>>
>>> Hi Igor,
>>>
>>> that's great blog and great observations.
>>> It could be interesting to check one more strategy here:
>>> asynchronous write queues. Unfortunately we don't have
>>> implementation of one for Grizzly 1.0.x.
>>> Unlike ENBA, asynchronous write queue first tries to write buffer
>>> immediately, and only if it's not able to write whole buffer - it
>>> registers for async. write (using the same selector, not new one).
>>>
>>> IMHO this approach could work better than ENBA, because, we will
>>> be able to avoid selector/thread switches.
>>>
>>> Thank you!
>>>
>>> WBR,
>>> Alexey.
>>>
>>> On May 11, 2009, at 4:17 , Igor Minar wrote:
>>>
>>>> I was hoping to post this sooner, but came across some *very*
>>>> surprising results that I had to thoroughly validate - the
>>>> blocking IO can scale!
>>>>
>>>> http://blog.igorminar.com/2009/05/grizzly-sendfile-and-comparison-of.html
>>>>
>>>> Let me know what you think, not very many people do multiplexed
>>>> blocking IO and I think that it is an area that has a lot of
>>>> potential and deserves more research.
>>>>
>>>> cheers,
>>>> Igor
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>