On Tue, 07 Oct 2008 00:07:34 +0900, Jeanfrancois Arcand
<Jeanfrancois.Arcand_at_sun.com> wrote:
> Salut,
>
> Emmanuel Lecharny wrote:
>> Hi guys,
>> just in case you missed it :
>>
>> http://feeds.feedburner.com/~r/trustin/~3/412416149/performance-comparison-between-nio-frameworks
>
> yes I know that was coming :-) But the benchmark contains some mistakes.
> Charlie (on that list) already pointed to me some issues, which we will
> share here and with Trustin.
Please feel free to contact me to fix the Grizzly-based implementation. I
want to make sure that the test code is as accurate as possible. That's
why I asked you to review the code briefly. Gregor of xSocket also
contacted me to fix my TODO item in xSocket echo server implementation.
>> It's a little bit annoying that Trustin didn't named the frameworks.
>
> Yes :-)
Performance test report is always annoying to some extent, regardless of
if the name of the frameworks were open or not. The masked names could
increase your curiosity though. It's JBoss policy to anonymize the names
of competitive technologies. On top of that, any performance test can
contain some mistakes, and I don't want a certain framework gets hurt by
my (potentially serious) mistake.
>> I'm
>> also questionning the specific test he is doing in this benchmark, as
>> it does not really exhibit the strengths and weaknesses of all the
>> tested frameworks, as it's just a very simple echo server, when the
>> power of a framework is to offer more than just transferring bytes in
>> and out.
Well, the performance test report specified scenario and environment as
precisely as I can. Write other type of tests and prove it.
>> However, I think that could be interesting to build some valid set of
>> benchmarks demonstrating different real life scenarii, helping us to
>> improve the code.
> I agree we should have something :-) Note that Trustin, when
> "re-creating" Netty, has the freedom to innovate and change any API he
> want. Something (at least with Grizzly) we can't do if we don't want to
> upset all our users.
Eventually, all NIO frameworks should have better performance than they
have now, and I believe any sort of automated benchmark will help us
somehow. Actually I was able to improve the performance of Netty much
more than I expected initially. Of course, there are many other places to
tune - codec framework and its implementation are good examples. I'd like
to write more tests that cover such use cases.
And... I believe Grizzly will do something cool in version 2. ;-)
>> Btw, I don't think that speed is the most important thing, ease of use
>> should also be a criterium when selecting a framework.
>
> Community is also important :-)
Sure. Ease of use is very important and performance should come later.
And the both factor should attract more people into the community.
Thanks for the feed back, and I'm looking forward to your idea to
improvement the accuracy of the test. :)
PS: BTW it takes really long time to run the tests (about 3 ~ 4 hrs for
each framework) to decrease the GC effect. I will run the test again when
I am confident that all suggested fixes are applied. Probably MINA and
Netty won't need any change but library upgrade, but Grizzly, xSocket, and
NIO framework might need some fix.
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/