dev@grizzly.java.net

Re: Strange performance on HP-UX

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Tue, 30 Jun 2009 10:09:26 -0400

Salut,

Bongjae Chang wrote:
> Hi,
>
> I debugged the problem on HP, I found the following error frequently.
>
> ---
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:764)
> at
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:453)
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:526)
> at
> com.sun.grizzly.http.SocketChannelOutputBuffer.sendFile(SocketChannelOutputBuffer.java:357)

So there is an issue with the HP JDK. One way to fix that is to avoid
using the send-file like we are doing on Linux with JDK < 1.7 (suffer a
similar problem). The workaround is to add
-Dcom.sun.grizzly.useSendFile=false. Can you try that once you have a
chance? If that works, we will just force that value to be false on HP-UX.

Great Work!!

-- Jeanfrancois



> at
> com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:217)
> at
> com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:139)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:124)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:749)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:657)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:908)
> at
> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
> at
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
> at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
> at
> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
> at
> com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:365)
> at
> com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:257)
> at
> com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:194)
> at
> com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:129)
> at
> com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379)
> at
> com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
> ... 26 more
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:764)
> at
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:453)
> at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:526)
> at
> com.sun.grizzly.http.SocketChannelOutputBuffer.sendFile(SocketChannelOutputBuffer.java:357)
> at
> com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:217)
> at
> com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:139)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:124)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180)
> at
> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:749)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:657)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:908)
> at
> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
> at
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
> at
> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
> at
> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
> at
> com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:365)
> at
> com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:257)
> at
> com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:194)
> at
> com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:129)
> at
> com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379)
> at
> com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
> ... 26 more
> ...
> ---
>
> So I used "-Dcom.sun.grizzly.useSendFile=false", then the perf problem
> was not occurred.
>
> When I saw the option and grizzly issue
> #327(https://grizzly.dev.java.net/issues/show_bug.cgi?id=327), linux had
> a problem in FileChannel.transferTo().
>
> So I thought that maybe HP could also have similar problem, but I am not
> sure.
>
> And I saw another problem when I tested it.
>
> Though I requested 1k.jpg, sometimes I received error page's text html.
>
> ---
> HTTP/1.1 200 OK
> Content-Type: image/jpeg
> Content-Length: 999
> Date: Tue, 30 Jun 2009 03:13:00 GMT
>
> <html><head><title>Grizzly</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
> B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}HR {color : #525D76;}--></style> </head><body><h1>Not
> Found</h1><HR size="1" noshade><h3>Grizzly</h3></body></html>
> ---
> ("Content-Length: 999" means 1k.jpg's length)
>
>
> It seems that http header and body are not matched. But when I saw
> StaticResourcesAdapter#service() and GrizzlyAdapterChain#service(), I
> could understand the result.
>
> 1. First, sends the header to the browser in StaticResourcesAdapter
> 2. Next, tries to send the body, but an exception is occurred in
> StaticResourcesAdapter
> 3. Catches the throwable and sends customizedErrorPage, but http header
> was already sent to the browser.
>
> Is the result expected?
>
> Thanks.
>
> --
> Bongjae Chang
>
>
>
> ----- Original Message -----
> *From:* Bongjae Chang <mailto:carryel_at_korea.com>
> *To:* dev_at_grizzly.dev.java.net <mailto:dev_at_grizzly.dev.java.net>
> *Sent:* Monday, June 29, 2009 9:29 PM
> *Subject:* Re: Strange performance on HP-UX
>
> Hi Alexey,
>
> Sure. I will look into this problem with profiler and reply again.
>
> Thanks.
>
> --
> Bongjae Chang
>
>
>
> ----- Original Message -----
> *From:* Oleksiy Stashok <mailto:Oleksiy.Stashok_at_Sun.COM>
> *To:* dev_at_grizzly.dev.java.net <mailto:dev_at_grizzly.dev.java.net>
> *Sent:* Monday, June 29, 2009 6:05 PM
> *Subject:* Re: Strange performance on HP-UX
>
> Hi Bongjae,
>
> that's interesting observation, I doubt we make any perf test on
> HP-UX :( AFAIK our main perf. tests are run on Solaris, we also
> make some local bm measurements on Linux, Window, MacOS.
> Is it possible for you to run some profiler to see, where this
> issue happens?
>
> Thanks.
>
> WBR,
> Alexey.
>
>>
>> When I tested Grizzly on HP-UX with MS stress tool, I met
>> strange performance result.
>>
>> I deployed a war with grizzly-http-servlet-deployer and just
>> downloaded a static content like 1k.jpg.
>>
>> On Windows, Linux and AIX, I didn't see the performance problem.
>>
>> But on only HP-UXs, I saw very very low performance.
>>
>> I tested it in two HP machine, but two machine's problems were
>> same.
>>
>> Here are system environments of two HP-UX.
>> ---
>> HP(IA64)
>> HOSTID: 627EAB14
>> NCPUS: 4
>> OPENFILES: 8192/10240
>> UNAME: HP-UX tschi2 B.11.23 U ia64 1652468500 unlimited-user
>> license"
>>
>> "java version ""1.6.0.04""
>> Java(TM) SE Runtime Environment (build
>> 1.6.0.04-jinteg_28_apr_2009_04_46-b00)
>> Java HotSpot(TM) Server VM (build 11.3-b02-jre1.6.0.04-rc2,
>> mixed mode)"
>> ---
>> HP(PA-RISC)
>> HOSTID: E0DD7C9E
>> NCPUS: 2
>> OPENFILES: 8192/10240
>> UNAME: HP-UX tschp2 B.11.23 U 9000/800 3772611742
>> unlimited-user license"
>>
>> "java version ""1.6.0.04""
>> Java(TM) SE Runtime Environment (build
>> 1.6.0.04-jinteg_28_apr_2009_04_46-b00)
>> Java HotSpot(TM) Server VM (build 11.3-b02-jre1.6.0.04-rc2
>> PA2.0 (aCC_AP), mixed mode)"
>> ---
>>
>> I don't understand the following result. I tested it in both
>> grizzly-1.9.16 and grizzly-1.9.17.
>>
>> Here is a sample result.
>>
>> [In Windows]
>> ---
>> Number of hits: *285010*
>> Requests per Second: 2373.28
>>
>> Socket Statistics
>> --------------------------------------------------------------------------------
>> Socket Connects: 285035
>> Total Bytes Sent (in KB): 56227.61
>> Bytes Sent Rate (in KB/s): 468.21
>> Total Bytes Recv (in KB): 312008.02
>> Bytes Recv Rate (in KB/s): 2598.09
>> ---
>>
>> [In HP]
>> ---
>> Number of hits: *1316*
>> Requests per Second: 10.97
>>
>> Socket Statistics
>> --------------------------------------------------------------------------------
>> Socket Connects: 1341
>> Total Bytes Sent (in KB): 265.84
>> Bytes Sent Rate (in KB/s): 2.22
>> Total Bytes Recv (in KB): 1335.58
>> Bytes Recv Rate (in KB/s): 11.13
>> ---
>>
>> Has anybody met this problem?
>>
>> I think that the Grizzly QA may do the performance test
>> regularly. Do you happen to test on HP-UX?
>>
>> Please advice me.
>>
>> Thanks.
>>
>> --
>> Bongjae Chang
>>
>>
>