dev@grizzly.java.net

Re: Strange performance on HP-UX

From: Bongjae Chang <carryel_at_korea.com>
Date: Tue, 30 Jun 2009 12:23:51 +0900

Hi,

I debugged the problem on HP, I found the following error frequently.

---
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:764)
        at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:453)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:526)
        at com.sun.grizzly.http.SocketChannelOutputBuffer.sendFile(SocketChannelOutputBuffer.java:357)
        at com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:217)
        at com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:139)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:124)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:749)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:657)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:908)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:365)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:257)
        at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:194)
        at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:129)
        at com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379)
        at com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
        ... 26 more
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:764)
        at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:453)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:526)
        at com.sun.grizzly.http.SocketChannelOutputBuffer.sendFile(SocketChannelOutputBuffer.java:357)
        at com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:217)
        at com.sun.grizzly.tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:139)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:124)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:749)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:657)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:908)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:161)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:365)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:257)
        at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:194)
        at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:129)
        at com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379)
        at com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:761)
        ... 26 more
...
---

So I used "-Dcom.sun.grizzly.useSendFile=false", then the perf problem was not occurred.

When I saw the option and grizzly issue #327(https://grizzly.dev.java.net/issues/show_bug.cgi?id=327), linux had a problem in FileChannel.transferTo().

So I thought that maybe HP could also have similar problem, but I am not sure.

And I saw another problem when I tested it.

Though I requested 1k.jpg, sometimes I received error page's text html.

---
HTTP/1.1 200 OK
Content-Type: image/jpeg
Content-Length: 999
Date: Tue, 30 Jun 2009 03:13:00 GMT

<html><head><title>Grizzly</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}HR {color : #525D76;}--></style> </head><body><h1>Not Found</h1><HR size="1" noshade><h3>Grizzly</h3></body></html>
---
("Content-Length: 999" means 1k.jpg's length)


It seems that http header and body are not matched. But when I saw StaticResourcesAdapter#service() and GrizzlyAdapterChain#service(), I could understand the result.

1. First, sends the header to the browser in StaticResourcesAdapter
2. Next, tries to send the body, but an exception is occurred in StaticResourcesAdapter
3. Catches the throwable and sends customizedErrorPage, but http header was already sent to the browser.

Is the result expected?

Thanks.

--
Bongjae Chang


  ----- Original Message -----
  From: Bongjae Chang
  To: dev_at_grizzly.dev.java.net
  Sent: Monday, June 29, 2009 9:29 PM
  Subject: Re: Strange performance on HP-UX


  Hi Alexey,

  Sure. I will look into this problem with profiler and reply again.

  Thanks.

  --
  Bongjae Chang


    ----- Original Message -----
    From: Oleksiy Stashok
    To: dev_at_grizzly.dev.java.net
    Sent: Monday, June 29, 2009 6:05 PM
    Subject: Re: Strange performance on HP-UX


    Hi Bongjae,


    that's interesting observation, I doubt we make any perf test on HP-UX :( AFAIK our main perf. tests are run on Solaris, we also make some local bm measurements on Linux, Window, MacOS.
    Is it possible for you to run some profiler to see, where this issue happens?


    Thanks.


    WBR,
    Alexey.




      When I tested Grizzly on HP-UX with MS stress tool, I met strange performance result.

      I deployed a war with grizzly-http-servlet-deployer and just downloaded a static content like 1k.jpg.

      On Windows, Linux and AIX, I didn't see the performance problem.

      But on only HP-UXs, I saw very very low performance.

      I tested it in two HP machine, but two machine's problems were same.

      Here are system environments of two HP-UX.
      ---
      HP(IA64)
      HOSTID: 627EAB14
      NCPUS: 4
      OPENFILES: 8192/10240
      UNAME: HP-UX tschi2 B.11.23 U ia64 1652468500 unlimited-user license"
       
      "java version ""1.6.0.04""
      Java(TM) SE Runtime Environment (build 1.6.0.04-jinteg_28_apr_2009_04_46-b00)
      Java HotSpot(TM) Server VM (build 11.3-b02-jre1.6.0.04-rc2, mixed mode)"
      ---
      HP(PA-RISC)
      HOSTID: E0DD7C9E
      NCPUS: 2
      OPENFILES: 8192/10240
      UNAME: HP-UX tschp2 B.11.23 U 9000/800 3772611742 unlimited-user license"
       
      "java version ""1.6.0.04""
      Java(TM) SE Runtime Environment (build 1.6.0.04-jinteg_28_apr_2009_04_46-b00)
      Java HotSpot(TM) Server VM (build 11.3-b02-jre1.6.0.04-rc2 PA2.0 (aCC_AP), mixed mode)"
      ---

      I don't understand the following result. I tested it in both grizzly-1.9.16 and grizzly-1.9.17.

      Here is a sample result.

      [In Windows]
      ---
      Number of hits: 285010
      Requests per Second: 2373.28

      Socket Statistics
      --------------------------------------------------------------------------------
      Socket Connects: 285035
      Total Bytes Sent (in KB): 56227.61
      Bytes Sent Rate (in KB/s): 468.21
      Total Bytes Recv (in KB): 312008.02
      Bytes Recv Rate (in KB/s): 2598.09
      ---

      [In HP]
      ---
      Number of hits: 1316
      Requests per Second: 10.97

      Socket Statistics
      --------------------------------------------------------------------------------
      Socket Connects: 1341
      Total Bytes Sent (in KB): 265.84
      Bytes Sent Rate (in KB/s): 2.22
      Total Bytes Recv (in KB): 1335.58
      Bytes Recv Rate (in KB/s): 11.13
      ---

      Has anybody met this problem?

      I think that the Grizzly QA may do the performance test regularly. Do you happen to test on HP-UX?

      Please advice me.

      Thanks.

      --
      Bongjae Chang