Hi Mladen,
I don't think it's JDK bug, most probably client <-> server connection
is pretty slow and the client is not able to receive data as fast as
server sends it, so server has to block a thread and wait until the
client is able to receive next portion of data.
There are 2 things you can do:
1) Increase the server thread-pool
asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=newvalue
asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=newvalue
2) Decrease write timeout value, so TCP connection will be considered as
closed if write can not be completed during certain period of time. You
can set it via system property: com.sun.grizzly.writeTimeout. Default
value is 30 seconds.
Hope that will help.
WBR,
Alexey.
On 24.10.13 05:35, Mladen Adamovic' wrote:
> After Google-ing, I reliazed this is most likely JDK related bug. It
> makes our server hang periodically.
>
> Any idea of change of glassfish parameters which which might prevent
> the server from hanging up? (I've seen upgrading JDK to 1.7 might
> help, which we'll do soon).
>
> Many threads endup in this:
>
> Thread 8216: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> -
> com.sun.grizzly.util.OutputWriter.flushChannel(java.nio.channels.SelectableChannel,
> java.nio.ByteBuffer, long) @bci=161, line=126 (Compiled frame)
> -
> com.sun.grizzly.util.OutputWriter.flushChannel(java.nio.channels.SelectableChannel,
> java.nio.ByteBuffer) @bci=6, line=76 (Compiled frame)
> -
> com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(java.nio.ByteBuffer)
> @bci=117, line=426 (Compiled frame)
> - com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer()
> @bci=30, line=498 (Compiled frame)
> - com.sun.grizzly.http.SocketChannelOutputBuffer.endRequest() @bci=5,
> line=489 (Compiled frame)
> -
> com.sun.grizzly.http.ProcessorTask.action(com.sun.grizzly.tcp.ActionCode,
> java.lang.Object) @bci=155, line=1127 (Compiled frame)
> - com.sun.grizzly.tcp.Response.action(com.sun.grizzly.tcp.ActionCode,
> java.lang.Object) @bci=31, line=268 (Compiled frame)
> - com.sun.grizzly.tcp.Response.finish() @bci=5, line=412 (Compiled frame)
> - org.apache.catalina.connector.OutputBuffer.close() @bci=75,
> line=341 (Compiled frame)
> - org.apache.catalina.connector.Response.finishResponse() @bci=4,
> line=582 (Compiled frame)
> -
> org.apache.catalina.connector.CoyoteAdapter.afterService(com.sun.grizzly.tcp.Request,
> com.sun.grizzly.tcp.Response) @bci=38, line=375 (Compiled frame)
> -
> com.sun.enterprise.v3.services.impl.ContainerMapper.afterService(com.sun.grizzly.tcp.Request,
> com.sun.grizzly.tcp.Response) @bci=30, line=483 (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.finishResponse() @bci=17,
> line=792 (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.postResponse() @bci=16, line=782
> (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.doProcess() @bci=392, line=758
> (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.process(java.io.InputStream,
> java.io.OutputStream) @bci=7, line=1056 (Compiled frame)
> -
> com.sun.grizzly.http.DefaultProtocolFilter.execute(com.sun.grizzly.Context)
> @bci=483, line=229 (Compiled frame)
> -
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(com.sun.grizzly.Context,
> int) @bci=46, line=137 (Compiled frame)
> -
> com.sun.grizzly.DefaultProtocolChain.execute(com.sun.grizzly.Context,
> int) @bci=21, line=104 (Compiled frame)
> -
> com.sun.grizzly.DefaultProtocolChain.execute(com.sun.grizzly.Context)
> @bci=3, line=90 (Interpreted frame)
> -
> com.sun.grizzly.http.HttpProtocolChain.execute(com.sun.grizzly.Context) @bci=63,
> line=79 (Interpreted frame)
> - com.sun.grizzly.ProtocolChainContextTask.doCall() @bci=13, line=54
> (Compiled frame)
> - com.sun.grizzly.SelectionKeyContextTask.call() @bci=7, line=59
> (Compiled frame)
> - com.sun.grizzly.ContextTask.run() @bci=1, line=71 (Compiled frame)
> - com.sun.grizzly.util.AbstractThreadPool$Worker.doWork() @bci=46,
> line=532 (Compiled frame)
> - com.sun.grizzly.util.AbstractThreadPool$Worker.run() @bci=9,
> line=513 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 8217: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65
> (Compiled frame)
>
>