Thanks Oleksly,
I've increased max-thread-pool-size to 2000, the previous value was 500. I
don't know why there are so many "beware of context switches if having too
many thread warnings".
It is important to increase linux sockets to max value $ulimit -n is 256000
to me.
Also, I've noticed that there is
selector-poll-timeout-millis
1000ms default value seems to be good, 400ms would be enough for Europe and
USA, but for users in Africa 1000ms might be needed.
Anyway, hope same high crazy limits will solve my problem - I did write
this email mostly for the archive completness in the case someone google's
that later.
Thanks again.
On Fri, Oct 25, 2013 at 12:28 AM, Oleksiy Stashok <
oleksiy.stashok_at_oracle.com> wrote:
> Hi Mladen,
>
> I don't think it's JDK bug, most probably client <-> server connection is
> pretty slow and the client is not able to receive data as fast as server
> sends it, so server has to block a thread and wait until the client is able
> to receive next portion of data.
>
> There are 2 things you can do:
> 1) Increase the server thread-pool
>
> asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.min-thread-pool-size=newvalue
> asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=newvalue
>
>
> 2) Decrease write timeout value, so TCP connection will be considered as
> closed if write can not be completed during certain period of time. You can
> set it via system property: com.sun.grizzly.writeTimeout. Default value is
> 30 seconds.
>
> Hope that will help.
>
> WBR,
> Alexey.
>
>
>
> On 24.10.13 05:35, Mladen Adamoviæ wrote:
>
> After Google-ing, I reliazed this is most likely JDK related bug. It
> makes our server hang periodically.
>
> Any idea of change of glassfish parameters which which might prevent the
> server from hanging up? (I've seen upgrading JDK to 1.7 might help, which
> we'll do soon).
>
> Many threads endup in this:
>
> Thread 8216: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
> frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> -
> com.sun.grizzly.util.OutputWriter.flushChannel(java.nio.channels.SelectableChannel,
> java.nio.ByteBuffer, long) @bci=161, line=126 (Compiled frame)
> -
> com.sun.grizzly.util.OutputWriter.flushChannel(java.nio.channels.SelectableChannel,
> java.nio.ByteBuffer) @bci=6, line=76 (Compiled frame)
> -
> com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(java.nio.ByteBuffer)
> @bci=117, line=426 (Compiled frame)
> - com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer() @bci=30,
> line=498 (Compiled frame)
> - com.sun.grizzly.http.SocketChannelOutputBuffer.endRequest() @bci=5,
> line=489 (Compiled frame)
> -
> com.sun.grizzly.http.ProcessorTask.action(com.sun.grizzly.tcp.ActionCode,
> java.lang.Object) @bci=155, line=1127 (Compiled frame)
> - com.sun.grizzly.tcp.Response.action(com.sun.grizzly.tcp.ActionCode,
> java.lang.Object) @bci=31, line=268 (Compiled frame)
> - com.sun.grizzly.tcp.Response.finish() @bci=5, line=412 (Compiled frame)
> - org.apache.catalina.connector.OutputBuffer.close() @bci=75, line=341
> (Compiled frame)
> - org.apache.catalina.connector.Response.finishResponse() @bci=4,
> line=582 (Compiled frame)
> -
> org.apache.catalina.connector.CoyoteAdapter.afterService(com.sun.grizzly.tcp.Request,
> com.sun.grizzly.tcp.Response) @bci=38, line=375 (Compiled frame)
> -
> com.sun.enterprise.v3.services.impl.ContainerMapper.afterService(com.sun.grizzly.tcp.Request,
> com.sun.grizzly.tcp.Response) @bci=30, line=483 (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.finishResponse() @bci=17, line=792
> (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.postResponse() @bci=16, line=782
> (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.doProcess() @bci=392, line=758
> (Interpreted frame)
> - com.sun.grizzly.http.ProcessorTask.process(java.io.InputStream,
> java.io.OutputStream) @bci=7, line=1056 (Compiled frame)
> -
> com.sun.grizzly.http.DefaultProtocolFilter.execute(com.sun.grizzly.Context)
> @bci=483, line=229 (Compiled frame)
> -
> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(com.sun.grizzly.Context,
> int) @bci=46, line=137 (Compiled frame)
> - com.sun.grizzly.DefaultProtocolChain.execute(com.sun.grizzly.Context,
> int) @bci=21, line=104 (Compiled frame)
> - com.sun.grizzly.DefaultProtocolChain.execute(com.sun.grizzly.Context)
> @bci=3, line=90 (Interpreted frame)
> - com.sun.grizzly.http.HttpProtocolChain.execute(com.sun.grizzly.Context)
> @bci=63, line=79 (Interpreted frame)
> - com.sun.grizzly.ProtocolChainContextTask.doCall() @bci=13, line=54
> (Compiled frame)
> - com.sun.grizzly.SelectionKeyContextTask.call() @bci=7, line=59
> (Compiled frame)
> - com.sun.grizzly.ContextTask.run() @bci=1, line=71 (Compiled frame)
> - com.sun.grizzly.util.AbstractThreadPool$Worker.doWork() @bci=46,
> line=532 (Compiled frame)
> - com.sun.grizzly.util.AbstractThreadPool$Worker.run() @bci=9, line=513
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 8217: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
> frame)
>
>
>
>