users@glassfish.java.net

Re: Problems with Glassfish <-> Apache mod_proxy_ajp

From: <glassfish_at_javadesktop.org>
Date: Thu, 28 Aug 2008 01:30:22 PDT

Hello Jeanfrancois

our system halted again a few minutes ago.

now we collected some data.

> When that happens, can you grap the output of:
> % netstat -an | grep 8009 to get the state of the connection?
[code]
tcp 0 0 0.0.0.0:8009 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8009 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8009 127.0.0.1:58594 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58575 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58593 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58591 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58503 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58565 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58556 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58577 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58551 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58519 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58590 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58547 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58514 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58595 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58599 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58533 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58513 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58516 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58512 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58515 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58568 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58524 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58511 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58508 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58537 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58566 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58521 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58555 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58539 SYN_RECV
tcp 0 0 127.0.0.1:8009 127.0.0.1:58531 SYN_RECV
tcp 0 1 127.0.0.1:58597 127.0.0.1:8009 SYN_SENT
tcp 0 1 127.0.0.1:58592 127.0.0.1:8009 SYN_SENT
tcp 0 1 127.0.0.1:58570 127.0.0.1:8009 SYN_SENT

tcp 0 0 127.0.0.1:8009 127.0.0.1:35691 VERBUNDEN
tcp 0 0 127.0.0.1:8009 127.0.0.1:42107 VERBUNDEN
tcp 438 0 127.0.0.1:8009 127.0.0.1:58540 VERBUNDEN
tcp 0 0 127.0.0.1:37629 127.0.0.1:8009 VERBUNDEN
tcp 495 0 127.0.0.1:8009 127.0.0.1:58553 VERBUNDEN
[...]
tcp 0 0 127.0.0.1:38667 127.0.0.1:8009 VERBUNDEN
tcp 741 0 127.0.0.1:8009 127.0.0.1:58538 VERBUNDEN
tcp 0 636 127.0.0.1:58524 127.0.0.1:8009 VERBUNDEN
tcp 0 861 127.0.0.1:58521 127.0.0.1:8009 VERBUNDEN
tcp 0 201216 127.0.0.1:8009 127.0.0.1:37080 VERBUNDEN

(522x Verbunden (connected))

[/code]


> Hum...can you get a thread dump of the GF process
> when than happens? I'm surprised GF needs to be restarted. Just do:
> % jstack PID > thread.dump

There are a couple of threads (17) running in SocketWrite. An example
[code]
Thread "TP-Processor1391" thread-id 90.456 thread-stateRUNNABLERunning in native
         at: java.net.SocketOutputStream.socketWrite0(Native Method)
         at: java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
         at: java.net.SocketOutputStream.write(SocketOutputStream.java:136)
         at: org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537)
         at: org.apache.jk.common.JkInputStream.doWrite(JkInputStream.java:162)
         at: org.apache.coyote.Response.doWrite(Response.java:599)
         at: org.apache.coyote.tomcat5.OutputBuffer.realWriteBytes(OutputBuffer.java:407)
         at: org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:417)
         at: org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
         at: org.apache.coyote.tomcat5.OutputBuffer.writeBytes(OutputBuffer.java:437)
         at: org.apache.coyote.tomcat5.OutputBuffer.write(OutputBuffer.java:424)
         at: org.apache.coyote.tomcat5.CoyoteOutputStream.write(CoyoteOutputStream.java:145)
         at: com.xxxxxxxxxxxxx.earth.GoogleEarthServlet.serveLightFile(GoogleEarthServlet.java:282)
         at: com.xxxxxxxxxxxxx.earth.GoogleEarthServlet.doGet(GoogleEarthServlet.java:166)
         at: javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
         at: javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
         at: org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
         at: org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:290)
         at: org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
         at: org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
         at: com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
         at: org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
         at: org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
         at: org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
         at: org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
         at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
         at: org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
         at: org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
         at: org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
         at: org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
         at: org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
         at: org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
         at: org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
         at: org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
         at: org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666)
         at: java.lang.Thread.run(Thread.java:619)
[/code]

> Are you able to get pages serviced when invoking
> GF directly (let's say on port 8080)?

Yes, that works:

wget -O /dev/null http://server.domain:8080/Home.html
--2008-08-28 09:23:01-- http://server.domain:8080/Home.html
Auflösen des Hostnamen »server.domain«.... 127.0.0.1
Verbindungsaufbau zu server.domain|127.0.0.1|:8080... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: nicht spezifiziert [text/html]
In »/dev/null« speichern.


Of course we can send you the complete stacktrace and netstat-output via email.


Thank you and with best regards

Nitram
[Message sent by forum member 'mleuschner' (mleuschner)]

http://forums.java.net/jive/thread.jspa?messageID=295825