users@glassfish.java.net

Glassfish 2.1.1 spawns 2 threads on 1 web service request and throw error

From: <glassfish_at_javadesktop.org>
Date: Fri, 09 Jul 2010 12:39:41 PDT

I have a very strange XML reader error on web service that happens at 1/1000 probability. My web service receives hundreds of requests per hour. This error happens anywhere between 0 to 30 times per day. I thought it was related to xml component. I upgraded GF from 2u2 to 2.1.1, then upgraded metro to 1.5, then upgraded grizzly to 1.0.34. The error message changed from "premature end of file" to "unexpected EOF of prolog" ....... Basically none of them worked.

When this error happens, GF spawns 2 threads for 1 web service request. 1 thread results in XML reader error (EOF or empty stream), the other thread processes the request successfully. User always receives the error as response.

My web service client (written in ASP.net) receives this error response (before I upgrade to metro 1.5, the error was "javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1] Message: Premature end of file."):

<?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode>S:Client</faultcode><faultstring>Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog&#xD;
 at [row,col {unknown-source}]: [1,0]</faultstring></S:Fault></S:Body></S:Envelope>

In GF's http access log, there is only one entry (in which response is 500 server error):
"09/Jul/2010:11:15:28 -0500" "POST /ManagerService/ManagerService HTTP/1.0" 500 428

In admin console's high-level web service message monitoring, there is only one entry (which is successful). I can see both request and response are normal (not the soap fault as above).

In server.log, it is interesting: there are TWO threads servicing this request! Thead 1 (ID=28) is simply the soap error that user receives:

[#|2010-07-09T11:15:33.986-0400|SEVERE|sun-appserver2.1|com.sun.xml.ws.transport.http.HttpAdapter|_ThreadID=28;_ThreadName=httpSSLWorkerThread-80-1;_RequestID=ce570649-17d1-4ac5-a337-54cd37cc3ed9;|Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
 at [row,col {unknown-source}]: [1,0]
com.sun.xml.ws.protocol.soap.MessageCreationException: Couldn't create SOAP message due to exception: XML reader error: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
 at [row,col {unknown-source}]: [1,0]
        at com.sun.xml.ws.encoding.SOAPBindingCodec.decode(SOAPBindingCodec.java:365)
......

Below that record is Thread 2 (ID=32), which is regular Hibernate operations generated by my web service method (in a ejb) code:
[#|2010-07-09T11:15:34.285-0400|INFO|sun-appserver2.1|javax.enterprise.system.stream.out|_ThreadID=32;_ThreadName=httpSSLWorkerThread-80-3;|
Hibernate: select ......

Generally, GF receives 1 request and asks 2 threads to work. Only one succeeds. User receives the failed result while GF web service monitor records the successful result.

I posted the detail error trace at metro board (now i don't believe it is a metro problem): http://forums.java.net/jive/thread.jspa?threadID=151600

There is also another thread from 1 year ago with exactly same problem but nobody has confirmed a solution:
http://forums.java.net/jive/thread.jspa?threadID=63273

Thank you in advance.
[Message sent by forum member 'haibei']

http://forums.java.net/jive/thread.jspa?messageID=477226