Hi Todd,
I tried to reproduce the issue, and I see the broken pipe, but can not
make any following request to fail. I've updated the ear (cleared the
web start cache etc) many times, but still no luck.
Can you pls. check, what is the Grizzly version used in the GF? You
can use [1] for this.
Thank you.
WBR,
Alexey.
[1] cat server.log | grep "Grizzly Framework"
On Aug 12, 2010, at 1:18 , Cerwin, Todd wrote:
> As a follow up, I actually restarted the application server and
> removed the text file from the META-INF folder without being able to
> reproduce, but then I added another text file to the same folder and
> the error occurred again.
>
> So you might need to iterate through this test case a couple of times.
>
> Thanks,
>
> -Todd
>
> From: Cerwin, Todd
> Sent: Wednesday, August 11, 2010 3:47 PM
> To: Cerwin, Todd; Oleksiy.Stashok_at_Sun.COM; users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn
> Subject: RE: Grizzly Bug?
>
> Can you confirm that you received the ear file?
>
> Thanks,
>
> -Todd
>
> From: Cerwin, Todd
> Sent: Wednesday, August 11, 2010 3:07 PM
> To: 'Oleksiy.Stashok_at_Sun.COM'; users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn
> Subject: RE: Grizzly Bug?
>
> Ok, I hope the ear makes it over to you and you can reproduce this.
>
> We are running on Glassfish V3.0.1, you probably need a JDBC JNDI
> lookup for jdbc/gf3-starter and a mail resource for mail/gf3-starter
> other than that everything should be standard.
>
> I think I found a way to reproduce this without actually re-building
> the application.
>
> Just try to follow these steps:
>
> 1. Deploy the ear through the admin console
> 2. Enable Web Start
> 3. Launch the appclient from the application action (gf3-
> starter-client-1.0-DEV.jar)
> 4. Close the client app
> 5. Select view endpoint for the “StarterWebServive” wsdl view
> this a couple of times, you can see the generated wsdl end point.
> 6. Make a slight change to something in the .ear package, for
> my test case I just added some bogus text file to the META-INF
> directory and save the archive with the same name
> 7. Undeploy the old version of the ear
> 8. Deploy the new ear with the new content
> 9. Enable Web Start
> 10. Launch the same appclient
> 11. Close the client app
> 12. Now select to view the same end point a few times. It depends
> when you hit the right thread, but the end point is occasionally
> unavailable
>
> Hopefully the first round of testing will reproduce my results.
>
> Thanks,
>
> -Todd
>
>
>
>
> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
> Sent: Wednesday, August 11, 2010 1:19 PM
> To: users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn; Cerwin, Todd
> Subject: Re: Grizzly Bug?
>
> Hi Todd,
>
> is it possible for you to provide the testcase, using which we can
> reproduce the problem. If you don't want to publish the code - you
> can send it to me directly.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On Aug 11, 2010, at 18:42 , Cerwin, Todd wrote:
>
>
> Hello,
>
> I was hoping someone from the Grizzly development team could help us
> troubleshoot a major issue we have been finding after upgrading from
> Glassfish V2 to V3. Basically we install our application as an ear
> module and have all EE functionality working, then launch Web Start
> in which we see certain exceptions related to closing http
> connections and subsequent Web Service requests on that same http
> thread will then fail until an application restart. There are more
> details in this thread and I can send you any server logs that might
> help.
>
> Thanks,
>
> -Todd
>
> Web Start Exception
> [#|2010-08-05T10:30:04.401-0700|SEVERE|oracle-glassfish3.0.1|
> javax
> .enterprise
> .system.container.appclient.org.glassfish.appclient.server.core.jws|
> _ThreadID=37;_ThreadName=Thread-1;|Adapter[/___JWSappclient/
> ___system] s1as/glassfish/modules/org.eclipse.persistence.core.jar
> java.io.IOException: An established connection was aborted by the
> software in your host machine
> at sun.nio.ch.SocketDispatcher.write0(Native Method)
> at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:33)
> at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:106)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushChannel(SocketChannelOutputBuffer.java:321)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushBuffer(SocketChannelOutputBuffer.java:392)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .realWriteBytes(SocketChannelOutputBuffer.java:279)
> at com.sun.grizzly.tcp.http11.InternalOutputBuffer
> $OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:884)
> at
> com
> .sun
> .grizzly
> .tcp
> .http11
> .filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:155)
> at
> com
> .sun
> .grizzly
> .tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:
> 640)
> at
> com.sun.grizzly.tcp.Response.doWrite(Response.java:676)
> at
> com
> .sun
> .grizzly
> .tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:229)
> at
> org
> .glassfish
> .appclient
> .server
> .core
> .jws
> .RestrictedContentAdapter
> .processContent(RestrictedContentAdapter.java:266)
> at
> org
> .glassfish
> .appclient
> .server
> .core
> .jws
> .RestrictedContentAdapter
> .serviceContent(RestrictedContentAdapter.java:226)
> at
> org
> .glassfish
> .appclient
> .server
> .core.jws.AppClientHTTPAdapter.service(AppClientHTTPAdapter.java:151)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
>
> Subsequent Web Service Exceptions
> [#|2010-08-05T10:30:39.135-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=37;_ThreadName=Thread-1;|invocation error on ejb endpoint
> LimsAutomationRunner at /LimsAutomationRunnerService/
> LimsAutomationRunner : null
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:
> 96)
> at
> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
> at
> com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:311)
> at
> com
> .sun
> .xml
> .ws
> .streaming
> .XMLStreamWriterUtil.getOutputStream(XMLStreamWriterUtil.java:97)
> at
> com
> .sun.xml.ws.message.jaxb.JAXBMessage.writePayloadTo(JAXBMessage.java:
> 315)
> at
> com
> .sun
> .xml.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:
> 142)
> at
> com.sun.xml.ws.encoding.StreamSOAPCodec.encode(StreamSOAPCodec.java:
> 109)
> at
> com
> .sun.xml.ws.encoding.SOAPBindingCodec.encode(SOAPBindingCodec.java:
> 278)
> at
> com
> .sun.xml.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:
> 383)
> at com.sun.xml.ws.transport.http.HttpAdapter.access
> $100(HttpAdapter.java:93)
> at com.sun.xml.ws.transport.http.HttpAdapter
> $HttpToolkit.handle(HttpAdapter.java:529)
> at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:288)
> at
> com
> .sun
> .xml
> .ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:
> 143)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:116)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> From: Tim Quinn [mailto:tim.quinn_at_oracle.com]
> Sent: Tuesday, August 10, 2010 5:55 PM
> To: Cerwin, Todd
> Cc: tjquinn_at_dev.java.net; King, Bryan
> Subject: Re: Server Logs
>
> Todd,
>
> I have been swamped lately. I'm sorry for the delay in responding.
>
> The Java Web Start support in GlassFish relies on Grizzly (as you
> know) and I have not worked on Grizzly's internals.
>
> I think at this point the right place for the discussion is the
> grizzly user's list users_at_grizzly.dev.java.net
>
> It would probably be best if you posted the question there yourself
> if you don't mind. I know it must feel a little like the run-
> around, and I don't intend it that way.
>
> I had thought that this was another manifestation of something I've
> seen before (I think I mentioned it earlier in our conversation but
> I'm not sure) but in those cases the Grizzly messages were "broken
> pipe" messages, meaning the remote client (Java Web Start) broke off
> the transfer. This looks different from that, especially
> considering the subsequent web services errors you are seeing.
>
> Thanks for your patience.
>
> - Tim
>
> On Aug 6, 2010, at 10:57 AM, Cerwin, Todd wrote:
>
>
> Tim,
>
> I made efforts to clean up those FileNotFound exceptions on my
> application deploy. I realized that most of the errors were related
> to the web module libraries getting compiled into the web-inf\lib
> folder, so I went ahead and changed all of those to provided
> resources and cleared up all exceptions except the one stack trace
> below. I found there were two different versions of the xstream jar
> referenced (1.2 and 1.2.2), and only version 1.2 made it over into
> the ear\lib folder. After removing reference to version 1.2, that
> exception is now gone and there are no errors on deployment.
>
> I am finding the same HTTP thread that causes the aborted
> connections during WebStart downloads will then fail for all
> subsequent WebService calls.
>
>
> Web Start
> [#|2010-08-06T08:48:19.850-0700|SEVERE|oracle-glassfish3.0.1|
> javax
> .enterprise
> .system.container.appclient.org.glassfish.appclient.server.core.jws|
> _ThreadID=28;_ThreadName=Thread-1;|Adapter[/___JWSappclient/___app/
> omfg-ear-1.0-DEV] lib/beansbinding-1.2.1.jar
> java.io.IOException: An established connection was aborted by the
> software in your host machine
>
> Web Service
> [#|2010-08-06T08:48:37.521-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=28;_ThreadName=Thread-1;|invocation error on ejb endpoint
> SideWinderTask at /SideWinderTaskService/SideWinderTask : null
> java.lang.NullPointerException
>
> Have you had a chance to look into those Grizzly related HTTP
> exceptions?
>
> Thanks,
>
> -Todd
>
>
> [#|2010-08-05T16:04:58.536-0700|WARNING|oracle-glassfish3.0.1|
> org.apache.jasper.runtime.TldScanner|
> _ThreadID=77;_ThreadName=Thread-1;|PWC6351: In TLD scanning, the
> supplied resource file:/C:/glassfishv3/glassfish/domains/domain1/
> applications/omfg-ear-1.0-DEV/xstream-1.2.2.jar does not exist
> java.io.FileNotFoundException: C:\glassfishv3\glassfish\domains
> \domain1\applications\omfg-ear-1.0-DEV\xstream-1.2.2.jar (The system
> cannot find the file specified)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:135)
> at java.util.jar.JarFile.<init>(JarFile.java:72)
> at
> sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:72)
> at
> sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:48)
> at
> sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:80)
> at
> sun
> .net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:
> 104)
> at sun.net.www.protocol.jar.JarURLConnection.getJarFile
> (JarURLConnection.java:71)
> at
> org.apache.jasper.runtime.TldScanner.scanJar(TldScanner.java:364)
> at
> org.apache.jasper.runtime.TldScanner.scanJars(TldScanner.java:607)
> at
> org.apache.jasper.runtime.TldScanner.scanTlds(TldScanner.java:282)
> at
> org.apache.jasper.runtime.TldScanner.onStartup(TldScanner.java:228)
> at
> org
> .apache
> .catalina
> .core
> .StandardContext
> .callServletContainerInitializers(StandardContext.java:5352)
> &n bsp; at com.sun.enterprise.web.WebModule.callServletContainerInitializers
> (WebModule.java:550)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:
> 5263)
> at
> com.sun.enterprise.web.WebModule.start(WebModule.java:499)
> at
> org
> .apache
> .catalina.core.ContainerBase.addChildInternal(ContainerBase.java:928)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:
> 912)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694)
> at
> com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:
> 1947)
> at
> com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:
> 1619)
> at
> com.sun.enterprise.web.WebApplication.start(WebApplication.java:90)
> at
> org.glassfish.internal.data.EngineRef.start(EngineRef.java:126)
> at
> org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:241)
> at
> org
> .glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:
> 236)
> at
> com
> .sun
> .enterprise
> .v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:339)
> at
> com
> .sun
> .enterprise
> .v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:183)
> at
> org
> .glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:
> 272)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $1.execute(CommandRunnerImpl.java:305)
> at
> com
> .sun
> .enterprise
> .v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:320)
> at
> com
> .sun
> .enterprise
> .v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1176)
> at
> com.sun.enterprise.v3.admin.CommandRunnerImpl.access
> $900(CommandRunnerImpl.java:83)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $ExecutionContext.execute(CommandRunnerImpl.java:1235)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $ExecutionContext.execute(CommandRunnerImpl.java:1224)
> at
> com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:
> 365)
> at
> com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:
> 204)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> From: Cerwin, Todd
> Sent: Tuesday, July 27, 2010 2:30 PM
> To: 'tjquinn_at_dev.java.net'
> Subject: RE: Server Logs
>
> Hi Tim,
>
> My application stability did not last very long. I just launched
> the client a few more times, invoked some remote EJBs, and SOAP Web
> Service methods and the Web Service calls started to intermittently
> fail with the exception below.
>
> Interestingly, I attempted to restart the domain and re-deploy the
> application and the Web Service calls continued to fail
> intermittently. What actually did work was to bring down the
> domain, delete the java-web-start contents and re-launch the
> application. This caused my Web Services to run without error
> repeatedly, until I again launched the client via Web Start and now
> they are failing again.
>
> I wonder if some of those resources are getting corrupt…
>
> Attached is the following server logs from the following test case:
>
> 1. Server started with a clear java-web-start folder
> 2. 10 Web Service calls made without any Grizzly related errors
> 3. Web client launched twice
> 4. 10 Web Service calls made with 4 out of 10 service calls
> failing.
>
> Hope you have some ideas for me to test out.
>
> Thanks,
>
> -Todd
>
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=43;_ThreadName=Thread-1;|invocation error on ejb endpoint
> SideWinderTask at /SideWinderTaskService/SideWinderTask : null
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:96)
> at
> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
> at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:
> 311)
> at
> com
> .sun
> .xml
> .ws
> .streaming
> .XMLStreamWriterUtil.getOutputStream(XMLStreamWriterUtil.java:97)
> at
> com
> .sun.xml.ws.message.jaxb.JAXBMessage.writePayloadTo(JAXBMessage.java:
> 315)
> at
> com
> .sun
> .xml.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:
> 142)
> at
> com.sun.xml.ws.encoding.StreamSOAPCodec.encode(StreamSOAPCodec.java:
> 109)
> at
> com
> .sun.xml.ws.encoding.SOAPBindingCodec.encode(SOAPBindingCodec.java:
> 278)
> at
> com
> .sun.xml.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:
> 383)
> at com.sun.xml.ws.transport.http.HttpAdapter.access
> $100(HttpAdapter.java:93)
> at com.sun.xml.ws.transport.http.HttpAdapter
> $HttpToolkit.handle(HttpAdapter.java:529)
> at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:288)
> at
> com
> .sun
> .xml
> .ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:
> 143)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:116)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=39;_ThreadName=Thread-1;|cannot write SOAPFault to the
> HTTP response
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at
> com.sun.xml.messaging.saaj.soap.MessageImpl.writeTo(MessageImpl.java:
> 1319)
> at org.glassfish.webservices.WsUtil.raiseException(WsUtil.java:
> 1359)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:134)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|INFO|oracle-glassfish3.0.1|
> javax.enterprise.system.core.security.com.sun.appserv.security|
> _ThreadID=39;_ThreadName=Thread-1;|Authenticating User: tcerwin_at_illumina.com
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=43;_ThreadName=Thread-1;|cannot write SOAPFault to the
> HTTP response
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at
> com.sun.xml.messaging.saaj.soap.MessageImpl.writeTo(MessageImpl.java:
> 1319)
> at org.glassfish.webservices.WsUtil.raiseException(WsUtil.java:
> 1359)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:134)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|INFO|oracle-glassfish3.0.1|
> javax.enterprise.system.core.security.com.sun.appserv.security|
> _ThreadID=43;_ThreadName=Thread-1;|Authenticating User: tcerwin_at_illumina.com
> |#]
>
>
> From: Cerwin, Todd
> Sent: Tuesday, July 27, 2010 11:02 AM
> To: 'tjquinn_at_dev.java.net'
> Subject: Server Logs
>
> Hi Tim and thanks again for your help investigating this!
>
> Ok, so I tried to replicate the server crash on my development
> application server, which should be identical except its running on
> Windows Server 2003 as compared to Windows Server 2008 and I was
> unable to reproduce the server crash, but I did see a couple of SOAP
> exceptions come up during the Web Start downs.
>
> Since the log output was gigantic with this added vm argument (added
> to help troubleshoot the Grizzly errors) (-
> Dcom.sun.grizzly.enableSnoop=true), I decided to remove it and I was
> unable to get the 2008 server to crash again, but on three Web Start
> launches I did get one that failed.
>
> The server logs I attached is from the following steps:
>
> Start app server
> Deploy application
> Launch three Web Start clients
> Fire a few SOAP Web Service calls
>
> Again, it seems to be stable after removing that vm argument, but
> perhaps something in the log might give you some clues as to why I
> needed to delete that Web Start cache in the first place?
>
> I will try a few more things over the course of the day to test out
> my application as it has been far from stable after the V3 upgrade.
>
> Thanks!
>
> -Todd
>
>
>
>
>
>