Hi Todd,
it's exactly what I did several times :)
That's why I'm asking to check the Grizzly version to make sure we're
using the same thing.
Thanks.
WBR,
Alexey.
On Aug 12, 2010, at 1:18 , Cerwin, Todd wrote:
> As a follow up, I actually restarted the application server and
> removed the text file from the META-INF folder without being able to
> reproduce, but then I added another text file to the same folder and
> the error occurred again.
>
> So you might need to iterate through this test case a couple of times.
>
> Thanks,
>
> -Todd
>
> From: Cerwin, Todd
> Sent: Wednesday, August 11, 2010 3:47 PM
> To: Cerwin, Todd; Oleksiy.Stashok_at_Sun.COM; users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn
> Subject: RE: Grizzly Bug?
>
> Can you confirm that you received the ear file?
>
> Thanks,
>
> -Todd
>
> From: Cerwin, Todd
> Sent: Wednesday, August 11, 2010 3:07 PM
> To: 'Oleksiy.Stashok_at_Sun.COM'; users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn
> Subject: RE: Grizzly Bug?
>
> Ok, I hope the ear makes it over to you and you can reproduce this.
>
> We are running on Glassfish V3.0.1, you probably need a JDBC JNDI
> lookup for jdbc/gf3-starter and a mail resource for mail/gf3-starter
> other than that everything should be standard.
>
> I think I found a way to reproduce this without actually re-building
> the application.
>
> Just try to follow these steps:
>
> 1. Deploy the ear through the admin console
> 2. Enable Web Start
> 3. Launch the appclient from the application action (gf3-
> starter-client-1.0-DEV.jar)
> 4. Close the client app
> 5. Select view endpoint for the “StarterWebServive” wsdl view
> this a couple of times, you can see the generated wsdl end point.
> 6. Make a slight change to something in the .ear package, for
> my test case I just added some bogus text file to the META-INF
> directory and save the archive with the same name
> 7. Undeploy the old version of the ear
> 8. Deploy the new ear with the new content
> 9. Enable Web Start
> 10. Launch the same appclient
> 11. Close the client app
> 12. Now select to view the same end point a few times. It depends
> when you hit the right thread, but the end point is occasionally
> unavailable
>
> Hopefully the first round of testing will reproduce my results.
>
> Thanks,
>
> -Todd
>
>
>
>
> From: Oleksiy.Stashok_at_Sun.COM [mailto:Oleksiy.Stashok_at_Sun.COM]
> Sent: Wednesday, August 11, 2010 1:19 PM
> To: users_at_grizzly.dev.java.net
> Cc: tjquinn_at_dev.java.net; King, Bryan; Tim Quinn; Cerwin, Todd
> Subject: Re: Grizzly Bug?
>
> Hi Todd,
>
> is it possible for you to provide the testcase, using which we can
> reproduce the problem. If you don't want to publish the code - you
> can send it to me directly.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On Aug 11, 2010, at 18:42 , Cerwin, Todd wrote:
>
>
> Hello,
>
> I was hoping someone from the Grizzly development team could help us
> troubleshoot a major issue we have been finding after upgrading from
> Glassfish V2 to V3. Basically we install our application as an ear
> module and have all EE functionality working, then launch Web Start
> in which we see certain exceptions related to closing http
> connections and subsequent Web Service requests on that same http
> thread will then fail until an application restart. There are more
> details in this thread and I can send you any server logs that might
> help.
>
> Thanks,
>
> -Todd
>
> Web Start Exception
> [#|2010-08-05T10:30:04.401-0700|SEVERE|oracle-glassfish3.0.1|
> javax
> .enterprise
> .system.container.appclient.org.glassfish.appclient.server.core.jws|
> _ThreadID=37;_ThreadName=Thread-1;|Adapter[/___JWSappclient/
> ___system] s1as/glassfish/modules/org.eclipse.persistence.core.jar
> java.io.IOException: An established connection was aborted by the
> software in your host machine
> at sun.nio.ch.SocketDispatcher.write0(Native Method)
> at
> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:33)
> at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
> at sun.nio.ch.IOUtil.write(IOUtil.java:71)
> at
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:106)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:74)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushChannel(SocketChannelOutputBuffer.java:321)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .flushBuffer(SocketChannelOutputBuffer.java:392)
> at
> com
> .sun
> .grizzly
> .http
> .SocketChannelOutputBuffer
> .realWriteBytes(SocketChannelOutputBuffer.java:279)
> at com.sun.grizzly.tcp.http11.InternalOutputBuffer
> $OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:884)
> at
> com
> .sun
> .grizzly
> .tcp
> .http11
> .filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:155)
> at
> com
> .sun
> .grizzly
> .tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:
> 640)
> at
> com.sun.grizzly.tcp.Response.doWrite(Response.java:676)
> at
> com
> .sun
> .grizzly
> .tcp.StaticResourcesAdapter.service(StaticResourcesAdapter.java:229)
> at
> org
> .glassfish
> .appclient
> .server
> .core
> .jws
> .RestrictedContentAdapter
> .processContent(RestrictedContentAdapter.java:266)
> at
> org
> .glassfish
> .appclient
> .server
> .core
> .jws
> .RestrictedContentAdapter
> .serviceContent(RestrictedContentAdapter.java:226)
> at
> org
> .glassfish
> .appclient
> .server
> .core.jws.AppClientHTTPAdapter.service(AppClientHTTPAdapter.java:151)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
>
> Subsequent Web Service Exceptions
> [#|2010-08-05T10:30:39.135-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=37;_ThreadName=Thread-1;|invocation error on ejb endpoint
> LimsAutomationRunner at /LimsAutomationRunnerService/
> LimsAutomationRunner : null
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:
> 96)
> at
> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
> at
> com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:311)
> at
> com
> .sun
> .xml
> .ws
> .streaming
> .XMLStreamWriterUtil.getOutputStream(XMLStreamWriterUtil.java:97)
> at
> com
> .sun.xml.ws.message.jaxb.JAXBMessage.writePayloadTo(JAXBMessage.java:
> 315)
> at
> com
> .sun
> .xml.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:
> 142)
> at
> com.sun.xml.ws.encoding.StreamSOAPCodec.encode(StreamSOAPCodec.java:
> 109)
> at
> com
> .sun.xml.ws.encoding.SOAPBindingCodec.encode(SOAPBindingCodec.java:
> 278)
> at
> com
> .sun.xml.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:
> 383)
> at com.sun.xml.ws.transport.http.HttpAdapter.access
> $100(HttpAdapter.java:93)
> at com.sun.xml.ws.transport.http.HttpAdapter
> $HttpToolkit.handle(HttpAdapter.java:529)
> at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:288)
> at
> com
> .sun
> .xml
> .ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:
> 143)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:116)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> From: Tim Quinn [mailto:tim.quinn_at_oracle.com]
> Sent: Tuesday, August 10, 2010 5:55 PM
> To: Cerwin, Todd
> Cc: tjquinn_at_dev.java.net; King, Bryan
> Subject: Re: Server Logs
>
> Todd,
>
> I have been swamped lately. I'm sorry for the delay in responding.
>
> The Java Web Start support in GlassFish relies on Grizzly (as you
> know) and I have not worked on Grizzly's internals.
>
> I think at this point the right place for the discussion is the
> grizzly user's list users_at_grizzly.dev.java.net
>
> It would probably be best if you posted the question there yourself
> if you don't mind. I know it must feel a little like the run-
> around, and I don't intend it that way.
>
> I had thought that this was another manifestation of something I've
> seen before (I think I mentioned it earlier in our conversation but
> I'm not sure) but in those cases the Grizzly messages were "broken
> pipe" messages, meaning the remote client (Java Web Start) broke off
> the transfer. This looks different from that, especially
> considering the subsequent web services errors you are seeing.
>
> Thanks for your patience.
>
> - Tim
>
> On Aug 6, 2010, at 10:57 AM, Cerwin, Todd wrote:
>
>
> Tim,
>
> I made efforts to clean up those FileNotFound exceptions on my
> application deploy. I realized that most of the errors were related
> to the web module libraries getting compiled into the web-inf\lib
> folder, so I went ahead and changed all of those to provided
> resources and cleared up all exceptions except the one stack trace
> below. I found there were two different versions of the xstream jar
> referenced (1.2 and 1.2.2), and only version 1.2 made it over into
> the ear\lib folder. After removing reference to version 1.2, that
> exception is now gone and there are no errors on deployment.
>
> I am finding the same HTTP thread that causes the aborted
> connections during WebStart downloads will then fail for all
> subsequent WebService calls.
>
>
> Web Start
> [#|2010-08-06T08:48:19.850-0700|SEVERE|oracle-glassfish3.0.1|
> javax
> .enterprise
> .system.container.appclient.org.glassfish.appclient.server.core.jws|
> _ThreadID=28;_ThreadName=Thread-1;|Adapter[/___JWSappclient/___app/
> omfg-ear-1.0-DEV] lib/beansbinding-1.2.1.jar
> java.io.IOException: An established connection was aborted by the
> software in your host machine
>
> Web Service
> [#|2010-08-06T08:48:37.521-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=28;_ThreadName=Thread-1;|invocation error on ejb endpoint
> SideWinderTask at /SideWinderTaskService/SideWinderTask : null
> java.lang.NullPointerException
>
> Have you had a chance to look into those Grizzly related HTTP
> exceptions?
>
> Thanks,
>
> -Todd
>
>
> [#|2010-08-05T16:04:58.536-0700|WARNING|oracle-glassfish3.0.1|
> org.apache.jasper.runtime.TldScanner|
> _ThreadID=77;_ThreadName=Thread-1;|PWC6351: In TLD scanning, the
> supplied resource file:/C:/glassfishv3/glassfish/domains/domain1/
> applications/omfg-ear-1.0-DEV/xstream-1.2.2.jar does not exist
> java.io.FileNotFoundException: C:\glassfishv3\glassfish\domains
> \domain1\applications\omfg-ear-1.0-DEV\xstream-1.2.2.jar (The system
> cannot find the file specified)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:135)
> at java.util.jar.JarFile.<init>(JarFile.java:72)
> at
> sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:72)
> at
> sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:48)
> at
> sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:80)
> at
> sun
> .net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:
> 104)
> at sun.net.www.protocol.jar.JarURLConnection.getJarFile
> (JarURLConnection.java:71)
> at
> org.apache.jasper.runtime.TldScanner.scanJar(TldScanner.java:364)
> at
> org.apache.jasper.runtime.TldScanner.scanJars(TldScanner.java:607)
> at
> org.apache.jasper.runtime.TldScanner.scanTlds(TldScanner.java:282)
> at
> org.apache.jasper.runtime.TldScanner.onStartup(TldScanner.java:228)
> at
> org
> .apache
> .catalina
> .core
> .StandardContext
> .callServletContainerInitializers(StandardContext.java:5352)
> &n bsp; at com.sun.enterprise.web.WebModule.callServletContainerInitializers
> (WebModule.java:550)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:
> 5263)
> at
> com.sun.enterprise.web.WebModule.start(WebModule.java:499)
> at
> org
> .apache
> .catalina.core.ContainerBase.addChildInternal(ContainerBase.java:928)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:
> 912)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:694)
> at
> com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:
> 1947)
> at
> com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:
> 1619)
> at
> com.sun.enterprise.web.WebApplication.start(WebApplication.java:90)
> at
> org.glassfish.internal.data.EngineRef.start(EngineRef.java:126)
> at
> org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:241)
> at
> org
> .glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:
> 236)
> at
> com
> .sun
> .enterprise
> .v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:339)
> at
> com
> .sun
> .enterprise
> .v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:183)
> at
> org
> .glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:
> 272)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $1.execute(CommandRunnerImpl.java:305)
> at
> com
> .sun
> .enterprise
> .v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:320)
> at
> com
> .sun
> .enterprise
> .v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1176)
> at
> com.sun.enterprise.v3.admin.CommandRunnerImpl.access
> $900(CommandRunnerImpl.java:83)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $ExecutionContext.execute(CommandRunnerImpl.java:1235)
> at com.sun.enterprise.v3.admin.CommandRunnerImpl
> $ExecutionContext.execute(CommandRunnerImpl.java:1224)
> at
> com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:
> 365)
> at
> com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:
> 204)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:
> 69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> From: Cerwin, Todd
> Sent: Tuesday, July 27, 2010 2:30 PM
> To: 'tjquinn_at_dev.java.net'
> Subject: RE: Server Logs
>
> Hi Tim,
>
> My application stability did not last very long. I just launched
> the client a few more times, invoked some remote EJBs, and SOAP Web
> Service methods and the Web Service calls started to intermittently
> fail with the exception below.
>
> Interestingly, I attempted to restart the domain and re-deploy the
> application and the Web Service calls continued to fail
> intermittently. What actually did work was to bring down the
> domain, delete the java-web-start contents and re-launch the
> application. This caused my Web Services to run without error
> repeatedly, until I again launched the client via Web Start and now
> they are failing again.
>
> I wonder if some of those resources are getting corrupt…
>
> Attached is the following server logs from the following test case:
>
> 1. Server started with a clear java-web-start folder
> 2. 10 Web Service calls made without any Grizzly related errors
> 3. Web client launched twice
> 4. 10 Web Service calls made with 4 out of 10 service calls
> failing.
>
> Hope you have some ideas for me to test out.
>
> Thanks,
>
> -Todd
>
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=43;_ThreadName=Thread-1;|invocation error on ejb endpoint
> SideWinderTask at /SideWinderTaskService/SideWinderTask : null
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:96)
> at
> com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
> at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:
> 311)
> at
> com
> .sun
> .xml
> .ws
> .streaming
> .XMLStreamWriterUtil.getOutputStream(XMLStreamWriterUtil.java:97)
> at
> com
> .sun.xml.ws.message.jaxb.JAXBMessage.writePayloadTo(JAXBMessage.java:
> 315)
> at
> com
> .sun
> .xml.ws.message.AbstractMessageImpl.writeTo(AbstractMessageImpl.java:
> 142)
> at
> com.sun.xml.ws.encoding.StreamSOAPCodec.encode(StreamSOAPCodec.java:
> 109)
> at
> com
> .sun.xml.ws.encoding.SOAPBindingCodec.encode(SOAPBindingCodec.java:
> 278)
> at
> com
> .sun.xml.ws.transport.http.HttpAdapter.encodePacket(HttpAdapter.java:
> 383)
> at com.sun.xml.ws.transport.http.HttpAdapter.access
> $100(HttpAdapter.java:93)
> at com.sun.xml.ws.transport.http.HttpAdapter
> $HttpToolkit.handle(HttpAdapter.java:529)
> at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:288)
> at
> com
> .sun
> .xml
> .ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:
> 143)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:116)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=39;_ThreadName=Thread-1;|cannot write SOAPFault to the
> HTTP response
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at
> com.sun.xml.messaging.saaj.soap.MessageImpl.writeTo(MessageImpl.java:
> 1319)
> at org.glassfish.webservices.WsUtil.raiseException(WsUtil.java:
> 1359)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:134)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|INFO|oracle-glassfish3.0.1|
> javax.enterprise.system.core.security.com.sun.appserv.security|
> _ThreadID=39;_ThreadName=Thread-1;|Authenticating User: tcerwin_at_illumina.com
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|WARNING|oracle-glassfish3.0.1|
> javax.enterprise.webservices.org.glassfish.webservices|
> _ThreadID=43;_ThreadName=Thread-1;|cannot write SOAPFault to the
> HTTP response
> java.lang.NullPointerException
> at
> com
> .sun
> .grizzly
> .tcp.http11.GrizzlyOutputStream.write(GrizzlyOutputStream.java:128)
> at
> com
> .sun
> .grizzly
> .http
> .servlet.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:
> 84)
> at
> com.sun.xml.messaging.saaj.soap.MessageImpl.writeTo(MessageImpl.java:
> 1319)
> at org.glassfish.webservices.WsUtil.raiseException(WsUtil.java:
> 1359)
> at
> org
> .glassfish
> .webservices
> .Ejb3MessageDispatcher.handlePost(Ejb3MessageDispatcher.java:134)
> at
> org
> .glassfish
> .webservices.Ejb3MessageDispatcher.invoke(Ejb3MessageDispatcher.java:
> 87)
> at
> org
> .glassfish
> .webservices
> .EjbWebServiceServlet
> .dispatchToEjbEndpoint(EjbWebServiceServlet.java:196)
> at
> org
> .glassfish
> .webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:
> 127)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at
> com
> .sun
> .grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:
> 195)
> at
> com
> .sun
> .grizzly
> .http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:
> 139)
> at
> com
> .sun
> .grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:
> 376)
> at
> com
> .sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:
> 329)
> at
> com
> .sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:
> 166)
> at
> com
> .sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:
> 100)
> at
> com
> .sun
> .enterprise
> .v3.services.impl.ContainerMapper.service(ContainerMapper.java:245)
> at
> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:
> 791)
> at
> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:693)
> at
> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:954)
> at
> com
> .sun
> .grizzly
> .http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:170)
> at
> com
> .sun
> .grizzly
> .DefaultProtocolChain
> .executeProtocolFilter(DefaultProtocolChain.java:135)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 102)
> at
> com
> .sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
> 88)
> at
> com
> .sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
> at
> com
> .sun
> .grizzly
> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.doWork(AbstractThreadPool.java:330)
> at com.sun.grizzly.util.AbstractThreadPool
> $Worker.run(AbstractThreadPool.java:309)
> at java.lang.Thread.run(Thread.java:619)
> |#]
>
> [#|2010-07-27T14:07:03.222-0700|INFO|oracle-glassfish3.0.1|
> javax.enterprise.system.core.security.com.sun.appserv.security|
> _ThreadID=43;_ThreadName=Thread-1;|Authenticating User: tcerwin_at_illumina.com
> |#]
>
>
> From: Cerwin, Todd
> Sent: Tuesday, July 27, 2010 11:02 AM
> To: 'tjquinn_at_dev.java.net'
> Subject: Server Logs
>
> Hi Tim and thanks again for your help investigating this!
>
> Ok, so I tried to replicate the server crash on my development
> application server, which should be identical except its running on
> Windows Server 2003 as compared to Windows Server 2008 and I was
> unable to reproduce the server crash, but I did see a couple of SOAP
> exceptions come up during the Web Start downs.
>
> Since the log output was gigantic with this added vm argument (added
> to help troubleshoot the Grizzly errors) (-
> Dcom.sun.grizzly.enableSnoop=true), I decided to remove it and I was
> unable to get the 2008 server to crash again, but on three Web Start
> launches I did get one that failed.
>
> The server logs I attached is from the following steps:
>
> Start app server
> Deploy application
> Launch three Web Start clients
> Fire a few SOAP Web Service calls
>
> Again, it seems to be stable after removing that vm argument, but
> perhaps something in the log might give you some clues as to why I
> needed to delete that Web Start cache in the first place?
>
> I will try a few more things over the course of the day to test out
> my application as it has been far from stable after the V3 upgrade.
>
> Thanks!
>
> -Todd
>
>
>
>
>
>