I am not sure what HTTP JMX Connector can do here. If you see, all failures happen during receive. And server stack trace #2 gives some idea what that 500 server internal error is: Apparently the JMX HTTP servlet on the server end fails to fetch data. The driver of code for servlet connection from the client side and data sending is done by deployment code. Do you want to double check deployment code for this? Perhaps we can request Web container team too to add more data on the stack traces they see below. [#|2007-04-09T14:21:15.132-0500|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=13;_ThreadName=httpWorkerThread-4848-1;_RequestID=1c1be0a8-73b2-49cd-b017-cb79cf98e7ed;|StandardWrapperValve[RemoteJmxConnectorServlet]: PWC1406: Servlet.service() for servlet RemoteJmxConnectorServlet threw exception java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2228) at java.io.ObjectInputStream$BlockDataInputStream.readUnsignedShort(ObjectInputStream.java:2706) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2764) at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1032) at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:600) at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:789) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1534) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1699) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.readRequestMessage(RemoteJmxConnectorServlet.java:192) > > I am still getting errors, but you may be right that the errors we saw initially were at least in part due to Grizzly issues. > > The first time I ran my test program it worked perfectly and I was excited that maybe the Grizzly changes had resolved the problem. This is the expected output (from three concurrent threads) simply asking the back-end to list all known apps, and this is what the first test run produced: > > No start delay specified; conti > [0]Applications: > [1]Applications: > [2]Applications: > [0] MEjbApp > [0] __JWSappclients > [0] __ejb_container_timer_app > [0] showArgsGUI-client > [0] > [1] MEjbApp > [1] __JWSappclients > [1] __ejb_container_timer_app > [1] showArgsGUI-client > [1] > [2] MEjbApp > [2] __JWSappclients > [2] __ejb_container_timer_app > [2] showArgsGUI-client > [2] > > When I ran the test the second time I got this stack trace below on the client side. Farther below is a stack trace from the server.log file. > > I ran the test a third time, and I got some of the expected output but then the client got into an infinite loop. There were no errors added to the server.log file. > > I ran the test a fourth time and it worked fine. Same for a fifth time. Then the sixth time I got the second client-side trace below with no errors logged on the server side. > > These results are all from a workspace I updated this morning, rebootstrapped and rebuilt. > > Client-side stack trace from run #2: > > javax.enterprise.deploy.spi.exceptions.TargetException: Error getting required modules > at com.sun.enterprise.deployapi.SunDeploymentManager.getModules(SunDeploymentManager.java:380) > at com.sun.enterprise.deployapi.SunDeploymentManager.getAvailableModules(SunDeploymentManager.java:315) > at other.TestJSR88.getApps(TestJSR88.java:188) > at other.TestJSR88.printApps(TestJSR88.java:195) > at other.TestJSR88.performFunction(TestJSR88.java:214) > at other.TestJSR88.access$000(TestJSR88.java:43) > at other.TestJSR88$Runner.run(TestJSR88.java:281) > at java.lang.Thread.run(Thread.java:595) > Caused by: java.lang.reflect.UndeclaredThrowableException > at $Proxy1.getContainee(Unknown Source) > at com.sun.enterprise.deployapi.SunDeploymentManager.getModulesOnATarget(SunDeploymentManager.java:416) > at com.sun.enterprise.deployapi.SunDeploymentManager.getModules(SunDeploymentManager.java:366) > ... 7 more > Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:4848/web1/remotejmx > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:494) > at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1206) > at java.security.AccessController.doPrivileged(Native Method) > at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1200) > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:889) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:122) > at com.sun.enterprise.admin.jmx.remote.comm.MBeanServerMessageConductor.invoke(MBeanServerMessageConductor.java:71) > at com.sun.enterprise.admin.jmx.remote.internal.RemoteMBeanServerConnection.invoke(RemoteMBeanServerConnection.java:408) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeTarget(AMXProxyHandler.java:442) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeSingleProxyGetter(AMXProxyHandler.java:498) > at com.sun.appserv.management.client.handler.AMXProxyHandler._invoke(AMXProxyHandler.java:1071) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invoke(AMXProxyHandler.java:1002) > ... 10 more > Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:4848/web1/remotejmx > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1153) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:367) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:114) > ... 16 more > javax.enterprise.deploy.spi.exceptions.TargetException: Error getting required modules > at com.sun.enterprise.deployapi.SunDeploymentManager.getModules(SunDeploymentManager.java:380) > at com.sun.enterprise.deployapi.SunDeploymentManager.getAvailableModules(SunDeploymentManager.java:315) > at other.TestJSR88.getApps(TestJSR88.java:188) > at other.TestJSR88.printApps(TestJSR88.java:195) > at other.TestJSR88.performFunction(TestJSR88.java:214) > at other.TestJSR88.access$000(TestJSR88.java:43) > at other.TestJSR88$Runner.run(TestJSR88.java:281) > at java.lang.Thread.run(Thread.java:595) > Caused by: java.lang.reflect.UndeclaredThrowableException > at $Proxy1.getContainee(Unknown Source) > at com.sun.enterprise.deployapi.SunDeploymentManager.getModulesOnATarget(SunDeploymentManager.java:416) > at com.sun.enterprise.deployapi.SunDeploymentManager.getModules(SunDeploymentManager.java:366) > ... 7 more > Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:4848/web1/remotejmx > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:494) > at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1206) > at java.security.AccessController.doPrivileged(Native Method) > at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1200) > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:889) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:122) > at com.sun.enterprise.admin.jmx.remote.comm.MBeanServerMessageConductor.invoke(MBeanServerMessageConductor.java:71) > at com.sun.enterprise.admin.jmx.remote.internal.RemoteMBeanServerConnection.invoke(RemoteMBeanServerConnection.java:408) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeTarget(AMXProxyHandler.java:442) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeSingleProxyGetter(AMXProxyHandler.java:498) > at com.sun.appserv.management.client.handler.AMXProxyHandler._invoke(AMXProxyHandler.java:1071) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invoke(AMXProxyHandler.java:1002) > ... 10 more > Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:4848/web1/remotejmx > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1153) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:367) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:114) > ... 16 more > [1] MEjbApp > [1] __JWSappclients > [1] __ejb_container_timer_app > [1] showArgsGUI-client > [1] > Server-side stack trace from run #2: > > Here is the server.log output: > [#|2007-04-09T14:21:15.132-0500|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=13;_ThreadName=httpWorkerThread-4848-1;_RequestID=1c1be0a8-73b2-49cd-b017-cb79cf98e7ed;|StandardWrapperValve[RemoteJmxConnectorServlet]: PWC1406: Servlet.service() for servlet RemoteJmxConnectorServlet threw exception > java.io.EOFException > at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2228) > at java.io.ObjectInputStream$BlockDataInputStream.readUnsignedShort(ObjectInputStream.java:2706) > at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2764) > at java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1032) > at java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:600) > at java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:789) > at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1534) > at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1699) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) > at com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.readRequestMessage(RemoteJmxConnectorServlet.java:192) > at com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.processRequest(RemoteJmxConnectorServlet.java:108) > at com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.doPost(RemoteJmxConnectorServlet.java:180) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:818) > at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:398) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277) > at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:258) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:189) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564) > at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:81) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:193) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:611) > at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:564) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:558) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1067) > at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:255) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:618) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:549) > at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:790) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:326) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:248) > at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:199) > at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252) > at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:103) > > > Client-side stack trace from run # 6 (no corresponding server stack trace): > > java.lang.IllegalStateException > at com.sun.enterprise.deployapi.SunDeploymentManager.getTargets(SunDeploymentManager.java:215) > at other.TestJSR88.getTargets(TestJSR88.java:180) > at other.TestJSR88.getApps(TestJSR88.java:184) > at other.TestJSR88.printApps(TestJSR88.java:195) > at other.TestJSR88.performFunction(TestJSR88.java:214) > at other.TestJSR88.access$000(TestJSR88.java:43) > at other.TestJSR88$Runner.run(TestJSR88.java:281) > at java.lang.Thread.run(Thread.java:595) > Caused by: java.lang.reflect.UndeclaredThrowableException > at $Proxy1.getStandaloneServerConfigMap(Unknown Source) > at com.sun.enterprise.deployapi.SunDeploymentManager.getTargets(SunDeploymentManager.java:194) > ... 7 more > Caused by: java.io.IOException > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1332) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1908) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1832) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1908) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1832) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1719) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:124) > at com.sun.enterprise.admin.jmx.remote.comm.MBeanServerMessageConductor.invoke(MBeanServerMessageConductor.java:71) > at com.sun.enterprise.admin.jmx.remote.internal.RemoteMBeanServerConnection.getAttribute(RemoteMBeanServerConnection.java:28 > 4) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeTarget(AMXProxyHandler.java:438) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeProxyMapGetter(AMXProxyHandler.java:632) > at com.sun.appserv.management.client.handler.AMXProxyHandler._invoke(AMXProxyHandler.java:1079) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invoke(AMXProxyHandler.java:1002) > ... 9 more > java.lang.IllegalStateException > at com.sun.enterprise.deployapi.SunDeploymentManager.getTargets(SunDeploymentManager.java:215) > at other.TestJSR88.getTargets(TestJSR88.java:180) > at other.TestJSR88.getApps(TestJSR88.java:184) > at other.TestJSR88.printApps(TestJSR88.java:195) > at other.TestJSR88.performFunction(TestJSR88.java:214) > at other.TestJSR88.access$000(TestJSR88.java:43) > at other.TestJSR88$Runner.run(TestJSR88.java:281) > at java.lang.Thread.run(Thread.java:595) > Caused by: java.lang.reflect.UndeclaredThrowableException > at $Proxy1.getStandaloneServerConfigMap(Unknown Source) > at com.sun.enterprise.deployapi.SunDeploymentManager.getTargets(SunDeploymentManager.java:194) > ... 7 more > Caused by: java.io.IOException: invalid stream header > at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:764) > at java.io.ObjectInputStream.(ObjectInputStream.java:277) > at com.sun.enterprise.admin.jmx.remote.comm.ServletConnection.receive(ServletConnection.java:123) > at com.sun.enterprise.admin.jmx.remote.comm.MBeanServerMessageConductor.invoke(MBeanServerMessageConductor.java:71) > at com.sun.enterprise.admin.jmx.remote.internal.RemoteMBeanServerConnection.getAttribute(RemoteMBeanServerConnection.java:28 > 4) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeTarget(AMXProxyHandler.java:438) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invokeProxyMapGetter(AMXProxyHandler.java:632) > at com.sun.appserv.management.client.handler.AMXProxyHandler._invoke(AMXProxyHandler.java:1079) > at com.sun.appserv.management.client.handler.AMXProxyHandler.invoke(AMXProxyHandler.java:1002) > ... 9 more