Hi, I am new to Glassfish and Java EE development.
I am using Glassfish V2 b53 with Java 1.6.0_02 b05.
We are currently in the process of choosing a new platform and are looking at Java EE. Before starting any large development project we want to prove to ourself that this is a scalable solution.
I am using a small test application that consists of a few EJBs, all session ones, a connection pool and a jsf web interface. I use openSta to load test the application on a Windows 2003 server.
When number of active users reaches around 400, the server starts to struggle. I start getting connection problems to the database, then EJB methods can't be called, then eventually I cannot connect the the server machine at all. When I stop all the openSTA instances the server still doesn't recover. I have to shut down Glassfish to be able to connect to the machine again.
So it looks like glassfish is locked and holding on to loads of network resources or something.
The stack trace I get are not always the same, but always related to connection issues, normally connection to the database.
[#|2007-07-18T15:44:05.234-0700|INFO|sun-appserver9.1|javax.enterprise.resource.resourceadapter|_ThreadID=17;_ThreadName=httpSSLWorkerThread-8080-1;dual;|RAR5074 : Table based validation detected invalid connection. Querying the table dual failed. Set resource-adapter log-level to FINE for exception stack trace. |#]
[#|2007-07-18T15:44:05.234-0700|SEVERE|sun-appserver9.1|javax.enterprise.resource.resourceadapter|_ThreadID=18;_ThreadName=httpSSLWorkerThread-8080-3;_RequestID=873b07be-c246-463f-afa8-8ea22b956f95;|RAR5031:System Exception
javax.resource.spi.LocalTransactionException: Io exception: Connection reset
at com.sun.gjc.spi.LocalTransaction.commit(LocalTransaction.java:93)
at com.sun.enterprise.resource.ConnectorXAResource.commit(ConnectorXAResource.java:113)
at com.sun.enterprise.distributedtx.J2EETransaction.commit(J2EETransaction.java:462)
at com.sun.enterprise.distributedtx.J2EETransactionManagerOpt.commit(J2EETransactionManagerOpt.java:371)
at com.sun.ejb.containers.BaseContainer.completeNewTx(BaseContainer.java:3781)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:3560)
at com.sun.ejb.containers.StatefulSessionContainer.postInvokeTx(StatefulSessionContainer.java:2714)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1343)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1305)
at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:210)
at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:117)
at $Proxy23.getWallchartData(Unknown Source)
at sun.reflect.GeneratedMethodAccessor329.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:204)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:123)
at com.sun.corba.ee.impl.presentation.rmi.bcel.BCELStubBase.invoke(BCELStubBase.java:196)
at com.retaininternational.remoteejb.__WallchartDataRemote_Remote_DynamicStub.getWallchartData(com/retaininternational/remoteejb/__WallchartDataRemote_Remote_DynamicStub.java)
at com.retaininternational.remoteejb._WallchartDataRemote_Wrapper.getWallchartData(com/retaininternational/remoteejb/_WallchartDataRemote_Wrapper.java)
at com.retain.ws.RtnBkgWSClient.getWallchartData(RtnBkgWSClient.java:109)
at com.retain.component.wallchart.RtnWallchartBean.getWallchartData(RtnWallchartBean.java:74)
at sun.reflect.GeneratedMethodAccessor328.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)
at com.sun.faces.application.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:77)
at org.apache.shale.remoting.impl.MethodBindingProcessor.process(MethodBindingProcessor.java:122)
at org.apache.shale.remoting.faces.RemotingPhaseListener.afterPhase(RemotingPhaseListener.java:114)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:280)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
at com.sun.faces.extensions.avatar.lifecycle.PartialTraversalLifecycle.execute(PartialTraversalLifecycle.java:80)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:244)
at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:266)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:624)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:268)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:631)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:562)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:803)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:339)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:261)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:212)
at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:361)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Then eventually I will get quite a few messages like this:
[#|2007-07-18T15:46:25.171-0700|WARNING|sun-appserver9.1|javax.enterprise.resource.resourceadapter|_ThreadID=19;_ThreadName=Timer-6;Connection could not be allocated because: Io exception: The Network Adapter could not establish the connection;_RequestID=aa0cbd47-ec87-47ef-aaa8-f78ed18d3985;|RAR5038:Unexpected exception while creating resource|#]
Then it stops writing logs and just hangs.
My connection pool settings are:
Oracle 10g database
ojdbc14.jar
oracle.jdbc.pool.OracleDataSource
I have connection validation on, validation method table using DUAL. I have leak timeout and leak reclaim set.
Creation Retry attempts: 20
Retry Interval: 3
I have tested validation and recreation of connections by shutting down my datbase server machine, and that recovers fine, but connection issues after being under load does not.
Any help on this issue would be very much appreciated.
Anymore information I should provide?
Thanks
Hans
[Message sent by forum member 'hansflhotmailcom' (hansflhotmailcom)]
http://forums.java.net/jive/thread.jspa?messageID=227237