Copying admin_at_gf for help.
Thanks,
-Jagadish
On Thu, 2009-01-15 at 07:25 -0600, Brian Repko wrote:
> Jagadish,
>
> I'll get the domain.xml(s) and logs together. I've noticed issues when the DAS domain.xml and the instance/nodeagent domain.xmls get out of sync. I've also noticed issues where the admin says something different from the xml (enabled is true vs false). I don't have a test case - I have a sample app but it runs on my local cluster. I've tried to setup local as close to the failing cluster as possible but it still works for me. That's why I think its part of the domain.xml or our domains are setup in such a way that there is cross-talk (bad port assignments).
>
> We are using Spring's JmsTemplate as you can see from the stack trace so we are holding on to the ConnectionFactory and the Destination.
>
> I'll send files when I get them.
>
> Brian
>
>
> ----- Original message -----
> From: "Jagadish Prasath Ramu" <Jagadish.Ramu_at_Sun.COM>
> To: users_at_glassfish.dev.java.net
> Date: Thu, 15 Jan 2009 17:19:54 +0530
> Subject: Re: NPEs during PoolManagerImpl.getResourceFromPool with Connector Connection Pools
>
> Hi Brian,
> Can you post your domain.xml snapshot & server.log from DAS & cluster
> instances when you face the issue [FINEST for "connectors" &
> "resource-adapter" module log levels] ?
> If you have exact steps to reproduce the issue (or a test-case), that
> would be helpful.
>
> * Somehow the pool is removed (via delete-connector-resource,
> delete-connector-connection-pool or disable connector resource). So, the
> NPE is an after effect.
> * Does your application cache the connection factory ? (eg: as an
> instance variable and is used during a web request after the
> resource/pool is removed)
>
> Thanks,
> -Jagadish
>
>
>
> On Wed, 2009-01-14 at 15:30 -0600, Brian Repko wrote:
> >
> > Folks,
> >
> > Looking for some insight on this one. I'm using the sun-jms-adapter
> > as our connector to OpenMQ.
> > We are SunAS 9.1 ur2 in a clustered environment. At some point, we
> > started getting NPEs when
> > accessing our JMS objects (ConnectionFactories in particular). This
> > gets reported as as the following:
> >
> > [#|2009-01-14T08:11:40.942-0600|WARNING|sun-appserver9.1|
> > javax.enterprise.system.stream.err|
> > _ThreadID=2066;_ThreadName=httpSSLWorkerThread-50011-1;_RequestID=49b9cf44-0580-494d-93b2-f175a7edd24a;|.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:153)
> > at
> > com.stc.jmsjca.core.JConnection.createSession(JConnection.java:314)
> > at
> > com.stc.jmsjca.core.WConnection.createSession(WConnection.java:94)
> > at sun.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > org.springframework.jms.connection.TransactionAwareConnectionFactoryProxy$TransactionAwareConnectionInvocationHandler.invoke(TransactionAwareConnectionFactoryProxy.java:258)
> > at $Proxy554.createSession(Unknown Source)
> > at
> > org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> > at
> > org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> > ... 165 more
> > Caused by: java.lang.NullPointerException
> >
> > After some investigation into the actual cause of the NPE (yes,
> > printing that stack trace would have been helpful),
> > I find the following:
> >
> > Caused by: java.lang.NullPointerException
> > at
> > com.sun.enterprise.resource.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:248)
> > at
> > com.sun.enterprise.resource.PoolManagerImpl.getResource(PoolManagerImpl.java:176)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:337)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:189)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
> > at
> > com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
> > at
> > com.stc.jmsjca.core.JConnection.createSessionByApplication(JConnection.java:131)
> > ... 36 more
> >
> > So its not the adapter - its the server. This code looks like the
> > following:
> >
> > public ResourceHandle getResourceFromPool(ResourceSpec spec,
> > ResourceAllocator alloc, ClientSecurityInfo info, Transaction tran)
> > throws PoolingException {
> > ResourcePool pool = getPool( spec.getConnectionPoolName() );
> > // pool.getResource() has been modified to:
> > // - be able to create new resource if needed
> > // - block the caller until a resource is acquired or
> > // the max-wait-time expires
> > return pool.getResource(spec, alloc, tran); <<<<< this is
> > line 248
> > }
> >
> > So clearly, getPool returns null and this code never checks for that.
> >
> > I've tried rebooting the cluster (which for us is stop/start instance1
> > and then stop/start instance2) and I've tried rebooting
> > the DAS. I also believe that a full stop on the cluster (both
> > instances) has been done as well. I now get the NPEs even
> > after all the reboots.
> >
> > I've turned on logging (FINEST) on the RAR and for Connectors and I
> > have also turned on Monitoring for Connectors
> > and have the logs for those startups. Monitoring doesn't seem to work
> > for this pool so that was pointless. And the logs
> > show all kinds of pools getting added but then removed with a line
> > about them being empty. We do have the pools
> > set to a steady state of 0 since the RAR actually pools the
> > connections/sessions. Its like they get created to deal with JNDI
> > and then removed for some reason after that.
> >
> > Things seem to be bound to the JNDI but the internal pool is not
> > there. I'm not sure how to get around this.
> > I've thought about adding a JSP to a web app that gets the pool table
> > and printing it out just to see.
> >
> > I've also noticed a CVS commit from 3 months ago to PoolManagerImpl
> > (appsrv-core) related to cleaning up resources
> > after transaction commit. I'm nervous that key functionality is not
> > in our version but I don't understand the tags to know
> > which release is which.
> >
> > I put together a sample program and when I went to deploy it, I kept
> > getting JNDI errors. I didn't understand since
> > everything was correct (targets/enabled state). It turned out that my
> > resource-ref was listed as false in the domain.xml
> > but true on the screen. This was easily reproducable by
> > asadmin-creating them into the server and then retargetting
> > them to the cluster and resetting enabled to true. If I add them via
> > the gui directly (no asadmin), then they go in correctly.
> >
> > The common thread is that when enabled=false, I also see the pools
> > getting added and then removed. But the JNDI
> > is not there in one case and seems to still be there in the other
> > case.
> >
> > Looking for clues as to what to do - let me know whatever you might
> > need. I can get it for you. Thanks in advance.
> > At this point I can't get my server to reboot and applications to
> > deploy properly.
> >
> > Brian
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>