I see that your leak-timeout is 120 seconds and max-wait-time is 60
seconds. It is possible that when there is heavy load (and the
connections are leaked), all connections are given out. Further requests
will fail as the wait-timeout is 60 seconds.
Leaked connections will be reclaimed only after 120 seconds.
Can you try a setting such that 2*leak-timeout < max-wait-time
eg: leak-timeout = 25 seconds, max-wait-time is 60 seconds ?
Thanks,
-Jagadish
On Tue, 2009-01-13 at 14:21 +0000, Steve Essery wrote:
> Hi,
>
> If its that simple you should be able to distill it down to a simple
> jsp/servlet and be able to reproduce it outside of the main
> application. If you can that would give the engineers something
> concrete to look at.
>
> The leak detection works by recording the stack when the resource is
> marked as busy by the pool just prior to it being returned to the
> calling code, in the process it creates a Timer which is set to fire
> when the leak timeout has passed. If the connection is closed, as
> part of freeing it and returned to the pool the Timer is canceled and
> the stack information removed from the HashMap its stored in. If the
> connection is closed then you shouldn't see a leak report being
> generated.
>
> You might want to consider what happens if you have multiple threads
> executing the code at the same time.
>
> Regards,
> Steve
>
> Chopra, Rishi wrote:
> > Steve,
> >
> > initializeGUI() --- open two connections with the following code
> > snippet
> >
> > ********************************************************************************************************************
> > connection_= arp3DSObj.getConnection(); ----1
> >
> > connection_.setAutoCommit(false);
> >
> > // Create one for getting fx rates
> >
> > Connection connection = arp3DSObj.getConnection(); ---2
> >
> > connection.setAutoCommit(true);
> >
> >
> > ********************************************************************************************************************
> > This is being to initialize and then the code performs db tasks
> > using connection 1 [db procedure called that are in chained mode]
> > and connection 2 [ loading the static data / retrival procs in
> > unchained mode are called using this]. After the work, it also calls
> > another method to clean up the connections [ i.e close is being
> > called on the both connection objects] . There is no caching of
> > connection done in the code..
> >
> > Thanks & Regards,
> > Rishi Chopra.
> >
> >
> >
> > ____________________________________________________________________
> > From: Steve.Essery_at_Sun.COM [mailto:Steve.Essery_at_Sun.COM]
> > Sent: 13 January 2009 12:48
> > To: users_at_glassfish.dev.java.net
> > Subject: Re: FW: Glassfish Problem
> >
> >
> > So does
> > com.espeed.BR.PaymentProcessing.PaymentIniterface.initializeGUI()
> > close the connection it obtains from the pool?
> >
> > If it doesn't then that is the cause of your leak. Ideally the code
> > should get and close connections as needed, and not cache them. If
> > the code caches a connection then you'd have to size the pool to be
> > big enough to support the number of concurrent clients so each could
> > have its own connection - which defeats the object of using a
> > resource pool which is meant to allow greater reuse of a limited
> > number of resources between a larger number of clients.
> >
> > Regards,
> > Steve
> >
> > Chopra, Rishi wrote:
> > > Hi,
> > >
> > > 1. I have been managed to reproduce this issue in our DEV Environment by
> > > changing the following configurations for Connection pool
> > >
> > > Max Pool Size = 3 Connections
> > > Initial and Minimum Pool Size : 1 Connections
> > > Leak Tracing is on as well.
> > > (Seems a weird way though) and then accessed multiple instances of the
> > > application to reproduce this issue
> > >
> > > 2. Version = Sun Java System Application Server 9.1_02 (build b04-fcs)
> > >
> > > 3. Leak Trace is attached as a screenshot from the console.
> > >
> > > 4. It's a Web Application.
> > >
> > > Thanks & Regards,
> > > Rishi Chopra.
> > >
> > > -----Original Message-----
> > > From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
> > > Sent: 13 January 2009 11:08
> > > To: Chopra, Rishi
> > > Cc: users_at_glassfish.dev.java.net
> > > Subject: RE: Re: FW: Glassfish Problem
> > >
> > > Hi Rishi,
> > > * Do you have a test case to reproduce the issue ?
> > >
> > > https://glassfish.dev.java.net/issues/show_bug.cgi?id=3279
> > > indicated in the forum thread you have posted is fixed in 9.1 b 56 (Its
> > > not actually a leak)
> > >
> > > * what is the version & build of your GlassFish (asadmin version
> > > --verbose) ?
> > > * can you post the leak stack trace from your server.log ?
> > > * What type of application are you using (Web / EJB ) ?
> > >
> > > Thanks,
> > > -Jagadish
> > >
> > >
> > >
> > > On Tue, 2009-01-13 at 10:20 +0000, Chopra, Rishi wrote:
> > >
> > > > Hi,
> > > >
> > > > After setting the leak reclaim on its saying that there is a potential
> > > >
> > >
> > >
> > > > connection leak from with the application.
> > > >
> > > > There are code bits that exist in the system that may be causing this
> > > > issue.
> > > >
> > > > Connection con1 = DbSource.getConnection(); --- on this setAutocommit
> > > > is false [ This one set the chained mode on, therefore is used for
> > > > calling the procedures in the chained mode ]
> > > >
> > > > Connection con2 = DbSource.getConnection(); --- on this setAutocommit
> > > > is true
> > > >
> > > > I was going through this article and seems that my problem has been
> > > > occuring due to this
> > > >
> > > > http://forums.java.net/jive/message.jspa?messageID=224047
> > > >
> > > > Any suggestions.. Struck for a long time on this.
> > > >
> > > > Thanks & Regards,
> > > > Rishi Chopra.
> > > >
> > > > -----Original Message-----
> > > > From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
> > > > Sent: 06 January 2009 17:22
> > > > To: Chopra, Rishi
> > > > Cc: users_at_glassfish.dev.java.net
> > > > Subject: RE: Re: FW: Glassfish Problem
> > > >
> > > >
> > > >
> > > > On Tue, 2009-01-06 at 16:53 +0000, Chopra, Rishi wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > 1) Pool configuration is as below :
> > > > > <jdbc-connection-pool allow-non-component-callers="false"
> > > > > associate-with-thread="false" connection-creation-retry-attempts="0"
> > > > > connection-creation-retry-interval-in-seconds="10"
> > > > > connection-leak-reclaim="true"
> > > > >
> > > > connection-leak-timeout-in-seconds="120"
> > > >
> > > > > connection-validation-method="auto-commit"
> > > > > datasource-classname="com.sybase.jdbc3.jdbc.SybDataSource"
> > > > > fail-all-connections="false" idle-timeout-in-seconds="300"
> > > > > is-connection-validation-required="false"
> > > > > is-isolation-level-guaranteed="false"
> > > > > lazy-connection-association="false"
> > > > >
> > > lazy-connection-enlistment="false"
> > >
> > > > > match-connections="false" max-connection-usage-count="0"
> > > > > max-pool-size="75" max-wait-time-in-millis="60000"
> > > > > name="ARP3_SybaseConnectionPool"
> > > > >
> > > non-transactional-connections="false"
> > >
> > > > > pool-resize-quantity="10" res-type="javax.sql.DataSource"
> > > > > statement-timeout-in-seconds="-1" steady-pool-size="10"
> > > > > validate-atmost-once-period-in-seconds="0"
> > > > >
> > > wrap-jdbc-objects="false">
> > >
> > > > > <description>Primary Connection Pool</description>
> > > > > <property name="datasourceName" value="ARP3_SybDataSource"/>
> > > > > <property name="user" value="ABCDEF"/>
> > > > > <property name="password" value="XXXXX"/>
> > > > > <property name="networkProtocol" value="Tds"/>
> > > > > <property name="portNumber" value="9000"/>
> > > > > <property name="databaseName" value="YYYYY"/>
> > > > > <property name="serverName" value="sybukarprd01"/>
> > > > > </jdbc-connection-pool>
> > > > >
> > > > Steady pool size : 10
> > > > Max pool size : 75
> > > >
> > > > There are two reasons why the pool complains that there are no more
> > > > connections to provide.
> > > > 1) load to the pool is such that all 75 connections are given to
> > > > applications and new requests could not get a connection in a "wait
> > > > time period" of 60 seconds or
> > > > 2) connections are leaked. Do you see any leak trace in server.log ?
> > > > [Though I see leak-reclaim is ON]
> > > >
> > > > Thanks
> > > > -Jagadish
> > > >
> > > > > 2) I don't think so because the application did not reported this
> > > > > issue when it was on Weblogic...
> > > > >
> > > > > This issue has come up only after the migration to glassfish...the
> > > > > application still runs fine on weblogic.
> > > > >
> > > > >
> > > > > Thanks & Regards,
> > > > > Rishi Chopra.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
> > > > > Sent: 06 January 2009 16:01
> > > > > To: users_at_glassfish.dev.java.net
> > > > > Subject: Re: FW: Glassfish Problem
> > > > >
> > > > > I am not sure whether the two set of logs (grizzly & connection
> > > > > pool) are related.
> > > > >
> > > > > w.r.t SybaseConnectionPool :
> > > > > * Can you provide your pool configuration from domain.xml ?
> > > > > * Is it possible that the connections are leaked by your application
> > > > >
> > >
> > >
> > > > > as there are no more connections left in the pool ?
> > > > > You can refer :
> > > > > http://blogs.sun.com/kshitiz/entry/connection_leak_tracing
> > > > > for using leak tracing feature.
> > > > >
> > > > > Thanks,
> > > > > -Jagadish
> > > > >
> > > > >
> > > > >
> > > > > On Tue, 2009-01-06 at 14:48 +0000, Chopra, Rishi wrote:
> > > > >
> > > > > > I have another issue with the connection pool which throws me an
> > > > > > exception and does not allow me reconnect to the app unless the
> > > > > > server
> > > > > >
> > > > > > is restarted.
> > > > > > Could you please help ???
> > > > > >
> > > > > > Error Log is mentioned below :
> > > > > >
> > > > > >
> > > > > > [#|2009-01-06T12:11:30.660+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.system.container.web|_ThreadID=37;_ThreadName=SelectorThre
> > > > > > ad-80;_RequestID=6efe8a2e-b43c-4340-a67d-35366516a5a8;|setTcpNoDel
> > > > > > ay
> > > > > > exception
> > > > > > java.net.SocketException: Invalid argument
> > > > > > at sun.nio.ch.Net.setIntOption0(Native Method)
> > > > > > at sun.nio.ch.Net.setIntOption(Net.java:154)
> > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:406)
> > > > > > at
> > > > > >
> > > > > sun.nio.ch.SocketOptsImpl.setBoolean(SocketOptsImpl.java:38)
> > > > >
> > > > > > at
> > > > > > sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(SocketOptsImpl.java:284)
> > > > > > at
> > > > > >
> > > > > sun.nio.ch.OptionAdaptor.setTcpNoDelay(OptionAdaptor.java:48)
> > > > >
> > > > > > at
> > > > > > sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:268)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketO
> > > > > > pt
> > > > > > io
> > > > > > ns
> > > > > > (SelectorThread.java:1902)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAcce
> > > > > > pt
> > > > > > (S
> > > > > > el
> > > > > > ectorThread.java:1475)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConn
> > > > > > ec
> > > > > > ti
> > > > > > on
> > > > > > (SelectorThread.java:1439)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(S
> > > > > > el
> > > > > > ec
> > > > > > to
> > > > > > rThread.java:1350)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.startListe
> > > > > > ne
> > > > > > r(
> > > > > > Se
> > > > > > lectorThread.java:1284)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpo
> > > > > > in
> > > > > > t(
> > > > > > Se
> > > > > > lectorThread.java:1247)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.run(Select
> > > > > > or
> > > > > > Th
> > > > > > re
> > > > > > ad.java:1223)
> > > > > > |#]
> > > > > >
> > > > > > [#|2009-01-06T12:11:30.661+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.system.container.web|_ThreadID=37;_ThreadName=SelectorThre
> > > > > > ad-80;_RequestID=6efe8a2e-b43c-4340-a67d-35366516a5a8;|setReuseAdd
> > > > > > re
> > > > > > ss
> > > > > > exception
> > > > > > java.net.SocketException: Invalid argument
> > > > > > at sun.nio.ch.Net.setIntOption0(Native Method)
> > > > > > at sun.nio.ch.Net.setIntOption(Net.java:154)
> > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:406)
> > > > > > at
> > > > > >
> > > > > sun.nio.ch.SocketOptsImpl.setBoolean(SocketOptsImpl.java:38)
> > > > >
> > > > > > at
> > > > > > sun.nio.ch.SocketOptsImpl.reuseAddress(SocketOptsImpl.java:156)
> > > > > > at
> > > > > >
> > > > > >
> > > > sun.nio.ch.SocketOptsImpl$IP$TCP.reuseAddress(SocketOptsImpl.java:268)
> > > >
> > > > > > at
> > > > > > sun.nio.ch.OptionAdaptor.setReuseAddress(OptionAdaptor.java:177)
> > > > > > at
> > > > > > sun.nio.ch.SocketAdaptor.setReuseAddress(SocketAdaptor.java:343)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketO
> > > > > > pt
> > > > > > io
> > > > > > ns
> > > > > > (SelectorThread.java:1910)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAcce
> > > > > > pt
> > > > > > (S
> > > > > > el
> > > > > > ectorThread.java:1475)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConn
> > > > > > ec
> > > > > > ti
> > > > > > on
> > > > > > (SelectorThread.java:1439)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(S
> > > > > > el
> > > > > > ec
> > > > > > to
> > > > > > rThread.java:1350)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.startListe
> > > > > > ne
> > > > > > r(
> > > > > > Se
> > > > > > lectorThread.java:1284)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpo
> > > > > > in
> > > > > > t(
> > > > > > Se
> > > > > > lectorThread.java:1247)
> > > > > > at
> > > > > > com.sun.enterprise.web.connector.grizzly.SelectorThread.run(Select
> > > > > > or
> > > > > > Th
> > > > > > re
> > > > > > ad.java:1223)
> > > > > > |#]
> > > > > >
> > > > > > [#|2009-01-06T12:11:31.928+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.resource.resourceadapter|_ThreadID=22;_ThreadName=httpWork
> > > > > > erThread-80-6;ARP3_SybaseConnectionPool;In-use connections equal
> > > > > > max-pool-size and expired max-wait-time. Cannot allocate more
> > > > > > conne
> > > > > > ctions.;_RequestID=1e35fcca-aa23-4ea6-9370-bf52d6f0a317;|RAR5117 :
> > > > > > Failed to obtain/create connection from connection pool [ ARP3_Sy
> > > > > > baseConnectionPool ]. Reason : In-use connections equal
> > > > > > max-pool-size and expired max-wait-time. Cannot allocate more
> > > > > > connections.|# ]
> > > > > >
> > > > > > [#|2009-01-06T12:11:31.928+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.resource.resourceadapter|_ThreadID=25;_ThreadName=httpWork
> > > > > > erThread-80-9;ARP3_SybaseConnectionPool;In-use connections equal
> > > > > > max-pool-size and expired max-wait-time. Cannot allocate more
> > > > > > conne
> > > > > > ctions.;_RequestID=74d7ffa2-cc14-4b01-b1f6-a869c6ea366d;|RAR5117 :
> > > > > > Failed to obtain/create connection from connection pool [ ARP3_Sy
> > > > > > baseConnectionPool ]. Reason : In-use connections equal
> > > > > > max-pool-size and expired max-wait-time. Cannot allocate more
> > > > > > connections.|# ]
> > > > > >
> > > > > > [#|2009-01-06T12:11:31.929+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.resource.resourceadapter|_ThreadID=22;_ThreadName=httpWork
> > > > > > erThread-80-6;Error in allocating a connection. Cause: In-use
> > > > > > connections equal max-pool-size and expired max-wait-time. Cannot
> > > > > > allo
> > > > > >
> > > > > > cate more
> > > > > > connections.;_RequestID=1e35fcca-aa23-4ea6-9370-bf52d6f0a317;|RAR5
> > > > > > 11
> > > > > > 4
> > > > > >
> > > > > :
> > > > >
> > > > > > Error allocating connection : [Error in allocating a connection.
> > > > > > Cause: In-use connections equal max-pool-size and expired
> > > > > > max-wait-time. Cannot allocate more connections.]|#]
> > > > > >
> > > > > > [#|2009-01-06T12:11:31.930+0000|WARNING|sun-appserver9.1|javax.ent
> > > > > > er pr is
> > > > > > e.resource.resourceadapter|_ThreadID=25;_ThreadName=httpWork
> > > > > > erThread-80-9;Error in allocating a connection. Cause: In-use
> > > > > > connections equal max-pool-size and expired max-wait-time. Cannot
> > > > > > allo
> > > > > >
> > > > > > cate more
> > > > > > connections.;_RequestID=74d7ffa2-cc14-4b01-b1f6-a869c6ea366d;|RAR5
> > > > > > 11
> > > > > > 4
> > > > > >
> > > > > :
> > > > >
> > > > > > Error allocating connection : [Error in allocating a connection.
> > > > > > Cause: In-use connections equal max-pool-size and expired
> > > > > > max-wait-time. Cannot allocate more connections.]|#]
> > > > > >
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Rishi Chopra.
> > > > > > http://www.bgcpartners.com
> > > > > > CONFIDENTIAL: This e-mail has been sent to you by one of the BGC
> > > > > > entities (collectively BGC) listed at the following link
> > > > > >
> > > > > http://www.bgcpartners.com/legal/disclaimers/index.html#email_discla
> > > > > im
> > > > > er
> > > > > .
> > > > >
> > > > > > The link contains company and FSA registration numbers. This
> > > > > > e-mail,
> > > > > >
> > > > > > including its contents and attachments, if any, are confidential.
> > > > > > If
> > > > > >
> > > > > > you are not the named recipient please notify the sender and
> > > > > > immediately delete it. You may not disseminate, distribute, or
> > > > > > forward
> > > > > >
> > > > > > this e-mail message or disclose its contents to anybody else.
> > > > > >
> > > > > Copyright and any other intellectual property rights in its contents
> > > > >
> > >
> > >
> > > > > are the sole property of BGC and its affiliates. E-mail transmission
> > > > >
> > >
> > >
> > > > > cannot be guaranteed to be secure or error-free.
> > > > >
> > > > > > The sender therefore does not accept liability for any errors or
> > > > > > omissions in the contents of this message which arise as a result
> > > > > > of
> > > > > >
> > > > > > e-mail transmission. If verification is required please request a
> > > > > >
> > >
> > >
> > > > > > hard-copy version. Although we routinely screen for viruses,
> > > > > > addressees should check this e-mail and any attachments for
> > > > > >
> > > viruses.
> > >
> > > > > > We make no representation or warranty as to the absence of viruses
> > > > > >
> > >
> > >
> > > > > > in
> > > > > >
> > > > > this e-mail or any attachments. Please note that to ensure
> > > > > regulatory compliance and for the protection of our customers and
> > > > > business, we may monitor and read e-mails sent to and from our
> > > > >
> > > server(s).
> > >
> > > > > > The registered offices of the BGC entities are at 1 Churchill
> > > > > > Place,
> > > > > >
> > > > > London, E14 5RD.
> > > > >
> > > > > > For any issues arising from this email please reply to the sender.
> > > > > >
> > >
> > >
> > > > > > The FSA register appears at http://www.fsa.gov.uk/register/.
> > > > > > The FSA regulates the financial services industry in the United
> > > > > > Kingdom and is located at
> > > > > > 25 The North Colonnade, Canary Wharf, London, E14 5HS.
> > > > > >
> > > > > > ------------------------------------------------------------------
> > > > > > --
> > > > > > - To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > > > > > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> > > > > >
> > > > > >
> > > > > --------------------------------------------------------------------
> > > > > - To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > > > > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> > > __________________________________________________________________
> > >
> > > Subject: FW: Connection Leak generated in cs_arp3_loadtest
> > > Date: Tue, 13 Jan 2009 11:14:50 +0000
> > >
> > >
> > >
> > >
> > >
> > >
> > > Picture (Metafile)
> > >
> > >
> > >
> > >
> > > __________________________________________________________________
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> > > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
> > >
> >
>