users@glassfish.java.net

Re: FW: Glassfish Problem

From: Steve Essery <Steve.Essery_at_Sun.COM>
Date: Tue, 13 Jan 2009 14:21:13 +0000

Hi,

If its that simple you should be able to distill it down to a simple
jsp/servlet and be able to reproduce it outside of the main
application. If you can that would give the engineers something
concrete to look at.

The leak detection works by recording the stack when the resource is
marked as busy by the pool just prior to it being returned to the
calling code, in the process it creates a Timer which is set to fire
when the leak timeout has passed. If the connection is closed, as part
of freeing it and returned to the pool the Timer is canceled and the
stack information removed from the HashMap its stored in. If the
connection is closed then you shouldn't see a leak report being generated.

You might want to consider what happens if you have multiple threads
executing the code at the same time.

Regards,
Steve

Chopra, Rishi wrote:
> Steve,
>
> initializeGUI() --- open two connections with the following code snippet
>
> ********************************************************************************************************************
>
> connection_= arp3DSObj.getConnection(); ----1
>
> connection_.setAutoCommit(*false*);
>
> // Create one for getting fx rates
>
> Connection connection = arp3DSObj.getConnection(); ---2
>
> connection.setAutoCommit(*true*);
>
> ********************************************************************************************************************
> This is being to initialize and then the code performs db tasks using
> connection 1 [db procedure called that are in chained mode] and
> connection 2 [ loading the static data / retrival procs in unchained
> mode are called using this]. After the work, it also calls another
> method to clean up the connections [ i.e close is being called on the
> both connection objects] . There is no caching of connection done in
> the code..
>
> Thanks & Regards,
> Rishi Chopra.
>
>
> ------------------------------------------------------------------------
> *From:* Steve.Essery_at_Sun.COM [mailto:Steve.Essery_at_Sun.COM]
> *Sent:* 13 January 2009 12:48
> *To:* users_at_glassfish.dev.java.net
> *Subject:* Re: FW: Glassfish Problem
>
> So does
> com.espeed.BR.PaymentProcessing.PaymentIniterface.initializeGUI()
> close the connection it obtains from the pool?
>
> If it doesn't then that is the cause of your leak. Ideally the code
> should get and close connections as needed, and not cache them. If
> the code caches a connection then you'd have to size the pool to be
> big enough to support the number of concurrent clients so each could
> have its own connection - which defeats the object of using a resource
> pool which is meant to allow greater reuse of a limited number of
> resources between a larger number of clients.
>
> Regards,
> Steve
>
> Chopra, Rishi wrote:
>> Hi,
>>
>> 1. I have been managed to reproduce this issue in our DEV Environment by
>> changing the following configurations for Connection pool
>>
>> Max Pool Size = 3 Connections
>> Initial and Minimum Pool Size : 1 Connections
>> Leak Tracing is on as well.
>> (Seems a weird way though) and then accessed multiple instances of the
>> application to reproduce this issue
>>
>> 2. Version = Sun Java System Application Server 9.1_02 (build b04-fcs)
>>
>> 3. Leak Trace is attached as a screenshot from the console.
>>
>> 4. It's a Web Application.
>>
>> Thanks & Regards,
>> Rishi Chopra.
>>
>> -----Original Message-----
>> From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
>> Sent: 13 January 2009 11:08
>> To: Chopra, Rishi
>> Cc: users_at_glassfish.dev.java.net
>> Subject: RE: Re: FW: Glassfish Problem
>>
>> Hi Rishi,
>> * Do you have a test case to reproduce the issue ?
>>
>> https://glassfish.dev.java.net/issues/show_bug.cgi?id=3279
>> indicated in the forum thread you have posted is fixed in 9.1 b 56 (Its
>> not actually a leak)
>>
>> * what is the version & build of your GlassFish (asadmin version
>> --verbose) ?
>> * can you post the leak stack trace from your server.log ?
>> * What type of application are you using (Web / EJB ) ?
>>
>> Thanks,
>> -Jagadish
>>
>>
>>
>> On Tue, 2009-01-13 at 10:20 +0000, Chopra, Rishi wrote:
>>
>>> Hi,
>>>
>>> After setting the leak reclaim on its saying that there is a potential
>>>
>>
>>
>>> connection leak from with the application.
>>>
>>> There are code bits that exist in the system that may be causing this
>>> issue.
>>>
>>> Connection con1 = DbSource.getConnection(); --- on this setAutocommit
>>> is false [ This one set the chained mode on, therefore is used for
>>> calling the procedures in the chained mode ]
>>>
>>> Connection con2 = DbSource.getConnection(); --- on this setAutocommit
>>> is true
>>>
>>> I was going through this article and seems that my problem has been
>>> occuring due to this
>>>
>>> http://forums.java.net/jive/message.jspa?messageID=224047
>>>
>>> Any suggestions.. Struck for a long time on this.
>>>
>>> Thanks & Regards,
>>> Rishi Chopra.
>>>
>>> -----Original Message-----
>>> From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
>>> Sent: 06 January 2009 17:22
>>> To: Chopra, Rishi
>>> Cc: users_at_glassfish.dev.java.net
>>> Subject: RE: Re: FW: Glassfish Problem
>>>
>>>
>>>
>>> On Tue, 2009-01-06 at 16:53 +0000, Chopra, Rishi wrote:
>>>
>>>> Hi,
>>>>
>>>> 1) Pool configuration is as below :
>>>> <jdbc-connection-pool allow-non-component-callers="false"
>>>> associate-with-thread="false" connection-creation-retry-attempts="0"
>>>> connection-creation-retry-interval-in-seconds="10"
>>>> connection-leak-reclaim="true"
>>>>
>>> connection-leak-timeout-in-seconds="120"
>>>
>>>> connection-validation-method="auto-commit"
>>>> datasource-classname="com.sybase.jdbc3.jdbc.SybDataSource"
>>>> fail-all-connections="false" idle-timeout-in-seconds="300"
>>>> is-connection-validation-required="false"
>>>> is-isolation-level-guaranteed="false"
>>>> lazy-connection-association="false"
>>>>
>> lazy-connection-enlistment="false"
>>
>>>> match-connections="false" max-connection-usage-count="0"
>>>> max-pool-size="75" max-wait-time-in-millis="60000"
>>>> name="ARP3_SybaseConnectionPool"
>>>>
>> non-transactional-connections="false"
>>
>>>> pool-resize-quantity="10" res-type="javax.sql.DataSource"
>>>> statement-timeout-in-seconds="-1" steady-pool-size="10"
>>>> validate-atmost-once-period-in-seconds="0"
>>>>
>> wrap-jdbc-objects="false">
>>
>>>> <description>Primary Connection Pool</description>
>>>> <property name="datasourceName" value="ARP3_SybDataSource"/>
>>>> <property name="user" value="ABCDEF"/>
>>>> <property name="password" value="XXXXX"/>
>>>> <property name="networkProtocol" value="Tds"/>
>>>> <property name="portNumber" value="9000"/>
>>>> <property name="databaseName" value="YYYYY"/>
>>>> <property name="serverName" value="sybukarprd01"/>
>>>> </jdbc-connection-pool>
>>>>
>>> Steady pool size : 10
>>> Max pool size : 75
>>>
>>> There are two reasons why the pool complains that there are no more
>>> connections to provide.
>>> 1) load to the pool is such that all 75 connections are given to
>>> applications and new requests could not get a connection in a "wait
>>> time period" of 60 seconds or
>>> 2) connections are leaked. Do you see any leak trace in server.log ?
>>> [Though I see leak-reclaim is ON]
>>>
>>> Thanks
>>> -Jagadish
>>>
>>>> 2) I don't think so because the application did not reported this
>>>> issue when it was on Weblogic...
>>>>
>>>> This issue has come up only after the migration to glassfish...the
>>>> application still runs fine on weblogic.
>>>>
>>>>
>>>> Thanks & Regards,
>>>> Rishi Chopra.
>>>>
>>>> -----Original Message-----
>>>> From: Jagadish.Ramu_at_Sun.COM [mailto:Jagadish.Ramu_at_Sun.COM]
>>>> Sent: 06 January 2009 16:01
>>>> To: users_at_glassfish.dev.java.net
>>>> Subject: Re: FW: Glassfish Problem
>>>>
>>>> I am not sure whether the two set of logs (grizzly & connection
>>>> pool) are related.
>>>>
>>>> w.r.t SybaseConnectionPool :
>>>> * Can you provide your pool configuration from domain.xml ?
>>>> * Is it possible that the connections are leaked by your application
>>>>
>>
>>
>>>> as there are no more connections left in the pool ?
>>>> You can refer :
>>>> http://blogs.sun.com/kshitiz/entry/connection_leak_tracing
>>>> for using leak tracing feature.
>>>>
>>>> Thanks,
>>>> -Jagadish
>>>>
>>>>
>>>>
>>>> On Tue, 2009-01-06 at 14:48 +0000, Chopra, Rishi wrote:
>>>>
>>>>> I have another issue with the connection pool which throws me an
>>>>> exception and does not allow me reconnect to the app unless the
>>>>> server
>>>>>
>>>>> is restarted.
>>>>> Could you please help ???
>>>>>
>>>>> Error Log is mentioned below :
>>>>>
>>>>>
>>>>> [#|2009-01-06T12:11:30.660+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.system.container.web|_ThreadID=37;_ThreadName=SelectorThre
>>>>> ad-80;_RequestID=6efe8a2e-b43c-4340-a67d-35366516a5a8;|setTcpNoDel
>>>>> ay
>>>>> exception
>>>>> java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:154)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:406)
>>>>> at
>>>>>
>>>> sun.nio.ch.SocketOptsImpl.setBoolean(SocketOptsImpl.java:38)
>>>>
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl$IP$TCP.noDelay(SocketOptsImpl.java:284)
>>>>> at
>>>>>
>>>> sun.nio.ch.OptionAdaptor.setTcpNoDelay(OptionAdaptor.java:48)
>>>>
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:268)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketO
>>>>> pt
>>>>> io
>>>>> ns
>>>>> (SelectorThread.java:1902)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAcce
>>>>> pt
>>>>> (S
>>>>> el
>>>>> ectorThread.java:1475)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConn
>>>>> ec
>>>>> ti
>>>>> on
>>>>> (SelectorThread.java:1439)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(S
>>>>> el
>>>>> ec
>>>>> to
>>>>> rThread.java:1350)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListe
>>>>> ne
>>>>> r(
>>>>> Se
>>>>> lectorThread.java:1284)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpo
>>>>> in
>>>>> t(
>>>>> Se
>>>>> lectorThread.java:1247)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(Select
>>>>> or
>>>>> Th
>>>>> re
>>>>> ad.java:1223)
>>>>> |#]
>>>>>
>>>>> [#|2009-01-06T12:11:30.661+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.system.container.web|_ThreadID=37;_ThreadName=SelectorThre
>>>>> ad-80;_RequestID=6efe8a2e-b43c-4340-a67d-35366516a5a8;|setReuseAdd
>>>>> re
>>>>> ss
>>>>> exception
>>>>> java.net.SocketException: Invalid argument
>>>>> at sun.nio.ch.Net.setIntOption0(Native Method)
>>>>> at sun.nio.ch.Net.setIntOption(Net.java:154)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl$1.setInt(SocketChannelImpl.java:406)
>>>>> at
>>>>>
>>>> sun.nio.ch.SocketOptsImpl.setBoolean(SocketOptsImpl.java:38)
>>>>
>>>>> at
>>>>> sun.nio.ch.SocketOptsImpl.reuseAddress(SocketOptsImpl.java:156)
>>>>> at
>>>>>
>>>>>
>>> sun.nio.ch.SocketOptsImpl$IP$TCP.reuseAddress(SocketOptsImpl.java:268)
>>>
>>>>> at
>>>>> sun.nio.ch.OptionAdaptor.setReuseAddress(OptionAdaptor.java:177)
>>>>> at
>>>>> sun.nio.ch.SocketAdaptor.setReuseAddress(SocketAdaptor.java:343)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketO
>>>>> pt
>>>>> io
>>>>> ns
>>>>> (SelectorThread.java:1910)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAcce
>>>>> pt
>>>>> (S
>>>>> el
>>>>> ectorThread.java:1475)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConn
>>>>> ec
>>>>> ti
>>>>> on
>>>>> (SelectorThread.java:1439)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(S
>>>>> el
>>>>> ec
>>>>> to
>>>>> rThread.java:1350)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startListe
>>>>> ne
>>>>> r(
>>>>> Se
>>>>> lectorThread.java:1284)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpo
>>>>> in
>>>>> t(
>>>>> Se
>>>>> lectorThread.java:1247)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.SelectorThread.run(Select
>>>>> or
>>>>> Th
>>>>> re
>>>>> ad.java:1223)
>>>>> |#]
>>>>>
>>>>> [#|2009-01-06T12:11:31.928+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.resource.resourceadapter|_ThreadID=22;_ThreadName=httpWork
>>>>> erThread-80-6;ARP3_SybaseConnectionPool;In-use connections equal
>>>>> max-pool-size and expired max-wait-time. Cannot allocate more
>>>>> conne
>>>>> ctions.;_RequestID=1e35fcca-aa23-4ea6-9370-bf52d6f0a317;|RAR5117 :
>>>>> Failed to obtain/create connection from connection pool [ ARP3_Sy
>>>>> baseConnectionPool ]. Reason : In-use connections equal
>>>>> max-pool-size and expired max-wait-time. Cannot allocate more
>>>>> connections.|# ]
>>>>>
>>>>> [#|2009-01-06T12:11:31.928+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.resource.resourceadapter|_ThreadID=25;_ThreadName=httpWork
>>>>> erThread-80-9;ARP3_SybaseConnectionPool;In-use connections equal
>>>>> max-pool-size and expired max-wait-time. Cannot allocate more
>>>>> conne
>>>>> ctions.;_RequestID=74d7ffa2-cc14-4b01-b1f6-a869c6ea366d;|RAR5117 :
>>>>> Failed to obtain/create connection from connection pool [ ARP3_Sy
>>>>> baseConnectionPool ]. Reason : In-use connections equal
>>>>> max-pool-size and expired max-wait-time. Cannot allocate more
>>>>> connections.|# ]
>>>>>
>>>>> [#|2009-01-06T12:11:31.929+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.resource.resourceadapter|_ThreadID=22;_ThreadName=httpWork
>>>>> erThread-80-6;Error in allocating a connection. Cause: In-use
>>>>> connections equal max-pool-size and expired max-wait-time. Cannot
>>>>> allo
>>>>>
>>>>> cate more
>>>>> connections.;_RequestID=1e35fcca-aa23-4ea6-9370-bf52d6f0a317;|RAR5
>>>>> 11
>>>>> 4
>>>>>
>>>> :
>>>>
>>>>> Error allocating connection : [Error in allocating a connection.
>>>>> Cause: In-use connections equal max-pool-size and expired
>>>>> max-wait-time. Cannot allocate more connections.]|#]
>>>>>
>>>>> [#|2009-01-06T12:11:31.930+0000|WARNING|sun-appserver9.1|javax.ent
>>>>> er pr is
>>>>> e.resource.resourceadapter|_ThreadID=25;_ThreadName=httpWork
>>>>> erThread-80-9;Error in allocating a connection. Cause: In-use
>>>>> connections equal max-pool-size and expired max-wait-time. Cannot
>>>>> allo
>>>>>
>>>>> cate more
>>>>> connections.;_RequestID=74d7ffa2-cc14-4b01-b1f6-a869c6ea366d;|RAR5
>>>>> 11
>>>>> 4
>>>>>
>>>> :
>>>>
>>>>> Error allocating connection : [Error in allocating a connection.
>>>>> Cause: In-use connections equal max-pool-size and expired
>>>>> max-wait-time. Cannot allocate more connections.]|#]
>>>>>
>>>>>
>>>>> Thanks & Regards,
>>>>> Rishi Chopra.
>>>>> http://www.bgcpartners.com
>>>>> CONFIDENTIAL: This e-mail has been sent to you by one of the BGC
>>>>> entities (collectively BGC) listed at the following link
>>>>>
>>>> http://www.bgcpartners.com/legal/disclaimers/index.html#email_discla
>>>> im
>>>> er
>>>> .
>>>>
>>>>> The link contains company and FSA registration numbers. This
>>>>> e-mail,
>>>>>
>>>>> including its contents and attachments, if any, are confidential.
>>>>> If
>>>>>
>>>>> you are not the named recipient please notify the sender and
>>>>> immediately delete it. You may not disseminate, distribute, or
>>>>> forward
>>>>>
>>>>> this e-mail message or disclose its contents to anybody else.
>>>>>
>>>> Copyright and any other intellectual property rights in its contents
>>>>
>>
>>
>>>> are the sole property of BGC and its affiliates. E-mail transmission
>>>>
>>
>>
>>>> cannot be guaranteed to be secure or error-free.
>>>>
>>>>> The sender therefore does not accept liability for any errors or
>>>>> omissions in the contents of this message which arise as a result
>>>>> of
>>>>>
>>>>> e-mail transmission. If verification is required please request a
>>>>>
>>
>>
>>>>> hard-copy version. Although we routinely screen for viruses,
>>>>> addressees should check this e-mail and any attachments for
>>>>>
>> viruses.
>>
>>>>> We make no representation or warranty as to the absence of viruses
>>>>>
>>
>>
>>>>> in
>>>>>
>>>> this e-mail or any attachments. Please note that to ensure
>>>> regulatory compliance and for the protection of our customers and
>>>> business, we may monitor and read e-mails sent to and from our
>>>>
>> server(s).
>>
>>>>> The registered offices of the BGC entities are at 1 Churchill
>>>>> Place,
>>>>>
>>>> London, E14 5RD.
>>>>
>>>>> For any issues arising from this email please reply to the sender.
>>>>>
>>
>>
>>>>> The FSA register appears at http://www.fsa.gov.uk/register/.
>>>>> The FSA regulates the financial services industry in the United
>>>>> Kingdom and is located at
>>>>> 25 The North Colonnade, Canary Wharf, London, E14 5HS.
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> - To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>> --------------------------------------------------------------------
>>>> - To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>>>
>>>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> Subject:
>> FW: Connection Leak generated in cs_arp3_loadtest
>> Date:
>> Tue, 13 Jan 2009 11:14:50 +0000
>>
>>
>>
>>
>>
>> Picture (Metafile)
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>
>