users@glassfish.java.net

jdbc connection pool reinitialization problem after database restart

From: Marius Masalas <m.masalas_at_gmail.com>
Date: Thu, 10 Oct 2013 13:34:12 +0300

Hello everyone,

We got GF which is connected to Sybase database via JDBC. Database runs on
active-standby cluster. JDBC connections are made to cluster's virtual IP
address which moves from active to standby during failover. Such failover
hapened not so long ago, i.e., database was stopped on first node and
started on another. JDBC pool's properties are set in such a way that
connection validation is required and it's table-based:

domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.is-connection-validation-required
= true
domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.connection-validation-method
= table
domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.validate-atmost-once-period-in-seconds
= 0
domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.validation-table-name =
sysobjects
domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.connection-creation-retry-attempts
= 0
domain.resources.jdbc-connection-pool.jdbc/ECDI_cp.connection-creation-retry-interval-in-seconds
= 10

With such configuration I'd expect that GF would re-establish connection to
database.

During failover we see such messages in server.log, showing that connection
validation has failed, but there were errors while destroying resource:

[#|2013-09-30T05:25:52.374+0300|INFO|sun-appserver2.1|javax.enterprise.resource.resourceadapter|_ThreadID=537;_ThreadName=p:
thread-pool-1; w: 26307;sysobjects;|RAR5074 : Table based validation
detected invalid connection. Querying the table sysobjects failed. Set
resource-adapter log-level to FINE for exception stack trace. |#]

[#|2013-09-30T05:25:52.392+0300|WARNING|sun-appserver2.1|javax.enterprise.resource.resourceadapter|_ThreadID=537;_ThreadName=p:
thread-pool-1; w:
26307;_RequestID=17935fc8-0568-4176-aa18-ec46bd2df0c2;|Error while
destroying resource :JZ0C0: Connection is already closed.|#]

[#|2013-09-30T05:25:52.397+0300|WARNING|sun-appserver2.1|javax.enterprise.resource.resourceadapter|_ThreadID=537;_ThreadName=p:
thread-pool-1; w:
26307;_RequestID=17935fc8-0568-4176-aa18-ec46bd2df0c2;|RAR503
5:Unexpected exception while destroying resource. To get exception stack,
please change log level to FINE.|#]

[#|2013-09-30T05:25:53.400+0300|INFO|sun-appserver2.1|javax.enterprise.resource.resourceadapter|_ThreadID=538;_ThreadName=p:
thread-pool-1; w: 26302;sysobjects;|RAR5074 : Table based validation
detected invalid connection. Querying the table sysobjects failed. Set
resource-adapter log-level to FINE for exception stack trace. |#]

Soon after failover errors in application log show that stored procedure
calls fail with error that "stored procedure X is not found". It looks like
connection to database was not reestablished, therefore application
couldn't access database. GF was restarted and connection became active
again, and I haven't check connection status on IP level.

I wonder if such behaviour is related to some misconfiguration of GF
itself. I.e., I'm not sure if it's good idea to have property
connection-creation-retry-attempts = 0. Or is it a problem that GF got
exception from JDBC driver while trying to destroy resource. Or perhaps
some other configuration is wrong which I missed. Any ideas where to look?

Regards,
Marius