users@glassfish.java.net

Segmentation fault after adding to native library path suffix

From: Chris Dumoulin <dumoulin_at_oanda.com>
Date: Wed, 15 Apr 2009 18:08:58 -0400

I need to add the path to some IBM DB2 libraries to the native library
path so that these libraries are accessible to an OpenSSO that will be
running in Glassfish. I've added a native library path suffix through
the admin console (web page) that shows up in domain.xml like this:
native-library-path-suffix="/opt/IBM/db2/V8.1/lib"

Now I get a segmentation fault when I start the domain. The output is
below. I've also attached the hs_err_pid16572.log file that was
generated. Any help would be appreciated.

[root_at_house-opensso config]# /oanda/system/glassfish/bin/asadmin
start-domain domain1
Starting Domain domain1, please wait.
Default Log location is
/oanda/system/glassfish/domains/domain1/logs/server.log.
Redirecting output to
/oanda/system/glassfish/domains/domain1/logs/server.log
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xc0f45ef4, pid=16572, tid=11
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_17-b04 mixed mode)
# Problematic frame:
# C [libdb2.so.1+0x545ef4]
__1cNsqlzRcToSqlca6FpnHUCintfc_pnFsqlca_pciHpnQsqlzMessageToken__v_+0x74
#
# An error report file with more information is saved as hs_err_pid16572.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
Domain domain1 failed to startup. Please check the server log for more
details.
CLI156 Could not start the domain domain1.




#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0xc0f45ef4, pid=16572, tid=11
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_17-b04 mixed mode)
# Problematic frame:
# C [libdb2.so.1+0x545ef4] __1cNsqlzRcToSqlca6FpnHUCintfc_pnFsqlca_pciHpnQsqlzMessageToken__v_+0x74
#

--------------- T H R E A D ---------------

Current thread (0x08811d48): JavaThread "pool-1-thread-3" [_thread_in_native, id=11]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000c

Registers:
EAX=0x00000000, EBX=0xc10ff0e8, ECX=0xc125fe58, EDX=0xc10636a2
ESP=0xc862da44, EBP=0xc862da84, ESI=0x00000000, EDI=0x870000cd
EIP=0xc0f45ef4, EFLAGS=0x00010206

Top of Stack: (sp=0xc862da44)
0xc862da44: 00000000 09078258 c10ff0e8 00000000
0xc862da54: 08720005 90000403 000003e7 00000002
0xc862da64: 00000002 c07f9e94 875c00cd fffffffb
0xc862da74: c124bfe4 c862daa0 c0e5651a c862dacc
0xc862da84: c862dac0 c0e97b36 09078258 00000000
0xc862da94: c1267e84 875c00cd 00000000 c862db24
0xc862daa4: 09079324 c128e4e8 c10ff0e8 09079324
0xc862dab4: c128e4e8 875c00cd c128e4e8 c862dc28

Instructions: (pc=0xc0f45ef4)
0xc0f45ee4: 00 81 ff 6d 00 00 80 0f 84 37 03 00 00 8b 75 0c
0xc0f45ef4: 8b 46 0c 85 c0 7d 17 81 ff 16 00 00 81 0f 85 08

Stack: [0xc85ef000,0xc862f000), sp=0xc862da44, free space=250k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libdb2.so.1+0x545ef4] __1cNsqlzRcToSqlca6FpnHUCintfc_pnFsqlca_pciHpnQsqlzMessageToken__v_+0x74
C [libdb2.so.1+0x497b36] __1cYsqleUCsetSecurityErrInfo6FpnHUCintfc_ipc2LlpnQsqlzMessageToken__v_+0x296
C [libdb2.so.1+0x4b7fc6] __1cSsqlexClientEncPass6FpnHUCintfc_pcL2pL_i_+0x12a
C [libdb2.so.1+0x497d9b] __1cWsqleuClientSetPassword6FpnLUCconHandle_pcL_i_+0x7f
C [libdb2jcct2.so.1+0x33a8]
C [libdb2jcct2.so.1+0x3f76] Java_com_ibm_db2_jcc_uw_UWConnection_DBConnect+0x3c6
j com.ibm.db2.jcc.uw.UWConnection.DBConnect(Ljava/lang/String;[I[Ljava/lang/Object;[Ljava/lang/String;[Ljava/lang/String;IZ)J+0
j com.ibm.db2.jcc.uw.UWConnection.a(Lcom/ibm/db2/jcc/uw/l;)V+199
j com.ibm.db2.jcc.uw.f.v()V+186
j com.ibm.db2.jcc.uw.UWConnection.a(Ljava/lang/String;Ljava/util/Properties;)V+595
j com.ibm.db2.jcc.uw.UWConnection.a(Lcom/ibm/db2/jcc/b/i;ILjava/lang/String;ILjava/util/Properties;)V+99
j com.ibm.db2.jcc.uw.UWConnection.<init>(Lcom/ibm/db2/jcc/b/i;ILjava/lang/String;Ljava/util/Properties;Lcom/ibm/db2/jcc/DB2Driver;)V+54
j com.ibm.db2.jcc.DB2Driver.connect(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+682
j java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/util/Properties;Ljava/lang/ClassLoader;)Ljava/sql/Connection;+165
j java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/sql/Connection;+41
j com.oanda.opensso.plugins.Db2DatabaseRepo.initialize(Ljava/util/Map;)V+273
j com.oanda.opensso.plugins.CrmEntityRepo.initialize(Ljava/util/Map;)V+2
j com.sun.identity.idm.server.IdRepoPluginsCache.constructIdRepoPlugin(Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;)Lcom/sun/identity/idm/IdRepo;+201
j com.sun.identity.idm.server.IdRepoPluginsCache.getIdRepoPlugins(Ljava/lang/String;)Ljava/util/Set;+348
j com.sun.identity.idm.server.IdRepoPluginsCache.getIdRepoPlugins(Ljava/lang/String;Lcom/sun/identity/idm/IdOperation;Lcom/sun/identity/idm/IdType;)Ljava/util/Set;+130
j com.sun.identity.idm.server.IdServicesImpl.isExists(Lcom/iplanet/sso/SSOToken;Lcom/sun/identity/idm/IdType;Ljava/lang/String;Ljava/lang/String;)Z+24
j com.sun.identity.idm.AMIdentity.isExists()Z+21
j com.sun.identity.authentication.service.AuthD.getIdentity(Lcom/sun/identity/idm/IdType;Ljava/lang/String;Ljava/lang/String;)Lcom/sun/identity/idm/AMIdentity;+153
j com.sun.identity.authentication.service.LoginState.searchUserProfile(Ljavax/security/auth/Subject;Lcom/sun/identity/authentication/AuthContext$IndexType;Ljava/lang/String;)Z+314
j com.sun.identity.authentication.service.AMLoginContext.runLogin()V+160
j com.sun.identity.authentication.server.AuthContextLocal.submitRequirements([Ljavax/security/auth/callback/Callback;)V+40
j com.sun.identity.authentication.AuthContext.submitRequirements([Ljavax/security/auth/callback/Callback;)V+63
j com.sun.identity.security.SystemAppTokenProvider.getAppSSOToken()Lcom/iplanet/sso/SSOToken;+52
j com.sun.identity.security.AdminTokenAction.getSSOToken()Lcom/iplanet/sso/SSOToken;+96
j com.sun.identity.security.AdminTokenAction.authenticationInitialized()V+4
j com.sun.identity.authentication.UI.LoginLogoutMapping.initializeAuth(Ljavax/servlet/ServletContext;)Z+39
j com.sun.identity.authentication.UI.LoginLogoutMapping.init(Ljavax/servlet/ServletConfig;)V+23
j org.apache.catalina.core.StandardWrapper.loadServlet()Ljavax/servlet/Servlet;+687
j org.apache.catalina.core.StandardWrapper.load()V+2
j org.apache.catalina.core.StandardContext.loadOnStartup([Lorg/apache/catalina/Container;)V+178
j org.apache.catalina.core.StandardContext.start()V+1695
j com.sun.enterprise.web.WebModule.start()V+1
j com.sun.enterprise.web.LifecycleStarter.doRun()V+4
j com.sun.appserv.management.util.misc.RunnableBase.runSync()V+64
j com.sun.appserv.management.util.misc.RunnableBase.run()V+18
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask$Sync.innerRun()V+22
j java.util.concurrent.FutureTask.run()V+4
j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+43
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0xa8d97]
V [libjvm.so+0xa8c04]
V [libjvm.so+0xa8be8]
V [libjvm.so+0xb664e]
V [libjvm.so+0xb657d]
V [libjvm.so+0xb64f0]
V [libjvm.so+0xb63f1]
V [libjvm.so+0xb6395]
V [libjvm.so+0x2d0552]
C [libc.so.1+0xa59a9]
C [libc.so.1+0xa5c90]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j com.ibm.db2.jcc.uw.UWConnection.DBConnect(Ljava/lang/String;[I[Ljava/lang/Object;[Ljava/lang/String;[Ljava/lang/String;IZ)J+0
j com.ibm.db2.jcc.uw.UWConnection.a(Lcom/ibm/db2/jcc/uw/l;)V+199
j com.ibm.db2.jcc.uw.f.v()V+186
j com.ibm.db2.jcc.uw.UWConnection.a(Ljava/lang/String;Ljava/util/Properties;)V+595
j com.ibm.db2.jcc.uw.UWConnection.a(Lcom/ibm/db2/jcc/b/i;ILjava/lang/String;ILjava/util/Properties;)V+99
j com.ibm.db2.jcc.uw.UWConnection.<init>(Lcom/ibm/db2/jcc/b/i;ILjava/lang/String;Ljava/util/Properties;Lcom/ibm/db2/jcc/DB2Driver;)V+54
j com.ibm.db2.jcc.DB2Driver.connect(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+682
j java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/util/Properties;Ljava/lang/ClassLoader;)Ljava/sql/Connection;+165
j java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/sql/Connection;+41
j com.oanda.opensso.plugins.Db2DatabaseRepo.initialize(Ljava/util/Map;)V+273
j com.oanda.opensso.plugins.CrmEntityRepo.initialize(Ljava/util/Map;)V+2
j com.sun.identity.idm.server.IdRepoPluginsCache.constructIdRepoPlugin(Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;)Lcom/sun/identity/idm/IdRepo;+201
j com.sun.identity.idm.server.IdRepoPluginsCache.getIdRepoPlugins(Ljava/lang/String;)Ljava/util/Set;+348
j com.sun.identity.idm.server.IdRepoPluginsCache.getIdRepoPlugins(Ljava/lang/String;Lcom/sun/identity/idm/IdOperation;Lcom/sun/identity/idm/IdType;)Ljava/util/Set;+130
j com.sun.identity.idm.server.IdServicesImpl.isExists(Lcom/iplanet/sso/SSOToken;Lcom/sun/identity/idm/IdType;Ljava/lang/String;Ljava/lang/String;)Z+24
j com.sun.identity.idm.AMIdentity.isExists()Z+21
j com.sun.identity.authentication.service.AuthD.getIdentity(Lcom/sun/identity/idm/IdType;Ljava/lang/String;Ljava/lang/String;)Lcom/sun/identity/idm/AMIdentity;+153
j com.sun.identity.authentication.service.LoginState.searchUserProfile(Ljavax/security/auth/Subject;Lcom/sun/identity/authentication/AuthContext$IndexType;Ljava/lang/String;)Z+314
j com.sun.identity.authentication.service.AMLoginContext.runLogin()V+160
j com.sun.identity.authentication.server.AuthContextLocal.submitRequirements([Ljavax/security/auth/callback/Callback;)V+40
j com.sun.identity.authentication.AuthContext.submitRequirements([Ljavax/security/auth/callback/Callback;)V+63
j com.sun.identity.security.SystemAppTokenProvider.getAppSSOToken()Lcom/iplanet/sso/SSOToken;+52
j com.sun.identity.security.AdminTokenAction.getSSOToken()Lcom/iplanet/sso/SSOToken;+96
j com.sun.identity.security.AdminTokenAction.authenticationInitialized()V+4
j com.sun.identity.authentication.UI.LoginLogoutMapping.initializeAuth(Ljavax/servlet/ServletContext;)Z+39
j com.sun.identity.authentication.UI.LoginLogoutMapping.init(Ljavax/servlet/ServletConfig;)V+23
j org.apache.catalina.core.StandardWrapper.loadServlet()Ljavax/servlet/Servlet;+687
j org.apache.catalina.core.StandardWrapper.load()V+2
j org.apache.catalina.core.StandardContext.loadOnStartup([Lorg/apache/catalina/Container;)V+178
j org.apache.catalina.core.StandardContext.start()V+1695
j com.sun.enterprise.web.WebModule.start()V+1
j com.sun.enterprise.web.LifecycleStarter.doRun()V+4
j com.sun.appserv.management.util.misc.RunnableBase.runSync()V+64
j com.sun.appserv.management.util.misc.RunnableBase.run()V+18
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask$Sync.innerRun()V+22
j java.util.concurrent.FutureTask.run()V+4
j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+43
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
  0x0810cdc8 JavaThread "SystemTimerPool" [_thread_blocked, id=168]
  0x09113478 JavaThread "SystemTimerPool" [_thread_blocked, id=167]
  0x0810e320 JavaThread "SystemTimerPool" [_thread_blocked, id=166]
  0x09112da8 JavaThread "SystemTimerPool-Scheduler" [_thread_blocked, id=165]
  0x0810b160 JavaThread "amSession" daemon [_thread_blocked, id=163]
  0x092ae6d8 JavaThread "amSession" daemon [_thread_blocked, id=162]
  0x0917e340 JavaThread "amSession" daemon [_thread_blocked, id=161]
  0x0917d680 JavaThread "amSession" daemon [_thread_blocked, id=160]
  0x092e1460 JavaThread "amSession" daemon [_thread_blocked, id=159]
  0x08112d28 JavaThread "amSession" daemon [_thread_blocked, id=158]
  0x08852010 JavaThread "amSession" daemon [_thread_blocked, id=157]
  0x08851e88 JavaThread "amSession" daemon [_thread_blocked, id=156]
  0x093278d0 JavaThread "amSession" daemon [_thread_blocked, id=155]
  0x092af590 JavaThread "amSession" daemon [_thread_blocked, id=154]
  0x092af408 JavaThread "SystemTimer" [_thread_blocked, id=153]
  0x08112300 JavaThread "SystemTimer-Scheduler" [_thread_blocked, id=152]
  0x09229318 JavaThread "Timer-11" [_thread_blocked, id=150]
  0x09070bd0 JavaThread "smIdmThreadPool" [_thread_blocked, id=146]
  0x09126be0 JavaThread "smIdmThreadPool" [_thread_blocked, id=145]
  0x09125f20 JavaThread "smIdmThreadPool" [_thread_blocked, id=144]
  0x091252c0 JavaThread "smIdmThreadPool" [_thread_blocked, id=143]
  0x09124660 JavaThread "smIdmThreadPool" [_thread_blocked, id=142]
  0x09123a00 JavaThread "smIdmThreadPool" [_thread_blocked, id=141]
  0x09114fa0 JavaThread "smIdmThreadPool" [_thread_blocked, id=140]
  0x09291a40 JavaThread "smIdmThreadPool" [_thread_blocked, id=139]
  0x09133508 JavaThread "smIdmThreadPool" [_thread_blocked, id=138]
  0x09070648 JavaThread "smIdmThreadPool" [_thread_blocked, id=137]
  0x091ca918 JavaThread "LDAPConnThread-4 ldap://localhost:50389" daemon [_thread_in_native, id=136]
  0x09178dd8 JavaThread "LDAPConnThread-0 ldap://localhost:50389" daemon [_thread_in_native, id=130]
  0x091760d0 JavaThread "Idle Time Limit Thread" daemon [_thread_blocked, id=129]
  0x091741b8 JavaThread "LDIFConnectionHandler" daemon [_thread_blocked, id=128]
  0x091749e8 JavaThread "LDAP Connection Handler 0.0.0.0 port 50389" daemon [_thread_in_native, id=127]
  0x091744b0 JavaThread "Work Queue Monitor Provider" daemon [_thread_blocked, id=126]
  0x09172d00 JavaThread "Worker Thread 23" daemon [_thread_blocked, id=125]
  0x091720a0 JavaThread "Worker Thread 22" daemon [_thread_blocked, id=124]
  0x09171440 JavaThread "Worker Thread 21" daemon [_thread_blocked, id=123]
  0x091707e0 JavaThread "Worker Thread 20" daemon [_thread_blocked, id=122]
  0x0916fb80 JavaThread "Worker Thread 19" daemon [_thread_blocked, id=121]
  0x0916ef20 JavaThread "Worker Thread 18" daemon [_thread_blocked, id=120]
  0x0916e2c0 JavaThread "Worker Thread 17" daemon [_thread_blocked, id=119]
  0x0916d660 JavaThread "Worker Thread 16" daemon [_thread_blocked, id=118]
  0x08b41ff8 JavaThread "Worker Thread 15" daemon [_thread_blocked, id=117]
  0x08b41398 JavaThread "Worker Thread 14" daemon [_thread_blocked, id=116]
  0x08b40738 JavaThread "Worker Thread 13" daemon [_thread_blocked, id=115]
  0x08b3fad8 JavaThread "Worker Thread 12" daemon [_thread_blocked, id=114]
  0x08b3ee78 JavaThread "Worker Thread 11" daemon [_thread_blocked, id=113]
  0x08b3eac8 JavaThread "Worker Thread 10" daemon [_thread_blocked, id=112]
  0x08b3d0e0 JavaThread "Worker Thread 9" daemon [_thread_blocked, id=111]
  0x08b3c480 JavaThread "Worker Thread 8" daemon [_thread_blocked, id=110]
  0x08b3b880 JavaThread "Worker Thread 7" daemon [_thread_blocked, id=109]
  0x08b8e510 JavaThread "Worker Thread 6" daemon [_thread_blocked, id=108]
  0x08b8d8b0 JavaThread "Worker Thread 5" daemon [_thread_blocked, id=107]
  0x08b8cc50 JavaThread "Worker Thread 4" daemon [_thread_blocked, id=106]
  0x08b8c0e0 JavaThread "Worker Thread 3" daemon [_thread_blocked, id=105]
  0x0902de98 JavaThread "Worker Thread 2" daemon [_thread_blocked, id=104]
  0x09029a90 JavaThread "Worker Thread 1" daemon [_thread_blocked, id=103]
  0x09029908 JavaThread "Worker Thread 0" daemon [_thread_blocked, id=102]
  0x081f0998 JavaThread "JVM Memory Usage Monitor Provider" daemon [_thread_blocked, id=101]
  0x08ca1af0 JavaThread "LDAP Request Handler 1 for connection handler LDAP Connection Handler 0.0.0.0 port 50389" daemon [_thread_in_native, id=100]
  0x0931c338 JavaThread "LDAP Request Handler 0 for connection handler LDAP Connection Handler 0.0.0.0 port 50389" daemon [_thread_in_native, id=99]
  0x08bf6420 JavaThread "Task Scheduler Thread" daemon [_thread_blocked, id=97]
  0x081f3050 JavaThread "Checkpointer" daemon [_thread_blocked, id=96]
  0x081f2ec8 JavaThread "Cleaner-1" daemon [_thread_blocked, id=95]
  0x0931e258 JavaThread "INCompressor" daemon [_thread_blocked, id=94]
  0x08fc4420 JavaThread "Multifile Text Writer for cn=replication repair logger,cn=loggers,cn=config" daemon [_thread_blocked, id=92]
  0x0921dbd0 JavaThread "Multifile Text Writer for cn=file-based error logger,cn=loggers,cn=config" daemon [_thread_blocked, id=89]
  0x092961e0 JavaThread "Asyncronous Text Writer for cn=file-based access logger,cn=loggers,cn=config" daemon [_thread_blocked, id=87]
  0x092211a0 JavaThread "Multifile Text Writer for cn=file-based access logger,cn=loggers,cn=config" daemon [_thread_blocked, id=86]
  0x09221018 JavaThread "Time Thread" daemon [_thread_blocked, id=84]
  0x088f09f8 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/opensso]]" daemon [_thread_blocked, id=81]
  0x08e84708 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]" daemon [_thread_blocked, id=80]
  0x08e7e278 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon [_thread_blocked, id=79]
  0x08e86b18 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]" daemon [_thread_blocked, id=78]
  0x08e668b8 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[]]" daemon [_thread_blocked, id=77]
  0x08e735a0 JavaThread "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[__asadmin].StandardContext[/web1]]" daemon [_thread_blocked, id=76]
  0x08c7ba18 JavaThread "httpWorkerThread-4848-1" daemon [_thread_blocked, id=75]
  0x08c7b7b0 JavaThread "httpWorkerThread-4848-0" daemon [_thread_blocked, id=74]
  0x0836e660 JavaThread "httpSSLWorkerThread-8181-1" daemon [_thread_blocked, id=73]
  0x08b70770 JavaThread "httpSSLWorkerThread-8181-0" daemon [_thread_blocked, id=72]
  0x0833e1b0 JavaThread "SelectorThread-4848" [_thread_in_native, id=71]
  0x08598548 JavaThread "SelectorThread-8181" [_thread_in_native, id=70]
  0x08595078 JavaThread "httpSSLWorkerThread-8080-1" daemon [_thread_blocked, id=69]
  0x08594ef0 JavaThread "httpSSLWorkerThread-8080-0" daemon [_thread_blocked, id=68]
  0x08b4ca20 JavaThread "SelectorThread-8080" [_thread_in_native, id=67]
  0x08b4cdc0 JavaThread "pool-2-thread-1" [_thread_blocked, id=66]
  0x0836bb20 JavaThread "Thread-20" [_thread_blocked, id=64]
  0x088810b8 JavaThread "RMI RenewClean-[10.1.1.76:44899,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_]" daemon [_thread_blocked, id=61]
  0x08bf97e0 JavaThread "ClusterServiceListener" daemon [_thread_in_native, id=60]
  0x0884cf88 JavaThread "MessageBusCallbackDispatcher" daemon [_thread_blocked, id=59]
  0x0884cd60 JavaThread "jms_ACCEPT" [_thread_in_native, id=58]
  0x08bf8c50 JavaThread "admin_ACCEPT" [_thread_in_native, id=57]
  0x0894bb70 JavaThread "RMI RenewClean-[10.1.1.76:44912]" daemon [_thread_blocked, id=54]
  0x0886a310 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=53]
  0x08bcac80 JavaThread "ClusterDiscoveryService" daemon [_thread_in_native, id=52]
  0x08bd4a18 JavaThread "Broker Monitor" daemon [_thread_blocked, id=51]
  0x08bce820 JavaThread "JMQPortMapper" daemon [_thread_in_native, id=50]
  0x08870268 JavaThread "MQTimer-Thread" daemon [_thread_blocked, id=49]
  0x0886fcd8 JavaThread "p: thread-pool-1; w: 3" daemon [_thread_blocked, id=48]
  0x08357588 JavaThread "RMI ConnectionExpiration-[house-opensso:8686]" daemon [_thread_blocked, id=44]
  0x08356b10 JavaThread "RMI ConnectionExpiration-[10.1.1.76:44905]" daemon [_thread_blocked, id=43]
  0x08372798 JavaThread "RMI TCP Connection(3)-10.1.1.76" daemon [_thread_in_native, id=42]
  0x08372488 JavaThread "RMI RenewClean-[10.1.1.76:44905]" daemon [_thread_blocked, id=41]
  0x0835c088 JavaThread "RMI TCP Connection(2)-10.1.1.76" daemon [_thread_in_native, id=40]
  0x08373408 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=39]
  0x08355bb8 JavaThread "RMI TCP Accept-8686" daemon [_thread_in_native, id=38]
  0x0896f100 JavaThread "Thread-12" daemon [_thread_blocked, id=35]
  0x08976728 JavaThread "Thread-11" [_thread_blocked, id=34]
  0x081f9bc0 JavaThread "Thread-9" [_thread_blocked, id=32]
  0x0885cbf0 JavaThread "p: thread-pool-1; w: 2" daemon [_thread_in_native, id=31]
  0x0885cea8 JavaThread "p: thread-pool-1; w: 1" daemon [_thread_in_native, id=30]
  0x083a5f48 JavaThread "SelectorThread" daemon [_thread_in_native, id=29]
  0x08acf3c8 JavaThread "Timer-4" daemon [_thread_blocked, id=28]
  0x08ad7b40 JavaThread "Timer-3" [_thread_blocked, id=27]
  0x08a7be90 JavaThread "Timer-2" [_thread_blocked, id=26]
  0x08a46ca8 JavaThread "Timer-1" [_thread_blocked, id=25]
  0x08a1fad0 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=24]
  0x08a24c38 JavaThread "RMI TCP Connection(1)-127.0.0.1" daemon [_thread_in_native, id=23]
  0x089b8910 JavaThread "GC Daemon" daemon [_thread_blocked, id=22]
  0x089b4310 JavaThread "RMI Reaper" [_thread_blocked, id=21]
  0x089b3f60 JavaThread "Timer-0" daemon [_thread_blocked, id=20]
  0x089aa108 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=19]
  0x08828fe8 JavaThread "pool-1-thread-10" [_thread_blocked, id=18]
  0x08806898 JavaThread "pool-1-thread-9" [_thread_blocked, id=17]
  0x08807970 JavaThread "pool-1-thread-8" [_thread_blocked, id=16]
  0x08806cb0 JavaThread "pool-1-thread-7" [_thread_blocked, id=15]
  0x0880cfe0 JavaThread "pool-1-thread-6" [_thread_blocked, id=14]
  0x0880a470 JavaThread "pool-1-thread-5" [_thread_blocked, id=13]
  0x088158c0 JavaThread "pool-1-thread-4" [_thread_blocked, id=12]
=>0x08811d48 JavaThread "pool-1-thread-3" [_thread_in_native, id=11]
  0x0875b008 JavaThread "pool-1-thread-2" [_thread_blocked, id=10]
  0x0875ac00 JavaThread "pool-1-thread-1" [_thread_blocked, id=9]
  0x081514d0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=7]
  0x0814eca0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=6]
  0x0814df30 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5]
  0x08141a88 JavaThread "Finalizer" daemon [_thread_blocked, id=4]
  0x08140df0 JavaThread "Reference Handler" daemon [_thread_blocked, id=3]
  0x08078620 JavaThread "main" [_thread_blocked, id=1]

Other Threads:
  0x0813e578 VMThread [id=2]
  0x08152690 WatcherThread [id=8]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation total 35520K, used 26615K [0xcd200000, 0xcf5b0000, 0xd7ca0000)
  eden space 34496K, 74% used [0xcd200000, 0xceafde60, 0xcf3b0000)
  from space 1024K, 100% used [0xcf4b0000, 0xcf5b0000, 0xcf5b0000)
  to space 1024K, 0% used [0xcf3b0000, 0xcf3b0000, 0xcf4b0000)
 tenured generation total 72732K, used 46278K [0xd7ca0000, 0xdc3a7000, 0xed200000)
   the space 72732K, 63% used [0xd7ca0000, 0xda9d1ae8, 0xda9d1c00, 0xdc3a7000)
 compacting perm gen total 60160K, used 60110K [0xed200000, 0xf0cc0000, 0xf9200000)
   the space 60160K, 99% used [0xed200000, 0xf0cb3b48, 0xf0cb3c00, 0xf0cc0000)
No shared spaces configured.

Dynamic libraries:
0x08050000 /usr/jdk/instances/jdk1.5.0/bin/java
0xfefb0000 /lib/libthread.so.1
0xfef80000 /lib/libdl.so.1
0xfee50000 /lib/libc.so.1
0xfea00000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/client/libjvm.so
0xfee20000 /lib/libsocket.so.1
0xfefa0000 /usr/lib/libsched.so.1
0xfe9d0000 /usr/lib/libCrun.so.1
0xfe9b0000 /lib/libm.so.1
0xfe910000 /lib/libnsl.so.1
0xfe8b0000 /lib/libm.so.2
0xfe880000 /lib/libscf.so.1
0xfe860000 /lib/libdoor.so.1
0xfe840000 /lib/libuutil.so.1
0xfe820000 /lib/libgen.so.1
0xfe800000 /lib/libmd.so.1
0xfe7e0000 /lib/libmp.so.2
0xfe7c0000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/native_threads/libhpi.so
0xfe770000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libverify.so
0xfe740000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libjava.so
0xfe710000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libzip.so
0xf9250000 /usr/lib/locale/en_CA.UTF-8/en_CA.UTF-8.so.3
0xf9230000 /usr/lib/locale/common/methods_unicode.so.3
0xcc490000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libnet.so
0xc8630000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libmanagement.so
0xc8400000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libj2pkcs11.so
0xc83d0000 /usr/lib/libpkcs11.so
0xc83b0000 /usr/lib/libcryptoutil.so.1
0xc8360000 /usr/lib/security/pkcs11_softtoken.so
0xc8240000 /oanda/system/glassfish/lib/libjvminfoutil.so
0xc8220000 /lib/librt.so.1
0xc8200000 /lib/libkstat.so.1
0xc8070000 /usr/lib/libCstd.so.1
0xc8050000 /lib/libaio.so.1
0xc7fd0000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/librmi.so
0xc7e70000 /usr/jdk/instances/jdk1.5.0/jre/lib/i386/libnio.so
0xc7e10000 /usr/lib/libsendfile.so.1
0xc3a10000 /opt/IBM/db2/V8.1/lib/libdb2jcct2.so.1
0xc0a00000 /opt/IBM/db2/V8.1/lib/libdb2.so.1
0xc0970000 /opt/IBM/db2/V8.1/lib/libdb2g11n.so.1
0xc15a0000 /lib/libresolv.so.2
0xc68a0000 /opt/IBM/db2/V8.1/lib/libdb2install.so.1
0xc1570000 /opt/IBM/db2/V8.1/lib/libdb2locale.so.1
0xc0770000 /opt/IBM/db2/V8.1/lib/libdb2osse.so.1
0xc0730000 /opt/IBM/db2/V8.1/lib/libdb2genreg.so.1
0xc2de0000 /opt/IBM/db2/V8.1/lib/libdb2trcapi.so.1
0xc0700000 /opt/IBM/db2/V8.1/lib/libdb2dascmn.so.1
0xc83a0000 /lib/libpthread.so.1
0xc06e0000 /usr/lib/libkvm.so.1
0xc06c0000 /opt/IBM/db2/V8.1/lib/libdb2osse_db2.so.1
0xc0690000 /lib/libelf.so.1

VM Arguments:
jvm_args: -Dcom.sun.aas.instanceRoot=/oanda/system/glassfish/domains/domain1 -Dcom.sun.aas.ClassPathPrefix= -Dcom.sun.aas.ClassPathSuffix= -Dcom.sun.aas.ServerClassPath= -Dcom.sun.aas.classloader.appserverChainJars.ee= -Dcom.sun.aas.classloader.appserverChainJars=admin-cli.jar,admin-cli-ee.jar,j2ee-svc.jar -Dcom.sun.aas.classloader.excludesList=admin-cli.jar,appserv-upgrade.jar,sun-appserv-ant.jar -Dcom.sun.aas.classloader.optionalOverrideableChain.ee= -Dcom.sun.aas.classloader.optionalOverrideableChain=webservices-rt.jar,webservices-tools.jar -Dcom.sun.aas.classloader.serverClassPath.ee=/lib/hadbjdbc4.jar,/oanda/system/glassfish/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,/lib/dbstate.jar,/lib/hadbm.jar,/lib/hadbmgt.jar,/lib/mfwk_instrum_tk.jar -Dcom.sun.aas.classloader.serverClassPath=/oanda/system/glassfish/lib/install/applications/jmsra/imqjmsra.jar,/oanda/system/glassfish/imq/lib/jaxm-api.jar,/oanda/system/glassfish/imq/lib/fscontext.jar,/oanda/system/glassfish/imq/lib/imqbroker.jar,/oanda/system/glassfish/imq/lib/imqjmx.jar,/oanda/system/glassfish/lib/ant/lib/ant.jar,/oanda/system/glassfish/lib/SUNWjdmk/5.1/lib/jdmkrt.jar -Dcom.sun.aas.classloader.sharedChainJars.ee=appserv-se.jar,appserv-ee.jar,jesmf-plugin.jar,/lib/dbstate.jar,/lib/hadbjdbc4.jar,jgroups-all.jar,/lib/mfwk_instrum_tk.jar -Dcom.sun.aas.classloader.sharedChainJars=javaee.jar,/usr/jdk/instances/jdk1.5.0/jre/../lib/tools.jar,install/applications/jmsra/imqjmsra.jar,com-sun-commons-launcher.jar,com-sun-commons-logging.jar,/oanda/system/glassfish/imq/lib/jaxm-api.jar,/oanda/system/glassfish/imq/lib/fscontext.jar,/oanda/system/glassfish/imq/lib/imqbroker.jar,/oanda/system/glassfish/imq/lib/imqjmx.jar,/oanda/system/glassfish/imq/lib/imqxm.jar,webservices-rt.jar,webservices-tools.jar,mail.jar,appserv-jstl.jar,jmxremote_optional.jar,/oanda/system/glassfish/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,activation.jar,appserv-rt.jar,appserv-admin.jar,appserv-cmp.jar,/oanda/system/glassfish/updatecenter/lib/updatecenter.jar,/oanda
java_command: com.sun.enterprise.server.PELaunch start
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/oanda/system/gnu/bin:/oanda/system/sbin:/oanda/system/bin:/usr/bin:/usr/sbin:/usr/ucb/bin:/usr/ccs/bin:/oanda/system/gnu/bin:/oanda/system/sbin:/oanda/system/bin:/usr/bin:/usr/sbin:/usr/ucb/bin:/usr/ccs/bin:/usr/bin:/home/dumoulin/bin
USERNAME=root
LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.5.0/jre/lib/i386/client:/usr/jdk/instances/jdk1.5.0/jre/lib/i386:/usr/jdk/instances/jdk1.5.0/jre/../lib/i386:/usr/jdk/instances/jdk1.5.0/jre/lib/i386/server:/usr/jdk/instances/jdk1.5.0/jre/lib/i386:/usr/jdk/instances/jdk1.5.0/jre/../lib/i386:/oanda/system/glassfish/lib:/oanda/system/glassfish/lib:/oanda/system/glassfish/lib:
SHELL=/sbin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x331be0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x331be0], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x16db60], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x16db60], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x16db60], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP: [libjvm.so+0x2d1330], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x2d1330], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x2d1330], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x2d1330], sa_mask[0]=0xffbffeff, sa_flags=0x00000004


--------------- S Y S T E M ---------------

OS: Solaris 10 10/08 s10x_u6wos_07b X86
           Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
                        Use is subject to license terms.
                            Assembled 27 October 2008

uname:SunOS 5.10 Generic_138889-03 i86pc (T2 libthread)
rlimit: STACK 10240k, CORE infinity, NOFILE 65536, AS infinity
load average:0.76 0.55 0.43

CPU:total 4 (cores per cpu 2, threads per core 1) family 15 model 65 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnowext, 3dnow

Memory: 4k page, physical 8387816k(1762256k free)

vm_info: Java HotSpot(TM) Client VM (1.5.0_17-b04) for solaris-x86, built on Nov 10 2008 03:14:00 by unknown with unknown Workshop:0x550