users@glassfish.java.net

Production Showstopper: unable to create new native thread

From: <glassfish_at_javadesktop.org>
Date: Mon, 26 May 2008 02:17:55 PDT

We are running a multi-user Java EE application on GlassFish v2ur1 b09d. After some usage it crashes with a stacktrace like this:

[code]
[#|2008-05-25T05:39:38.599+0200|WARNING|sun-appserver9.1|javax.enterprise.system.stream.err|_ThreadID=27;_ThreadName=RMI RenewClean-[127.0.0.1:49373,co
m.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory_at_1983a];_RequestID=e31da6a8-3a0d-466b-a9e7-96d1f06ca9da;|java.lang.OutOfMemoryE
rror: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:321)
        at sun.rmi.server.UnicastRef.free(UnicastRef.java:395)
        at sun.rmi.server.UnicastRef.done(UnicastRef.java:412)
        at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
        at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
        at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:540)
        at java.lang.Thread.run(Thread.java:595)
|#]
[/code]
I have already increased the size of the Thread Pool from the default setting to (Min: 10, Max: 500), but this seems to have no effect. Perhaps I should configure more than one work queue?

Of course there may be a memory leak in our application, however, since it doesn't open any threads itself, I don't know what to do about that. I have seen that apparently at least one 3rd party lib is trying to open threads: we are frequently connecting to the Sonic Message Queue and can see this stacktrace coming out of its driver:

[code]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at progress.message.zclient.DebugThread.start(Unknown Source)
        at progress.message.zclient.Connection.connectWithRecoveryOpt(Unknown Source)
        at progress.message.zclient.ReconnectHelper.connectAndChaseSingleFailoverRedirect(Unknown Source)
        at progress.message.zclient.ReconnectHelper.connect(Unknown Source)
        at progress.message.zclient.Connection.connect(Unknown Source)
[/code]
One other thing that our application does is, it occasionally fetches SQL connections from the appserver's connection pool datasource. After using them, it calls connection.close() on them. We are doing the bulk of our SQL work via appserver managed connections (using JPA), but we are incorporating some legacy code, which wants us to provide a java.sql.Connection.

The OS is Sparc Solaris 5.10.

Many thanks in advance for any pointers,

Ulrich
[Message sent by forum member 'ulim' (ulim)]

http://forums.java.net/jive/thread.jspa?messageID=276537