> After a few hours HTTP requests consistently stop reaching the web
> application. The java application continues to run, but HTTP requests
> are not being passed to servlets. If I go into the admin console and
> fiddle with an HTTP Service setting, like enabling access logging for
> example, POOF!! the site comes back to life. I can only assume that
> doing that causes something to reset in Glassfish. Clearly this is a
> Glassfish issue and nothing to do with the application code. Maddening
> and unacceptable. My only option is to get off Glassfish and onto
> something more stable. I am stopping/starting the domain on my
> production servers constantly. If anyone has a suggestion to make this
> problem go away I will be very grateful.
> How quickly do the waters become muddy. This thread now has multiple issues posted to it, which renders it pretty much useless to anyone trying to actually track down and fix a bug in Glassfish.
>
I would disagree. We are having the same exact issue, except I've been
having it for 6 months in production and it has several symptoms:
1) after some period of time, users get a blank screen forever waiting
for a response. We have to restart the Windows service to fix it.
2) With the same frequency, sometimes users get a "Maximum connections
reached (4096)" error immediately. We have to restart the Windows
service to fix it.
3) Same symptom as #1, but restarting app server doesn't help and
Windows has to be rebooted. This is the NP Pool bug in Windows 2003
server, and it has only happened a few times.
Through these emails, finally after 6 months someone has told me that
the first two issues are NOT caused by the NP Pool problem in Windows
2003 Server. Sun is giving me other ideas now, such as disabling
Grizzly and using Coyote with this JVM option:
-Dcom.sun.enterprise.web.useCoyoteConnector=true
These discussions used to happen off the mailing list because I was
communicating with Sun through our support contract. Now the world gets
to see these discussions. Nobody knows the answer to this problem yet,
so these discussions are the best thing you've got. Not even a Sun
support contract will help your resolve this issue.
I don't use a database in the web tier either, that happens in the EJB
tier. I'm not entirely convinced the problem is caused by database
connections or my application. An older version of the application used
to run on JBoss, same server, for over a year without these issues.
Today the system went down again (9 hours since last time) and I
remembered to try changing the HTTP Listener to see if it would come
back to life like you suggested (new idea to me). I first disabled it,
save, enable it, save. My app came back to life! I have a second HTTP
listener that uses SSL for web services, and disabling/re-enabling made
it come back with a response, but an HTTP 500 response. So I restarted
the app server.
This morning I've told the app server to use Coyote and we'll see how it
goes. I also captured what I can of a thread dump (999 lines max in
Windows console).
Thanks,
Ryan
glassfish_at_javadesktop.org wrote:
> The original issue that this thread was about has nothing whatever to do with database connections or deadlocked threads. My application doesn't even hit a database. The symptoms, and hints as to the source of the problem are to be found in my original post at the top of this thread. I wish the Glassfish team luck fixing that issue, because I very much believe it to be a bug in Glassfish, unlike deadlocked threads which is more likely to be application code.
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=272165
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>
>
>
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProto
col.java:281)
at: com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPS
ervice.java:574)
at: java.lang.Thread.run(Thread.java:619)
Thread "admin_ACCEPT" thread-id 64 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.net.tcp.TcpProtocol.accept(TcpProto
col.java:281)
at: com.sun.messaging.jmq.jmsserver.service.imq.IMQIPService.run(IMQIPS
ervice.java:574)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI RenewClean-[140.95.254.133:1093]" thread-id 62 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock_at_fdb2bd
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI TCP Accept-0" thread-id 61 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)
Thread "ClusterDiscoveryService" thread-id 60 thread-stateRUNNABLERunning in nat
ive
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.service.ClusterDiscoveryService.run
(ClusterDiscoveryService.java:318)
at: java.lang.Thread.run(Thread.java:619)
Thread "Broker Monitor" thread-id 59 thread-stateWAITINGWaiting on lock: java.ut
il.Collections$SynchronizedSet_at_1e1a630
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.messaging.jmq.jmsserver.core.cluster.BrokerConsumers.run(Mu
ltibrokerRouter.java:1342)
at: java.lang.Thread.run(Thread.java:619)
Thread "JMQPortMapper" thread-id 58 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: com.sun.messaging.jmq.jmsserver.service.PortMapper.run(PortMapper.j
ava:485)
at: java.lang.Thread.run(Thread.java:619)
Thread "MQTimer-Thread" thread-id 57 thread-stateTIMED_WAITINGWaiting on lock: j
ava.util.TaskQueue_at_165aab0
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)
Thread "RMI RenewClean-[140.95.254.133:1087]" thread-id 52 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock_at_1a93cd1
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI TCP Accept-0" thread-id 50 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI TCP Accept-8686" thread-id 49 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)
Thread "Thread-12" thread-id 46 thread-stateWAITINGWaiting on lock: java.util.co
ncurrent.locks.AbstractQueuedSynchronizer$ConditionObject_at_1d7f9b0
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje
ct.await(AbstractQueuedSynchronizer.java:1925)
at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.jav
a:317)
at: com.sun.enterprise.management.support.LoaderRegThread.process(Loade
rRegThread.java:243)
at: com.sun.enterprise.management.support.LoaderRegThread.run(LoaderReg
Thread.java:154)
Thread "Thread-11" thread-id 45 thread-stateTIMED_WAITING
at: java.lang.Thread.sleep(Native Method)
at: com.sun.enterprise.management.support.LoaderBase.mySleep(LoaderBase
.java:241)
at: com.sun.enterprise.management.support.Loader$DeferredRegistrationTh
read.run(Loader.java:389)
Thread "Thread-9" thread-id 43 thread-stateWAITINGWaiting on lock: com.sun.corba
.ee.impl.javax.rmi.CORBA.KeepAlive_at_49af7b
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:857)
Thread "p: thread-pool-1; w: 2" thread-id 42 thread-stateRUNNABLERunning in nati
ve
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSo
cketImpl.java:259)
at: com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(
SocketOrChannelAcceptorImpl.java:250)
at: com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerT
hreadImpl.java:107)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThrea
d.run(ThreadPoolImpl.java:555)
Thread "p: thread-pool-1; w: 1" thread-id 41 thread-stateRUNNABLERunning in nati
ve
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSo
cketImpl.java:259)
at: com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(
SocketOrChannelAcceptorImpl.java:250)
at: com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerT
hreadImpl.java:107)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThrea
d.run(ThreadPoolImpl.java:555)
Thread "SelectorThread" thread-id 40 thread-stateRUNNABLERunning in native
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl
.java:274)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelect
orImpl.java:256)
at: sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:13
7)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at: com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:
283)
Thread "Timer-3" thread-id 39 thread-stateTIMED_WAITINGWaiting on lock: java.uti
l.TaskQueue_at_19fbd9e
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)
Thread "Timer-2" thread-id 38 thread-stateWAITINGWaiting on lock: java.util.Task
Queue_at_59e257
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.util.TimerThread.mainLoop(Timer.java:483)
at: java.util.TimerThread.run(Timer.java:462)
Thread "Timer-1" thread-id 37 thread-stateTIMED_WAITINGWaiting on lock: java.uti
l.TaskQueue_at_11ae4ea
at: java.lang.Object.wait(Native Method)
at: java.util.TimerThread.mainLoop(Timer.java:509)
at: java.util.TimerThread.run(Timer.java:462)
Thread "Timer-0" thread-id 36 thread-stateWAITINGWaiting on lock: java.util.Task
Queue_at_aacfa0
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.util.TimerThread.mainLoop(Timer.java:483)
at: java.util.TimerThread.run(Timer.java:462)
Thread "Thread-3" thread-id 35 thread-stateTIMED_WAITING
at: java.lang.Thread.sleep(Native Method)
at: com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClien
t.java:151)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI RenewClean-[140.95.254.133:1079,com.sun.enterprise.admin.server.core
.channel.LocalRMIClientSocketFactory_at_14d044]" thread-id 33 thread-stateTIMED_WAI
TINGWaiting on lock: java.lang.ref.ReferenceQueue$Lock_at_25491c
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCC
lient.java:516)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI Scheduler(0)" thread-id 32 thread-stateTIMED_WAITINGWaiting on lock:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject_at_1e2ef3
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:1
98)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje
ct.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at: java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.t
ake(ScheduledThreadPoolExecutor.java:582)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.t
ake(ScheduledThreadPoolExecutor.java:575)
at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:946)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
or.java:906)
at: java.lang.Thread.run(Thread.java:619)
Thread "GC Daemon" thread-id 30 thread-stateTIMED_WAITINGWaiting on lock: sun.mi
sc.GC$LatencyLock_at_1c5894b
at: java.lang.Object.wait(Native Method)
at: sun.misc.GC$Daemon.run(GC.java:100)
Thread "RMI Reaper" thread-id 29 thread-stateWAITINGWaiting on lock: java.lang.r
ef.ReferenceQueue$Lock_at_1b8aa32
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at: sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at: java.lang.Thread.run(Thread.java:619)
Thread "RMI TCP Accept-0" thread-id 28 thread-stateRUNNABLERunning in native
at: java.net.PlainSocketImpl.socketAccept(Native Method)
at: java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at: java.net.ServerSocket.implAccept(ServerSocket.java:453)
at: java.net.ServerSocket.accept(ServerSocket.java:421)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCP
Transport.java:369)
at: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java
:341)
at: java.lang.Thread.run(Thread.java:619)
Thread "Attach Listener" thread-id 4 thread-stateRUNNABLE
Thread "Finalizer" thread-id 3 thread-stateWAITINGWaiting on lock: java.lang.ref
.ReferenceQueue$Lock_at_adc2bb
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread "Reference Handler" thread-id 2 thread-stateWAITINGWaiting on lock: java.
lang.ref.Reference$Lock_at_1cba429
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:485)
at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
No deadlock found
Command generate-jvm-report executed successfully.