I work with Ryan (rwillie6) and want to report that we believe we have [b]narrowed down the root of our problem, and have resolved our server slowdowns for the time being.[/b]
We recently set our acceptor threads count to 1 (it used to be 5). Unfortunately, when we did this something with lucene caused our threads to lock up permanently so we thought that setting the acceptor threads down to 1 didn't help. However after looking through thread dumps and finding out what lucene was doing we redeployed a version of our app with lucene commented out and have since had fantastic performance, even with just 1 acceptor thread.
However, thinking that the lucene conflict was the root of the problem, we set the acceptor threads back to 5 and discovered that the occasional slowness problem reappeared. After some more testing we discovered that the [b]problem occurs with any acceptor thread value above 1[/b].
I noticed that during the periods of "slowness" when the acceptor threads were greater than 1 our CPU usage on the server was 0%. I made a thread dump during this state. Im just shooting in the dark but I noticed there were two threads there were running sun.nio.ch.FileDispatcher.preClose(). Perhaps a file lock problem? I have attached the full dump so you can take a closer look.
Its still quite a mystery to us why we were able to run with 5 acceptor threads and no problems up until yesterday, especially given that yesterday was not an abnormal day in terms of traffic.
Thanks for all the great help so far. Let us know if we can do anything else to help detail the problem.
[Message sent by forum member 'arjunlall' (arjunlall)]
http://forums.java.net/jive/thread.jspa?messageID=339116