>> When the client (browsers) would attempt to access a page, it would
>> sit in "Waiting for response from web server..." forever, and people
>> would just close the browser. That is probably why we see those
>> errors.
>
> Right. If that's the case, it seems all threads are
> consumed/deadlocked. Most of the time this has nothing to do with the
> WebContainer, but with a db connection that is locked/slow. All
> Servlet try to get a db connection and they all wait of a lock. It
> happens when a db is slow usually...
Are you saying all of the previous transactions that completed
successfully are locking their threads on each page request? I'm
trying to understand how all of my database access code (JPA) that is
working fine is causing servlet threads to deadlock.
We use JDBC connection pools in GlassFish. I have two pools for SQL
Server 2005 databases, and one for a postgres database. The SQL Server
database uses the XADataSource, but with transactions disabled. I
don't remember the exact reason, other than this was the only way I
could get it to work when GlassFish started to enable distributed
transactions with JAX-WS by default. I could try changing that to
ConnectionPoolDataSource again, but after we get thread dumps.
> For sure getting a thread dump will tell us where all the thread dead
> lock. I don't think this is related to the nbpool....I bet you a dead
> lock with a bd connection :-) I will pay @ JavaOne if I'm wrong :-) :-)
Unfortunately I won't be at JavaOne this year. I was there last year
and attended one of your sessions. Anyway, I'll see what I can do to
get a thread dump. I like Scott Oak's idea about using :
asadmin generate-jvm-report --type=thread
Since the thread dump is so huge, it will probably go past the Windows
console buffer length. I'm trying to figure out how to pipe it to a
file. asadmin always asks me to login. I use --user admin but it
wants a password. I used asadmin login --host localhost --port 4848
to set up .asadminpass in my home directory, then ran the
generate-jvm-report command again and it still wants my username and
password. If I give it --user admin, it still wants a password. The
largest buffer size for the command prompt Windows will allow is 999.
>> Do you have a tool we can install on our test server to put the kind
>> of load you described on the system? I do not like the idea of doing
>> that on our production system. When you said it breaks, does the app
>> server not return any data when trying to access pages?
>
> No it throw an IOException. When the nbpool problems happens, the
> entire TCP stack is down on windows. Anything you will try to do on it
> will not work (reboot is the only operation). But I'm convinced this
> is not the nbpool.
>
>
> Does restarting the
>> app server solve the problem, or do you have to reboot Windows? For
>> us, just restarting the service solves the problem. We only reboot
>> once per month when installing Windows updates.
>
> That's means you aren't suffering the nbpool.
Good, I've been thinking the same.
>> console to interact with it. I like your suggestion of disabling
>> Grizzly from an other email:
>>
>> -Dcom.sun.enterprise.web.useCoyoteConnector=true
>
> You gonna face the same problem I suspect, or some requests will be
> dropped by Coyote if all the thread gets dead locked like It looks
> like. The difference between Grizzly & Coyote is Coyote drops requests
> (404), Grizzly queue them until it reach the max (4096 connection in
> the queue), and then close connection as well. The problem is if all
> Grizzly threads deadlock, then issuing a request will exactly produce
> what you are seeing, which is a browser spin. Now based on you
> exception, the queue is executed and Grizzly try to write response on
> closed connection. That means a thread has been released and grizzly
> is using it. So your Servlet seems to starts executing really slowly.
> We need to find why :-)
Ok I'll focus on getting a dump.
Thanks,
Ryan