Thanks - Actually that helps.
After many hours of trial and error I think I have some answers. The problems im seeing is two-fold
1.) The unexpected end of file/connection timeouts are definately from overloading the server. I wouldnt have thought 500 messages could kill my server this easily (i have eliminated things such as alfresco) but I do notice when i connect to the server it is very slow. I am fairly new to glassfish myself and the project I am working on is over 100 services and EJB's. If I submit 500 files into my system some services call other services and expect a response. As each file moves further along deeper into other services, specific operations are performed and it moves on until you get 3 or 4 services waiting on responses in a chain. I am making a guess that all of these hove opened connections and are waiting. Finally I have a pick/onalarm activity which may have to wait up to 30 minutes or longer and I am making another guess that all of these connections then have to wait until the pick times out or recieves a message??? I have not proved this yet so if anyone knows better please correct me if I am wrong. This I can definately fix with throttling and config changes.
I guess a better question is if a message enters Service A which calls Service B which then calls Service C with a pick/onalarm and all services are waiting on each other is that consuming 3 threads/connections?
2.) The second issue which is much worse is that when sending 500 or so messages through the system they sometimes lockup (from watching the logs) and notifications coming back to the "pick" activity are not recieved ... its like the system is locked up but I can still enter into the admin web gui. As for this I have figured out that I have to increase the http binding thread size to a very large number ... i.e. I get lockups if the outbound number is at 200 or so and no lockups if this is at 1000 or more. Im not positive but i think i may have to increase the bpel thread setting as well (or at least i noticed a huge performance increase when setting this higher - im just worried this is causing part of my problems setting it high) from examining netstat it looks like this is the case since on lockup I have almost 2000 connections in ESTABLISHED mode on 9080
I am definately a lot closer - i think the best action is to begin looking at clustering after I understand this a little better. Each test I run after i changes settings takes about an hour to stop and clear the old test and begin again so its time consuming work. ;)
[Message sent by forum member 'cornwellsb' ]
http://forums.java.net/jive/thread.jspa?messageID=375471