Summary:
I'm getting intermittent CORBA connection timed out problems, JNDI lookup problems, and slowness with an EJB 3 app with swing app clients. My expectation is that it should rarely time out, and rarely have jndi lookup problems. So where do I start with tuning?
Details:
Things seem to be working alright for the most part, but occasionally there are CORBA failures with connection timed out. There are also occasional, sometimes frequent, JNDI NamingExceptions happening (ejb ref resolution error for remote business interface <classname> [Root exception is java.lang.NullPointerException]). And there is slowness sometimes for requests. The app is EJB 3 with two session beans and a handful of entity beans. It does what I think are pretty standard remote method calls passing only the server ip address to the InitialContext, and passing back small to medium data sets. There are also JMS clients connecting to the server, along with some Rest web services.
I pulled a list of open connections on the linux box and there are some client ip addresses with 15-20 "established" connections on port 3700. One client machine in particular had 30 connections on port 3700. However, going to that client machine, it only had four connections showing with netstat -a. Shutting down the application on that machine dropped four connections, leaving the server showing 26 connections.
Why are these connections left open? Does it matter?
Should a client typically only have one RMI/IIOP connection open, even when connecting to multiple session beans, jms queues, etc?
Are there changes we could make to the ORB timeouts that might make a difference?
What else should I look at to improve the situation?
Some other things I have looked at are the JDBC connection pool is only using a few of the 60 max connections, and the thread pool is only using a few of its 200 max.
Here are the system stats and configs I know to be relevant:
Sun GlassFish Enterprise Server v2.1 (9.1.1) (build b60e-fcs)
Not clustered, I think it's the developer profile at the moment.
There are an average of 200 users.
Gentoo Linux - uname gives: 2.6.20-gentoo-r8 #4 SMP Sat Jun 7 14:13:51 MDT 2008 x86_64 Intel(R) Xeon(R) CPU E5410 @ 2.33GHz GenuineIntel GNU/Linux.
Dual quad core with 24GB ram.
MySQL 5 with isam tables on a similar separate machine.
ORB is set to default 1024 connections, and we have reached that limit, which two about two days after a restart.
couple highlights of domain.xml:
<jvm-options>-Xmx2560m</jvm-options>
<jvm-options>-server</jvm-options>
Please help me further my Glassfish education. Any tips are appreciated.
[Message sent by forum member 'culli' (culli)]
http://forums.java.net/jive/thread.jspa?messageID=357260