I had this problem once a few months ago. The default limit on open files in Linux is rather low, 1024 I think it was.
Check your open files limit with
ulimit -Hn # shows hard limit
ulimit -Sn # shows soft limit
To fix the problem I added two lines to the /etc/security/limits.conf file.
<user_id_for_running_glassfish> hard nofile 32768
<user_id_for_running_glassfish> soft nofile 32768
Obviously change <user_id_for_running_glassfish> to the value that is appropriate for your installation.
I don't remember if I rebooted the box or not. I don't think so. I think starting a new shell afterwards was sufficient. Check your limits again with ulimit to know that your change was successful:
[paulr_at_pascrpdapp02a ~]$ ulimit -Sn
open files (-n) 32768
The number-of-open-files limit was increased to 32768 and I haven't had a problem since. After I increased the limit I looked for a leak but there wasn't one. Glassfish just uses a lot of open files.
[Message sent by forum member 'paulr5930' (paulr5930)]
http://forums.java.net/jive/thread.jspa?messageID=234414