Hi,
which version of GlassFish are you using? For the information you
provided, it seems GlassFish might not be able to allocate a file
descriptor because others application doesn't seems to close their
connection (the CLOSE_WAIT). On which port GF is listening for http
request? Can you do a netstat -an | grep <<port>> once that happen to
see if the leak is from the http port or some other components.
One thing you might try is to add, in domain.xml, the following property
to see if that makes a difference:
<jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=false</jvm-options>
Thanks
-- Jeanfrancois
glassfish_at_javadesktop.org wrote:
> I have a web application that responds to http requests and has a few threads running in the background. The background threads continue to run long after the application has stopped responding to http requests. It takes awhile for this to happen, hours or days. From the logs it appears that http requests are simply no longer reaching the application. Just before the application stops responding a bunch of errors appeared in my logs like the one below. The appearance of these errors corresponds closely to the problem appearing. What is the app server doing? Could this be related to my problem? Is there something in the app server configuration I could be trying to resolve this?
>
> I've also including the output of a netstat command on the server.
>
> SEVERE|24|2008-03-30 07:56:02:081|com.sun.enterprise.deployment.autodeploy.AutoDeployControllerImpl$AutoDeployTask.run(): enterprise.deployment.backend.autoDeploymentFailure
> WARNING|25|2008-03-30 07:56:03:312|com.sun.enterprise.server.logging.SystemOutandErrHandler$LoggingByteArrayOutputStream.flush():
> java.lang.NullPointerException
> at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1044)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:330)
> at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
> at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:994)
> at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:227)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
> at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
> at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
> at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
> at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
> at javax.management.timer.Timer.sendNotification(Timer.java:1234)
> at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1203)
> at javax.management.timer.TimerAlarmClock.run(Timer.java:1286)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
>
> output from netstat
>
> root_at_localhost:~# netstat
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 ip-10-251-91-146.ec:www pool-141-155-18-1:50163 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www pool-72-79-189-18:50464 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www william.goodenoug:52560 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 70-91-85-201-washi:2181 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 75.141.45.150:2245 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 78.19.100.142:1223 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www cpe-65-24-208-235:34060 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www hlndoacache1.mt.g:47161 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www ip68-227-184-232.:42330 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-24-61-133-241.hs:2307 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-69-136-56-36.hs:52420 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www hlndoacache1.mt.g:33444 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-24-18-203-234.hs:2481 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www astound-66-234-198:3177 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www e179158028.adsl.al:1047 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 208.188.205.155:2224 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 4143_acl1.gw.smart:3953 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www gv-dialup-4-176.i:34233 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www rrcs-71-43-151-20:29198 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-69-250-197-188.:64098 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www dynamic.70-169-17:61328 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www d51538AB6.access.:55832 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www customer1.aao.org:15675 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www dD5763197.access.:33535 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www cpe-68-172-94-133.:2702 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-67-170-56-158.hs:2571 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-71-234-76-54.hsd:3387 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www pool-71-120-238-17:1625 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 203-214-86-111.dyn:1119 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www C-61-68-178-154.b:51021 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 89.1.51.183.dynami:1337 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-76-23-65-65.hsd1:1684 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www ip51ce538b.speed.:59095 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www c-67-180-217-82.hs:2124 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 207.61.123.57:52504 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 213.63.176.71:50039 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 317104228.telebar:57344 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www user-142g26k.cabl:33780 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www 79-117-201-26.rds:54345 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www seifert.plus.com:1852 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www hlndoacache1.mt.g:26773 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www seifert.plus.com:1852 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www hlndoacache1.mt.g:26773 SYN_RECV
> tcp 0 0 ip-10-251-91-146.ec:www delta9.student.utw:3416 SYN_RECV
> tcp 201 0 ip-10-251-91-146.ec:www 78.13.11.82:1125 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 66-240-64-131.atl:50441 CLOSE_WAIT
> tcp 0 372 ip-10-251-91-146.ec:ssh 75-146-62-233-Was:60286 ESTABLISHED
> tcp 308 0 ip-10-251-91-146.ec:www pool-71-127-255-18:1716 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 81.8.159.190:59711 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-69-138-22-65.hs:51911 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www mail.ghkco.com:32525 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www ool-18b8b4e5.dyn.:62961 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www william.goodenoug:47465 CLOSE_WAIT
> tcp 0 0 ip-10-251-91-146.:57420 72.21.203.132:https TIME_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-68-38-134-69.hs:62187 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 64-194-89-2.wcg.n:50923 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www CPE001a92cb1dad-CM:1318 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-24-12-224-131.h:39429 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 79-113-232-223.rds:1839 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www adsl-75-11-145-20:56331 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 201.216-66-87.ads:56850 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-71-205-112-45.hs:3806 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 24.137.36.69:1703 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www cac94-10-88-170-23:1222 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 201-130-248-45-cab:4847 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 149-40-249.oke2-b:55549 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www adsl-ull-64-37.49-:4074 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www cpe-24-242-46-175:25297 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 54.Red-88-14-113.d:4072 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www adsl-69-153-101-18:2020 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www pool-162-84-202-1:50315 CLOSE_WAIT
> tcp 0 0 ip-10-251-91-146.ec:ssh 75-146-62-233-Was:54741 ESTABLISHED
> tcp 308 0 ip-10-251-91-146.ec:www 125-98-235-201.fib:1893 CLOSE_WAIT
> tcp 307 0 ip-10-251-91-146.ec:www c-24-19-239-124.h:63767 ESTABLISHED
> tcp 308 0 ip-10-251-91-146.ec:www 0x5551c0cd.adsl.c:11721 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www adsl-75-7-199-178:10230 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www cpe-71-65-82-223.w:4678 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-65-34-132-137.h:63399 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-98-203-126-171.h:1938 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www p54B1CF08.dip.t-d:62084 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www adsl-65-67-115-236:1066 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www xdsl-83-150-81-20:46581 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 75.141.45.150:3814 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 71-83-94-173.dhcp:63095 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 108-82-252-216-sta:2810 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 78.162.10.192:3158 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 137.80-203-49.nex:50509 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www adsl-dynamic-pool:50447 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 78-69-202-128-no1:63530 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-75-67-240-124.hs:4067 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www host-84-221-79-221:1634 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www host-84-221-79-221:1634 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www adsl-70-232-97-51.:4284 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-24-8-197-67.hsd:37884 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-24-99-22-144.hsd:2717 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 90.210.19.21:1206 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www bas1-toronto12-108:1202 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www ip565cca8c.direct-:1662 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 32.212-66-87.adsl:62419 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www a50192.upc-a.chell:1083 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 15.caba27.xdsl.nau:1592 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 209.33.252.4:60556 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www s5592bcc4.adsl.wa:56496 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www ip68-101-174-48.s:33721 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-24-19-203-45.hsd:4278 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www chmuglab1b.chem.ps:2884 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www 90-229-208-68-no1:52277 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 24-217-109-209.dhc:1071 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-69-138-201-21.hs:2585 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www sac91-1-89-85-92-1:1355 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www c-76-112-191-55.hs:3400 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www host103-209-dynami:4361 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-24-16-87-21.hsd1:2538 CLOSE_WAIT
> tcp 340 0 ip-10-251-91-146.ec:www pool-96-232-215-1:50064 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 82-40-120-220.cabl:4649 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www cpe-71-72-57-127.n:3237 CLOSE_WAIT
> tcp 200 0 ip-10-251-91-146.ec:www d51537131.access.:49379 ESTABLISHED
> tcp 342 0 ip-10-251-91-146.ec:www ti0117a380-1363.b:56375 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www aolclient-24-33-80:1733 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www ip5659bc18.speed.:61483 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www pool-72-72-17-115:33279 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www pool-72-79-114-48.:3541 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www dsl-202-138-212-20:2896 CLOSE_WAIT
> tcp 200 0 ip-10-251-91-146.ec:www cc191256-b.assen1.:3375 ESTABLISHED
> tcp 201 0 ip-10-251-91-146.ec:www cc546786-e.gorre1.:1324 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www 72-55-206-206.mam:51455 CLOSE_WAIT
> tcp 308 0 ip-10-251-91-146.ec:www host86-155-126-17:58011 CLOSE_WAIT
> tcp 201 0 ip-10-251-91-146.ec:www c-69-254-200-132.:32892 CLOSE_WAIT
> Active UNIX domain sockets (w/o servers)
> Proto RefCnt Flags Type State I-Node Path
> unix 2 [ ] DGRAM 648 @/com/ubuntu/upstart
> unix 5 [ ] DGRAM 6476 /dev/log
> unix 2 [ ] DGRAM 844 @/org/kernel/udev/udevd
> unix 2 [ ] DGRAM 34002
> unix 2 [ ] DGRAM 6632
> unix 2 [ ] DGRAM 6586
> unix 2 [ ] STREAM CONNECTED 6185
> [Message sent by forum member 'kevinmacdonald' (kevinmacdonald)]
>
> http://forums.java.net/jive/thread.jspa?messageID=266930
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>