users@glassfish.java.net

Re: GFv2_b45: OpenFileDescriptorCount is growing, growing and eventually server crashes...

From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_Sun.COM>
Date: Wed, 27 Jun 2007 16:25:42 -0400

Hi,

Witold Szczerba wrote:
> The bug, you mentioned was how to obey Solaris limit of max 256 opened
> files. In my case, the problem is not that there are too many open
> files, but the fact that the number of opened files is growing
> constantly.
> On this forum, there were threads about similar problem, but it was
> about WebServices/other HTTP connections. My client applications does
> not use WebServices but IIOP and HTTP is used only to download
> application using WebStart. It looks like the longer/the more clients
> are using Glassfish the more files are being opened and never closed.
>
> I managed how to increase the OpenFileDescriptorCount limit. Now it is
> 4096. I will see if this limit is going to be overrun eventually. The
> 1024 was reached always before 20-30 hours of using (depending on how
> many clients were using it and for how long). I am worry that the new
> limit will not solve the problem, but only increase the period before
> failures :/
>

I agree :-)...that make little sense to increase the fd limit on
Solaris...looks like we have a fd leak somewhere. Can you file an issue
here:

https://glassfish.dev.java.net/servlets/ProjectIssues

Do you think you can come with a test case that demonstrate the problem?
  One thing to add to the bug description is the output of 'netstat -an
| grep <<http|iiop_port>>' to see the state of the fd opened. From you
lsof dump, the socket state cannot be found:

> java 25533 sop 265r FIFO 0,6 1246867 pipe
> java 25533 sop 266w FIFO 0,6 1246867 pipe


Do you have an idea about which protocol is producing the leak (IIOP or
HTTP)? Hopefully IIOP (just kidding :-))...netstat -an will tell which
one is leaking. Which JDK version are you using (because such leak was
present in JDK 1.5_01 to _09.

Thanks for reporting this!

-- Jeanfrancois


>
> 2007/6/27, Binod <Binod.Pg_at_sun.com>:
>> http://forums.java.net/jive/thread.jspa?messageID=216741
>>
>> Your issue seem to be as described in the above thread. Look like
>> you will need to set the limit to 2048 to escape the JDK bug
>>
>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6533291
>>
>> thanks,
>> Binod
>>
>> Witold Szczerba wrote:
>>
>> > Hi there,
>> > I have a major issue with Glassfish v2 beta3 b45. My project is going
>> > into last state, right now our customer is testing it on his own
>> > server and we are encountering problems with
>> > java.io.IOException: Too many open files.
>> >
>> > I really do not know what to do about this, I am monitoring server
>> > process using jconsole. When server starts, its
>> > OpenFileDescriptorCount < 270, and then it is growing constantly.
>> > After less than 24 hours the 1024 limit is overrun and server stops
>> > responding, it is filling logs with stacktrace full of IOExceptions
>> > and CPU is 100% on some single thread.
>> >
>> > Today, I have registered opened files using lsof 3 times like this:
>> > lsof |grep 25533 > lsof-2007-06-27-.....
>> >
>> > Here are the files:
>> > http://josh.neostrada.pl/lsof/lsof-2007-06-27-09h20m.txt
>> > http://josh.neostrada.pl/lsof/lsof-2007-06-27-12h32m.txt
>> > http://josh.neostrada.pl/lsof/lsof-2007-06-27-14h00m.txt
>> >
>> > The first file was made at 9:20am, it has 411 lines,
>> > second one (12:32pm) has 645 and
>> > last (2:00pm) one has 888 lines.
>> >
>> > What is wrong?
>> >
>> > About the enviroment:
>> > Ubuntu 7.04,
>> > Linux sop 2.6.20-15-server #2 SMP Sun Apr 15 07:41:34 UTC 2007 i686
>> > GNU/Linux
>> > Java(TM) SE Runtime Environment (build 1.6.0-b105)
>> > Glassfish v2 beta3 b45
>> >
>> > The application is using EJB3 + JPA, client application (application
>> > client container) is a desktop Swing, downloaded by Java WebStart.
>> >
>> > If there is anything more I should provide, please tell me
>> >
>> > Thanks,
>> > Witold Szczerba
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> > For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: users-help_at_glassfish.dev.java.net
>