This issue is now closed. Big thanks to Jeanfrancois!
On Tue, Apr 21, 2009 at 3:33 PM, Jeanfrancois Arcand <
Jeanfrancois.Arcand_at_sun.com> wrote:
> Salut,
>
> Marc Novakouski wrote:
>
>> Hi Jeanfrancois,
>> Attached is the result of the netstat operation you were looking for.
>> Rather not post the actual file to the mailing list but discussing it there
>> is fine.
>> Thanks again,
>>
>
> Yes this is clearly some kind of file leaks. It seems all connections open
> are never closed. Can you share a test case? Mainly, it seems you never
> de-register/ close the underlying channel. What is the rule for closing
> connection (I means when are you expecting?).
>
> Also you might want to increase the ulimit of the machine (as it seems
> pretty low right now). I'm sure the issue will go away (or at least
> delayed).
>
> Thanks
>
> -- Jeanfrancois
>
>
> Marc
>>
>> On Tue, Apr 21, 2009 at 1:56 PM, Marc Novakouski <mnovakouski_at_gmail.com<mailto:
>> mnovakouski_at_gmail.com>> wrote:
>>
>> Hi Jeanfrancois,
>>
>> On Tue, Apr 21, 2009 at 1:16 PM, Bill Simons <bill.simons_at_gmail.com
>> <mailto:bill.simons_at_gmail.com>> wrote:
>>
>> ---------- Forwarded message ----------
>> From: Jeanfrancois Arcand <Jeanfrancois.Arcand_at_sun.com
>> <mailto:Jeanfrancois.Arcand_at_sun.com>>
>> Date: Tue, Apr 21, 2009 at 11:30 AM
>> Subject: Re: Having trouble with ZipException/File Descriptor
>> leak in Grizzly
>> To: users_at_grizzly.dev.java.net <mailto:users_at_grizzly.dev.java.net>
>>
>>
>> Salut,
>>
>> Bill Simons wrote:
>> >
>> > Continuing this thread:
>> >
>> http://forums.java.net/jive/thread.jspa?threadID=60501&tstart=0
>> <http://forums.java.net/jive/thread.jspa?threadID=60501&tstart=0>
>> >
>> > We have two filters in the chain:
>> >
>> > public class GameProtocolFilter extends ParserProtocolFilter {
>> >
>> > private GameProtocolParser gameProtocolParser;
>> >
>> > public GameProtocolFilter(GameProtocolParser gpp) {
>> > super();
>> > this.gameProtocolParser = gpp;
>> > }
>> >
>> > public ProtocolParser newProtocolParser() {
>> > return new ProtocolParser() {
>> >
>> > private ByteBuffer buffer = null;
>> > private boolean expectingMoreData = false;
>> > private boolean hasMoreBytesToParse = false;
>> > private Action action = null;
>> >
>> > public boolean isExpectingMoreData() {
>> > return expectingMoreData;
>> > }
>> >
>> > public boolean hasMoreBytesToParse() {
>> > return hasMoreBytesToParse;
>> > }
>> >
>> > public Object getNextMessage() {
>> > return action;
>> > }
>> >
>> > public boolean hasNextMessage() {
>> > if (buffer == null) {
>> > return false;
>> > }
>> >
>> > ByteBuffer dup = buffer.duplicate();
>> >
>> > if (buffer.position() == 0) {
>> > expectingMoreData = true;
>> > return false;
>> > }
>> >
>> > dup.flip();
>> > if (dup.remaining() < 4) {
>> > expectingMoreData = true;
>> > return false;
>> > }
>> > int size = dup.getInt();
>> > if (size == 0 || dup.remaining() < size) {
>> > expectingMoreData = true;
>> > return false;
>> > }
>> >
>> > byte typeByte = dup.get();
>> > dup.position(0);
>> > byte[] testData = new byte[size + 4];
>> > dup.get(testData);
>> >
>> > action =
>> gameProtocolParser.parseAction(typeByte, testData);
>> >
>> > hasMoreBytesToParse = dup.hasRemaining();
>> > return (action != null);
>> > }
>> >
>> > public void startBuffer(ByteBuffer byteBuffer) {
>> > buffer = byteBuffer;
>> > }
>> >
>> > public boolean releaseBuffer() {
>> > buffer = null;
>> > action = null;
>> > expectingMoreData = false;
>> > hasMoreBytesToParse = false;
>> > return false;
>> > }
>> > };
>> > }
>> > }
>> >
>> >
>> > The next filter in our chain dispatches the parsed message to
>> another
>> > thread pool:
>> > public boolean execute(Context ctx) throws IOException {
>> > final Action action = (Action)
>> > ctx.removeAttribute(ProtocolParser.MESSAGE);
>> > final Context workerContext = ctx;
>> > ctx.incrementRefCount();
>> > executorService.execute(new Runnable() {
>> > public void run() {
>> > action.handle(handler, workerContext);
>> >
>> workerContext.getController().returnContext(workerContext);
>> > }
>> > });
>> > return false;
>> > }
>> >
>> > I think these are pretty straightforward but maybe we're
>> missing some
>> > fundamental?
>>
>> Right, I don't see any issue with the above code except it seems
>> the
>> SelectionKey are never set to be canceled like:
>>
>> ctx.setKeyRegistrationState(
>> Context.KeyRegistrationState.CANCEL);
>>
>> Can you gives more information on how the server is stressed
>> (the tool
>> used). Would you be able to share a test case (privately if that
>> help). Also, when the exception happens, can you grab a netstat
>> -an |
>> grep <port-used>?
>>
>> Thanks!
>>
>> -- Jeanfrancois
>>
>>
>> We stress the server with a custom tool which does the following:
>> 1. The tool creates 50 threads.
>> 2. Each thread opens one SocketChannel to the server and begins
>> communication.
>> Note 1: The tool is run on a different computer than the server, so
>> the communication is over the network instead of local.
>> 3. After roughly 200-300 message/response actions (client thread
>> sends message, server responds), the thread finishes execution and
>> exits. Note 2: There is no close operation currently performed when
>> the
>> thread exits
>> 4. The tool performs steps 1-3 repeatedly until either 2000 threads
>> have run successfully or the server is unable to finish serving
>> requests for the existing threads.
>>
>> The failure case we've observed is that somewhere either in the
>> execution of threads 850-900 or 900-950 the server starts spewing
>> the aforementioned error to the log until we run out of disk space.
>>
>> I'm running the test now, it takes a little while to get to the
>> failure state. When I see the error occur I'll run the netstat and
>> let you know.
>>
>> Thanks for your help!
>>
>>
>>
>>
>>
>> >
>> > We're using a grizzly controller with a DefaultPipeline and
>> > BaseSelectionKeyHandler.
>> >
>> > Thanks for your help.
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail:
>> users-unsubscribe_at_grizzly.dev.java.net
>> <mailto:users-unsubscribe_at_grizzly.dev.java.net>
>> > For additional commands, e-mail:
>> users-help_at_grizzly.dev.java.net
>> <mailto:users-help_at_grizzly.dev.java.net>
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> <mailto:users-unsubscribe_at_grizzly.dev.java.net>
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>> <mailto:users-help_at_grizzly.dev.java.net>
>>
>>
>>
>>