Hi John,
John C. Turnbull wrote:
> Hi Jeanfrancois,
>
> The problem is with UR1 and it has not yet been resolved. After about an
> hour, any Comet data sent from the server appears to go nowhere as it is not
> received at the applet even though there are no exceptions at either end.
> My analysis has revealed that the problem only occurs over the actual
> internet (not the LAN) and appears to be related to the server sending
> packets to the wrong port.
Sorry I forgot that part...I'm convinced the problem is not with
GlassFish, but with your firewall/proxy. Grizzly cannot set the
underlying port of a Socket (connection). When Grizzly try to write, it
use what the JDK/OS is passing, which is a Socket bind to a port. So
when it try to write, it always means a client has made a connection
Grizzly.
Now just to make sure, can you add in domain.xml:
<jvm-options>-Dcom.sun.enterprise.server.ss.ASQuickStartup=false</jvm-options>
and see if the problem disappear? 99% of the time adding that property
fix a lot of trouble in GlassFish.
Thanks
-- Jeanfrancois
>
> Thanks,
>
> John
>
>> -----Original Message-----
>> From: Jeanfrancois.Arcand_at_Sun.COM [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>> Sent: Thursday, 31 January 2008 03:24
>> To: users_at_grizzly.dev.java.net
>> Subject: Re: Comet data into the bit bucket
>>
>> Hi John,
>>
>> John C. Turnbull wrote:
>>> Hi Jeanfrancois,
>>>
>>> I would like to test the change you have made but I am not sure how
>> to do
>>> it. I have been using GlassFish v2 UR1 and not the standalone
>> Grizzly up to
>>> this point so I guess I need to try it with the latter now. Is that
>>> correct?
>> Pouua sorry I was completely confused when I asked that :-) Just to
>> make
>> sure I follow, what was the time out issue you are seeing? Was it with
>> v2 or v2 ur1? Is the problem fixed now?
>>
>> Thanks
>>
>> -- Jeanfrancois
>>
>> Does that mean I need to use the Grizzly Web Server instead of
>>> GlassFish?
>>>
>>> I have not used standalone Grizzly yet so I am unsure how to proceed
>> and
>>> deploy etc.
>>>
>>> Thanks,
>>>
>>> John
>>>
>>>> -----Original Message-----
>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>> Sent: Wednesday, 30 January 2008 07:21
>>>> To: users_at_grizzly.dev.java.net
>>>> Subject: Re: Comet data into the bit bucket
>>>>
>>>> Hi John,
>>>>
>>>> John C. Turnbull wrote:
>>>>> Hi Jeanfrancois,
>>>>>
>>>>> OK, firstly I just start GlassFish by using
>>>>>
>>>>> asadmin start-domain domain1
>>>>>
>>>>> Is that correct?
>>>> Yes.
>>>>
>>>>> I am about to rewrite parts of my application so I will wait and
>> see
>>>> if the
>>>>> problem persists after I have the new version working before
>> posting
>>>> code.
>>>>> At the moment, I am much more concerned about the AV and firewall
>>>> issues I
>>>>> refer to in my other post. Do you have any comments on that issue?
>>>> I need to read a little but I know there is issue with streaming and
>>>> firewall. Will come back on that one.
>>>>
>>>> As for your timeout problem in Grizzly standalone, I think I have
>> found
>>>> the problem (will commit the fix now). Give it a try and let me know
>> if
>>>> you have a chance.
>>>>
>>>> Thanks!
>>>>
>>>> -- Jeanfrancois
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> John
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>>>> Sent: Tuesday, 29 January 2008 02:15
>>>>>> To: users_at_grizzly.dev.java.net
>>>>>> Subject: Re: Comet data into the bit bucket
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> John C. Turnbull wrote:
>>>>>>> Hi Jeanfrancois,
>>>>>>>
>>>>>>> I will perform a more detailed analysis of the packet transfers
>> but
>>>>>> one
>>>>>>> thing that I do now know is that the problem does not occur when
>> I
>>>>>> run
>>>>>>> GlassFish on my local LAN and access it from another machine on
>> the
>>>>>> LAN. I
>>>>>>> have managed to keep the connection viable for 3 days so far
>>>> without
>>>>>> any
>>>>>>> activity so it seems that the problem only occurs when I run the
>>>>>> server over
>>>>>>> the actual internet. I am not sure what this suggests/indicates.
>>>>>> Great analysis :-) That means the problem is with Grizzly
>>>> only....Can
>>>>>> you refresh my mind and explain how you start Grizzly, and what
>> your
>>>>>> test is doing? Just a couple of lines to see if I can spot
>> something
>>>>>> inside Grizzly code.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> -- Jeanfrancois
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>>>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>>>>>> Sent: Thursday, 24 January 2008 12:33
>>>>>>>> To: users_at_grizzly.dev.java.net
>>>>>>>> Subject: Re: Comet data into the bit bucket
>>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> quite good analysis...
>>>>>>>>
>>>>>>>> John C. Turnbull wrote:
>>>>>>>>> Hi Jeanfrancois,
>>>>>>>>>
>>>>>>>>> OK, I have a lot of new information that hopefully will allow
>> us
>>>> to
>>>>>>>> finally
>>>>>>>>> get to the bottom of this problem.
>>>>>>>>>
>>>>>>>>> After running packet sniffers on both the server and client and
>>>>>> also
>>>>>>>> by
>>>>>>>>> running the netstat command, I have determined the following
>>>>>> things:
>>>>>>>>> (1) The exact period of idle time seems to be much more than 10
>>>>>>>> minutes and
>>>>>>>>> more like an hour although I cannot be certain if this is fixed
>>>> or
>>>>>>>>> pseudo-random.
>>>>>>>>> (2) The server is definitely sending data to the IP address of
>>>> the
>>>>>>>> client
>>>>>>>>> machine before and after the idle time.
>>>>>>>>> (3) The client machine is receiving data from the server's IP
>>>>>> address
>>>>>>>> before
>>>>>>>>> and after the idle time, even though it is not actually getting
>>>>>> into
>>>>>>>> the
>>>>>>>>> applet.
>>>>>>>>> (4) The server is using the same DataOutputStream to write the
>>>> data
>>>>>>>> before
>>>>>>>>> and after the idle time.
>>>>>>>>> (5) Before the idle time, the connections on the client look
>> like
>>>>>>>> this:
>>>>>>>>> TCP 10.1.1.3:63457 1.2.3.4:ssh ESTABLISHED
>>>>>> InHost
>>>>>>>>> TCP 10.1.1.3:63940 1.2.3.4:8080 ESTABLISHED
>>>>>> InHost
>>>>>>>>> Where 1.2.3.4 is the mock address of the server machine. That
>>>>>> looks
>>>>>>>> fine
>>>>>>>>> but I don't know why there's an SSH connection in there
>> (perhaps
>>>>>> you
>>>>>>>> could
>>>>>>>>> explain that).
>>>>>>>> No idea for ssh...
>>>>>>>>
>>>>>>>> Anyway, note that the port in the connection to the server's
>>>>>>>>> 8080 port is 63940.
>>>>>>>>>
>>>>>>>>> (6) Immediately after the idle time, the connections on the
>>>> client
>>>>>>>> look like
>>>>>>>>> this:
>>>>>>>>>
>>>>>>>>> TCP 10.1.1.3:63940 1.2.3.4:8080 ESTABLISHED
>>>>>> InHost
>>>>>>>>> TCP 10.1.1.3:64892 1.2.3.4:8080 FIN_WAIT_1
>>>>>> InHost
>>>>>>>>> TCP 10.1.1.3:64898 1.2.3.4:8080 FIN_WAIT_1
>>>>>> InHost
>>>>>>>>> Note that we still have the original connection but now we have
>> 2
>>>>>> new
>>>>>>>>> connections in FIN_WAIT_1 state (whatever that means) on
>>>> different
>>>>>>>> ports.
>>>>>>>>
>>>>>>>> That means two connections have been closed, and the file
>>>> descriptor
>>>>>>>> (socket) will eventually disappear.
>>>>>>>>
>>>>>>>>> (7) Wireshark reveals that the packets of data arriving on the
>>>>>> client
>>>>>>>> after
>>>>>>>>> the idle time are directed to port 63494 which is not the port
>> of
>>>>>> any
>>>>>>>> of the
>>>>>>>>> above connections. This must be why the applet is not
>> receiving
>>>>>> them
>>>>>>>> as it
>>>>>>>>> is still established on a connection on port 63490.
>>>>>>>> But that's strange, because NIO (not Grizzly) cannot change the
>>>>>> remote
>>>>>>>> port of SocketChannel. That means the connection has probably
>> been
>>>>>>>> closed on the client side and reopened (with a new port). What
>> is
>>>>>>>> puzzling me is Grizzly will wakes when the connection on the
>>>> client
>>>>>>>> side
>>>>>>>> is closed, and call onTerminate().
>>>>>>>>
>>>>>>>>
>>>>>>>>> So there you have it. Everything is functioning correctly
>> except
>>>>>>>> that the
>>>>>>>>> data after the idle time appears to be sent to the wrong port.
>> I
>>>>>>>> can't find
>>>>>>>>> any other anomaly.
>>>>>>>> Looks like there is a timeout, maybe from the Applet or the
>>>> browser,
>>>>>>>> and
>>>>>>>> the connection gets re-openned, hit your Servlet again and comet
>>>>>> gets
>>>>>>>> re-enabled on that connection (because you are seeing some
>> push).
>>>>>> Can
>>>>>>>> you grab what is exchanged between the client and the server? I
>>>>>> think
>>>>>>>> that's the only way we can find something.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> -- Jeanfrancois
>>>>>>>>
>>>>>>>>
>>>>>>>>> Any ideas why the destination port is changing after the idle
>>>> time?
>>>>>>>> Can you
>>>>>>>>> spot anything else?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>>>>>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>>>>>>>> Sent: Wednesday, 23 January 2008 05:17
>>>>>>>>>> To: users_at_grizzly.dev.java.net
>>>>>>>>>> Subject: Re: Comet data into the bit bucket
>>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>>
>>>>>>>>>> John C. Turnbull wrote:
>>>>>>>>>>> Hi Jeanfrancois,
>>>>>>>>>>>
>>>>>>>>>>> I have discovered that when the problem occurs, there is no
>>>>>>>> interrupt
>>>>>>>>>>> detected so it would appear that the connection is still open
>>>>>>>> (which
>>>>>>>>>>> correlates with the fact that there are no EOF exceptions at
>>>>>> either
>>>>>>>>>> end).
>>>>>>>>>>
>>>>>>>>>> Just in case, are you inside a firewall or a proxy? Can it be
>>>> the
>>>>>>>> proxy
>>>>>>>>>> that close the connection (I suspect not).
>>>>>>>>>>
>>>>>>>>>>> Hmm, so where do I go from here? The connection remains open
>>>> but
>>>>>>>> no
>>>>>>>>>> data is
>>>>>>>>>>> getting through.
>>>>>>>>>> From your CometHandler, invoking the
>>>>>>>>>> HttpServletResponse.getWriter().write(...) succeed, right? I
>>>> guess
>>>>>>>> you
>>>>>>>>>> should install wireshark or ngrep to see if the bytes are sent
>>>>>>>>>> properly.
>>>>>>>>>> This way we will know if the client or server has problems.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> -- Jeanfrancois
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Jeanfrancois.Arcand_at_Sun.COM
>>>>>>>>>> [mailto:Jeanfrancois.Arcand_at_Sun.COM]
>>>>>>>>>>>> Sent: Wednesday, 23 January 2008 03:37
>>>>>>>>>>>> To: users_at_grizzly.dev.java.net
>>>>>>>>>>>> Subject: Re: Comet data into the bit bucket
>>>>>>>>>>>>
>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>
>>>>>>>>>>>> John C. Turnbull wrote:
>>>>>>>>>>>>> Hi Jeanfrancois,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your reply. Comments inline.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have Comet working nicely between servlets and applets
>>>>>> except
>>>>>>>>>> for
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>> thing.
>>>>>>>>>>>>>> Great!
>>>>>>>>>>>>> [JCT] Yes, indeed. I am very happy with Grizzly Comet :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>> For some reason, if a connection is left idle for
>> several
>>>>>>>>>>>>>>> minutes then the data sent from the servlet seems to
>>>>>> disappear
>>>>>>>>>> into
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> ether somewhere.
>>>>>>>>>>>>>> Do you know how long exactly?
>>>>>>>>>>>>> [JCT] Not exactly but it's about 10 minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you have a test case I can take a look at? Do you know
>> if
>>>>>> the
>>>>>>>>>>>>>> connection is still active (are you handling the
>>>>>>>>>>>>>> onInterrupt/onTerminate
>>>>>>>>>>>>>> method of the CometHandler)?
>>>>>>>>>>>>> [JCT] It is difficult to provide a test case at this stage
>>>> but
>>>>>> I
>>>>>>>>>> can
>>>>>>>>>>>> say
>>>>>>>>>>>>> that I am not doing anything with onInterrupt or
>> onTerminate
>>>>>> yet
>>>>>>>> as
>>>>>>>>>> I
>>>>>>>>>>>> am not
>>>>>>>>>>>>> sure how these methods should be used. Is there
>>>> documentation
>>>>>>>> for
>>>>>>>>>>>> Comet
>>>>>>>>>>>>> somewhere? I mean something that gives a general outline
>> of
>>>>>> how
>>>>>>>>>>>> everything
>>>>>>>>>>>>> hangs together? It's a bit hit-or-miss for me at the
>> moment
>>>>>> and
>>>>>>>> I
>>>>>>>>>> am
>>>>>>>>>>>>> finding things out by experimentation.
>>>>>>>>>>>> That's the general problem with Grizzly...lack of
>>>>>> documentations.
>>>>>>>>>> The
>>>>>>>>>>>> only place I talk about it is here:
>>>>>>>>>>>>
>>>>>>>>>>>>
>> http://weblogs.java.net/blog/jfarcand/archive/2006/07/the_grizzly_com.h
>>>>>>>>>>>> tml
>>>>>>>>>>>>
>> https://grizzly.dev.java.net/nonav/xref/com/sun/grizzly/comet/CometHand
>>>>>>>>>>>> ler.html
>>>>>>>>>>>>
>>>>>>>>>>>> which is far from perfect. Mainly, the onInterrupt will be
>>>>>> called
>>>>>>>> by
>>>>>>>>>>>> Grizzly when the client close the connection or when the
>>>> timeout
>>>>>>>>>>>> expires. The onTerminate will be called when you invoke
>>>>>>>>>>>> CometContext.notify(CometEvent.TERMINATE);
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Any ideas what's actually happening here and where the
>> data
>>>>>> is
>>>>>>>>>>>> going?
>>>>>>>>>>>>>> Difficult to say. Are they any exceptions in the log (I
>>>> guess
>>>>>>>>>> not).
>>>>>>>>>>>> Do
>>>>>>>>>>>>>> you know if the bytes are sent over the network? A tool
>> like
>>>>>>>>>>>> wireshark
>>>>>>>>>>>>>> or ngrep could help determining if the server or the
>> client
>>>>>> have
>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>> [JCT] I haven't used a packet sniffer to analyse the
>> network
>>>>>> yet
>>>>>>>>>> but
>>>>>>>>>>>> I do
>>>>>>>>>>>>> know that the server "believes" that the data has been
>>>>>>>> transmitted
>>>>>>>>>> as
>>>>>>>>>>>> the
>>>>>>>>>>>>> DataOutputStream#size() method is reporting an ever-
>>>> increasing
>>>>>>>>>> number
>>>>>>>>>>>> of
>>>>>>>>>>>>> transmitted bytes as the data is written and there are no
>>>>>>>>>> exceptions.
>>>>>>>>>>>>> [JCT] So perhaps the problem has something to do with the
>>>> fact
>>>>>>>> that
>>>>>>>>>> I
>>>>>>>>>>>> am not
>>>>>>>>>>>>> properly handling the interrupt or terminate events in the
>>>>>>>>>>>> CometHandler but
>>>>>>>>>>>>> I still don't know where the data is ending up. When I get
>> a
>>>>>>>>>> chance
>>>>>>>>>>>> I will
>>>>>>>>>>>>> do some sniffing around for packets and try to see exactly
>>>> what
>>>>>>>> is
>>>>>>>>>>>> going on.
>>>>>>>>>>>>> [JCT] But, as I said, this is the only outstanding issue I
>>>> have
>>>>>>>>>> with
>>>>>>>>>>>> my
>>>>>>>>>>>>> usage of Comet.
>>>>>>>>>>>> OK let's work on that :-) First thing to try is to output
>> some
>>>>>>>> trace
>>>>>>>>>>>> inside the onInterrupt to see if that method gets invoked.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> -- Jeanfrancois
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -JCT
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------------------------------------------------
>> --
>>>> --
>>>>>> --
>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>>>>> To unsubscribe, e-mail: users-
>>>> unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>>> For additional commands, e-mail: users-
>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>>>>> ------------------------------------------------------------
>> --
>>>> --
>>>>>> --
>>>>>>>> --
>>>>>>>>>> -
>>>>>>>>>>>> To unsubscribe, e-mail: users-
>> unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>>> For additional commands, e-mail: users-
>>>> help_at_grizzly.dev.java.net
>>>>>>>>>>> -------------------------------------------------------------
>> --
>>>> --
>>>>>> --
>>>>>>>> --
>>>>>>>>>>> To unsubscribe, e-mail: users-
>> unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>>> For additional commands, e-mail: users-
>>>> help_at_grizzly.dev.java.net
>>>>>>>>>> --------------------------------------------------------------
>> --
>>>> --
>>>>>> --
>>>>>>>> -
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>> For additional commands, e-mail: users-
>> help_at_grizzly.dev.java.net
>>>>>>>>> ---------------------------------------------------------------
>> --
>>>> --
>>>>>> --
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-
>> help_at_grizzly.dev.java.net
>>>>>>>> ----------------------------------------------------------------
>> --
>>>> --
>>>>>> -
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>> -----------------------------------------------------------------
>> --
>>>> --
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>> ------------------------------------------------------------------
>> --
>>>> -
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>> -------------------------------------------------------------------
>> --
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>> --------------------------------------------------------------------
>> -
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>