dev@glassfish.java.net

Re: CoyoteAdapter recycle problem [was SelectorThread and recycling problem]

From: Jan Luehe <Jan.Luehe_at_Sun.COM>
Date: Tue, 20 Jan 2009 17:57:56 -0800

If one or more filters apply to a request, the applicable filters are
chained, and the resulting ApplicationFilterChain is set on the
request.

ApplicationFilterChain instances are reused: If a request to which one
or more filters apply already has an ApplicationFilterChain set on it,
the applicable filters will be added to the existing
ApplicationFilterChain (instead of creating a new
ApplicationFilterChain instance), overriding any filters that may have
been added during the processing of a previous request. This is to
save on the number of ApplicationFilterChain instances created,
and this is also the reason why a request's ApplicationFilterChain
is not set to null during Request#recycle.

Once a request returns from a servlet and filter chain invocation, the
length and current index of its ApplicationFilterChain will be reset
to zero (using ApplicationFilterChain#release), but any references to
the filters themselves, which are instances of
ApplicationFilterConfig, will not be nulled out.

Therefore, it is possible that an ApplicationFilterConfig remains in a
request's ApplicationFilterChain, and holds on to its context and the
context's parent classloader, even after the context (webapp) has been
undeployed, as was seen here:


--> org.apache.catalina.core.ApplicationFilterConfig_at_0x10e2de48 (20 bytes)
(field context:)
--> com.sun.enterprise.web.WebModule_at_0xf3b73b8 (501 bytes) (field
parentClassLoader:)
--> com.sun.enterprise.loader.EJBClassLoader_at_0xefde040 (88 bytes)


One way to fix this would be to null out any ApplicationFilterConfig
references during ApplicationFilterChain#release.

Another, simpler (and more performant) fix would be to null out an
ApplicationFilterConfig's reference to its context (webapp) during
ApplicationFilterConfig#release, which is called on all of the webapp's
ApplicationFilterConfig instances during the webapp's undeployment .

I'll attach a proposed patch to the IssueTracker issue.

Thanks,

Jan


On 01/20/09 09:02, Jeanfrancois Arcand wrote:
> Salut,
>
> Roman Pokhodzhai wrote:
>> Seems to me that *filterChain *in *CoyoteRequest *also should be
>> cleaned during recycling.
>
> First, afterService is called from DefaultProcessorTask.postResponse
> :-) . Request/Response/CoyoteRequest/CoyoteResponse are always recycled.
>
> So you see that count is growing over and over when you deploy
> application, request, then undeploy, but not when you just do
> deploy/undeploy? I don't think this is related to NotesManager :-)
>
> PwcCoyoteRequest references WebModule, which has EJBClassloader as its
> parent classloader. Since we are caching those classes, this is
> "normal" to see such references. But the perm gem would happens under
> high load, right? The number of CoyoteRequest's 'leak' is related to
> the number of DefaultProcessorTask Grizzly's creates. It normally not
> creates that many instance.
>
> Can you attach the complete trace to the issue?
>
> Thanks
>
> -- Jeanfrancois
>
>
>>
>> On Tue, Jan 20, 2009 at 2:16 PM, Oleksiy Stashok
>> <Oleksiy.Stashok_at_sun.com <mailto:Oleksiy.Stashok_at_sun.com>> wrote:
>>
>>> posted at
>>> https://glassfish.dev.java.net/issues/show_bug.cgi?id=7071
>> Thank you.
>>
>>>
>>> Actually request.recycle() and response.recycle() are called
>>> in /*postProcess */method through /*inputBuffer.recycle()*/,
>>> so request is recycled but
>>> /*org.apache.coyote.Request.recylce() */doesn't recycle
>>> /*notesManager*/ (doesn't clear it). And I'm not sure that
>>> simple clearing this property in recycle method will help.
>>>
>>
>> Right. I just checked that.
>> When CoyoteAdapter works in compatWithTomcat mode - it calls
>> CoyoteAdapter.afterService(), which recycles
>> internal PwcCoyoteRequest/Response, which are stored as Nodes.
>> But when CoyoteAdapter works in default mode - it doesn't clean that
>> nodes, which leads to the issues you observe.
>>
>> I'll forward this issue to web-tier team to check it.
>>
>> Thank you.
>>
>> WBR,
>> Alexey.
>>
>>>
>>>
>>> btw, I will post it.
>>>
>>>
>>> On Tue, Jan 20, 2009 at 12:21 PM, Oleksiy Stashok
>>> <Oleksiy.Stashok_at_sun.com <mailto:Oleksiy.Stashok_at_sun.com>>
>>> wrote:
>>>
>>> Hi,
>>>
>>> it looks like bug.
>>> Seems, when DefaultProcessorTask is getting recycled - we
>>> don't recycle Coyote Request and Response:
>>>
>>> @Override
>>> public void recycle(){
>>> if ( taskEvent != null ){
>>> taskEvent.setStatus(TaskEvent.START);
>>> }
>>>
>>> if ( listeners!= null && listeners.size() > 0)
>>> clearTaskListeners();
>>>
>>> socket = null;
>>> dropConnection = false;
>>> key = null;
>>> + request.recycle();
>>> + response.recycle();
>>> }
>>>
>>> Can you pls. file an issue at
>>> https://glassfish.dev.java.net/issues?
>>>
>>> Thank you.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>
>>>
>>> On Jan 20, 2009, at 10:38 , Roman Pokhodzhai wrote:
>>>
>>> Hi,
>>>
>>> 1) "Hum..I'm not sure the application itself is
>>> referenced here. Can
>>> you elaborate a little?"
>>> Our application is referenced through
>>> com.sun.enterprise.loader.EJBClassLoader_at_0xefde040.
>>> This is exactly
>>> the class loader that has loaded our application.
>>>
>>> 2) "Yes, the notes are used to avoid object creation.
>>> As an example,
>>> we are putting CoyoteRequest/Response in the note we
>>> don't have to
>>> re-create those object"
>>>
>>> Cool, but notes are not recycled anyway. And
>>> CoyoteRequest (you can
>>> find it in
>>>
>>> trace)(com.sun.enterprise.web.connector.coyote.PwcCoyoteRequest
>>>
>>> <http://web.connector.coyote.PwcCoyoteRequest>@0x10cda420)
>>> keeps a reference chain to our EJBClassLoader even
>>> after the
>>> application itself is undeployed. And this class
>>> loader keeps all our
>>> application classes in PermGen memory
>>>
>>>
>>>
>>> On 1/19/09, Jeanfrancois Arcand
>>> <Jeanfrancois.Arcand_at_sun.com
>>> <mailto:Jeanfrancois.Arcand_at_sun.com>> wrote:
>>>
>>> Salut,
>>>
>>> Roman Pokhodzhai wrote:
>>>
>>> Hi All,
>>>
>>> If you use tasks recycling for
>>>
>>> com.sun.enterprise.web.connector.grizzly.SelectorThread
>>> <http://web.connector.grizzly.SelectorThread>
>>> in your
>>> application (it is used by default) then you
>>> get the following
>>> reference chain that keeps the application
>>> class loader after the
>>> application is undeployed:
>>>
>>>
>>> Hum..I'm not sure the application itself is
>>> referenced here. Can you
>>> elaborate a little?
>>>
>>>
>>> -->
>>>
>>> com.sun.enterprise.web.connector.grizzly.SelectorThread
>>>
>>> <http://web.connector.grizzly.SelectorThread>@0xe0a64e0
>>> (433 bytes) (field processorTasks:)
>>> -->
>>>
>>> java.util.concurrent.ConcurrentLinkedQueue_at_0xe0ba468
>>> (16 bytes)
>>> (field head:)
>>> -->
>>>
>>> java.util.concurrent.ConcurrentLinkedQueue$Node_at_0xe0caa80
>>> (16
>>> bytes) (field next:)
>>> -->
>>>
>>> java.util.concurrent.ConcurrentLinkedQueue$Node_at_0x116288d0
>>> (16
>>> bytes) (field item:)
>>> -->
>>>
>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask
>>>
>>> <http://web.connector.grizzly.DefaultProcessorTask>@0xe0f6180
>>> (169 bytes) (field request:)
>>> --> org.apache.coyote.Request_at_0xe10bef0 (153
>>> bytes) (field notesManager:)
>>> -->
>>> org.apache.coyote.NotesManagerImpl_at_0xe16b140
>>> (12 bytes) (field notes:)
>>> --> [Ljava.lang.Object;@0xe1c6e90 (136 bytes)
>>> (Element 1 of
>>> [Ljava.lang.Object;@0xe1c6e90:)
>>> -->
>>>
>>> com.sun.enterprise.web.connector.coyote.PwcCoyoteRequest
>>>
>>> <http://web.connector.coyote.PwcCoyoteRequest>@0x10cda420
>>> (190 bytes) (field filterChain:)
>>> -->
>>>
>>> org.apache.catalina.core.ApplicationFilterChain_at_0x10ce8d48
>>> (28
>>> bytes) (field filters:)
>>> -->
>>>
>>> [Lorg.apache.catalina.core.ApplicationFilterConfig;@0x10ce8d68
>>> (48
>>> bytes) (Element 0 of
>>>
>>> [Lorg.apache.catalina.core.ApplicationFilterConfig;@0x10ce8d68:)
>>> -->
>>>
>>> org.apache.catalina.core.ApplicationFilterConfig_at_0x10e2de48
>>> (20
>>> bytes) (field context:)
>>> --> com.sun.enterprise.web.WebModule_at_0xf3b73b8
>>> (501 bytes) (field
>>> parentClassLoader:)
>>> -->
>>>
>>> com.sun.enterprise.loader.EJBClassLoader_at_0xefde040
>>> (88 bytes)
>>>
>>> This trace shows that Request objects are not
>>> fully recycled. Their
>>> 'notes' are not recycled.
>>> You can also see it at
>>> org.apache.coyote.Request.recycle.
>>>
>>>
>>> It there some idea behind it?
>>>
>>>
>>> Yes, the notes are used to avoid object creation.
>>> As an example, we are
>>> putting CoyoteRequest/Response in the note we
>>> don't have to re-create
>>> those object.
>>>
>>>
>>> Or it's just a bug?
>>>
>>> No. Tomcat does this as well.
>>>
>>> A+
>>>
>>> -- Jeanfrancois
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> dev-unsubscribe_at_glassfish.dev.java.net
>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net>
>>> For additional commands, e-mail:
>>> dev-help_at_glassfish.dev.java.net
>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> dev-unsubscribe_at_glassfish.dev.java.net
>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net>
>>> For additional commands, e-mail:
>>> dev-help_at_glassfish.dev.java.net
>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> dev-unsubscribe_at_glassfish.dev.java.net
>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net>
>>> For additional commands, e-mail:
>>> dev-help_at_glassfish.dev.java.net
>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> dev-unsubscribe_at_glassfish.dev.java.net
>>> <mailto:dev-unsubscribe_at_glassfish.dev.java.net>
>>> For additional commands, e-mail:
>>> dev-help_at_glassfish.dev.java.net
>>> <mailto:dev-help_at_glassfish.dev.java.net>
>>>
>>>
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>