dev@grizzly.java.net

Strange filter chaining when an EOFException occurred

From: Bongjae Chang <bongjae.chang_at_gmail.com>
Date: Tue, 05 Aug 2014 18:29:41 +0900

Hi,

When I met an EOFException at the TCPNIOTransportFilter, I saw some strange
filter chainings.

I have two TCPNIOTransports.

1. Custom client: it has only two filters(TransportFilter -->
MyCustomFilter)
2. HttpServer using AJP protocol: it maybe has two more
filters(TransportFilter --> httpServerCodecFilter --> IdleTimeoutFilter -->
...)

First, HttpServer received AJP packets then the specific HttpHandler called
the custom API with the custom client.
If the http server met EOFException, the following exception occurred. I
replaced real IP and domain into ${http_server_IP},${custom_server_domain}
and ${custom_server_IP}.

====
8¿ù 01, 2014 9:38:30 ¿ÀÀü org.glassfish.grizzly.http.server.HttpHandler
doHandle
°æ°í: GRIZZLY0200: Service exception
java.io.EOFException
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.jav
a:632)
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOT
ransportFilter.java:75)
        at
org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter
.java:173)
        at
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolve
r.java:119)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFi
lterChain.java:284)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(Defaul
tFilterChain.java:201)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:133)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransp
ort.java:561)
        at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOSt
rategy.java:112)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOS
trategy.java:117)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThr
eadIOStrategy.java:56)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable
.run(WorkerThreadIOStrategy.java:137)
        at
com.kakao.common.util.StatisticKakaoThreadPoolExecutor$StatisticRunnable.run
(StatisticKakaoThreadPoolExecutor.java:278)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.runTask(KakaoThreadPool
Executor.java:950)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.run(KakaoThreadPoolExec
utor.java:987)
        at java.lang.Thread.run(Thread.java:744)

8¿ù 01, 2014 9:38:30 ¿ÀÀü org.glassfish.grizzly.http.server.Request
onAfterService
°æ°í: GRIZZLY0204: Unexpected error during afterService notification
java.lang.NullPointerException
        at
org.glassfish.grizzly.filterchain.FilterChainContext.notifyDownstream(Filter
ChainContext.java:895)
        at
org.glassfish.grizzly.filterchain.FilterChainContext.notifyDownstream(Filter
ChainContext.java:890)
        at
org.glassfish.grizzly.http.server.HttpServerFilter$FlushResponseHandler.onAf
terService(HttpServerFilter.java:470)
        at
org.glassfish.grizzly.http.server.Request.onAfterService(Request.java:582)
        at
org.glassfish.grizzly.http.server.HttpServerFilter.afterService(HttpServerFi
lter.java:385)
        at
org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilt
er.java:260)
        at
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolve
r.java:119)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFi
lterChain.java:284)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(Defaul
tFilterChain.java:201)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:133)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransp
ort.java:561)
        at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOSt
rategy.java:112)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOS
trategy.java:117)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThr
eadIOStrategy.java:56)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable
.run(WorkerThreadIOStrategy.java:137)
        at
com.kakao.common.util.StatisticKakaoThreadPoolExecutor$StatisticRunnable.run
(StatisticKakaoThreadPoolExecutor.java:278)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.runTask(KakaoThreadPool
Executor.java:950)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.run(KakaoThreadPoolExec
utor.java:987)
        at java.lang.Thread.run(Thread.java:744)
        
8¿ù 01, 2014 9:38:30 ¿ÀÀü org.glassfish.grizzly.filterchain.DefaultFilterChain
execute
°æ°í: GRIZZLY0013: Exception during FilterChain execution
java.lang.IndexOutOfBoundsException: Index: 5, Size: 2
        at java.util.ArrayList.rangeCheck(ArrayList.java:635)
        at java.util.ArrayList.get(ArrayList.java:411)
        at
org.glassfish.grizzly.filterchain.ListFacadeFilterChain.get(ListFacadeFilter
Chain.java:145)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(Defaul
tFilterChain.java:193)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:133)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChai
n.java:413)
        at
org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:407)
        at
org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:371)
        at
com.kakao.common.client.AbstractClient.sendInternal(AbstractClient.java:582)
        at
com.kakao.capri.alfs.client.CapriAlfsClient.increaseAndCheck(CapriAlfsClient
.java:46)
        at
com.kakao.capri.alfs.client.spring.GrizzlyAlfsClientBean.increaseAndCheck(Gr
izzlyAlfsClientBean.java:300)
        at
com.kakao.capri.api.server.alfs.AlfsClientDelegator.increaseAndCheck(AlfsCli
entDelegator.java:178)
        at
com.kakao.capri.api.server.alfs.AlfsClientDelegator.increaseAndCheck(AlfsCli
entDelegator.java:156)
        at
com.kakao.capri.api.server.handler.APIHandler.checkAlfsFilter(APIHandler.jav
a:130)
        at
com.kakao.capri.api.server.handler.APIHandler.beforeService(APIHandler.java:
81)
        at
com.kakao.server.web.handler.GrizzlyHandler.service(GrizzlyHandler.java:127)
        at
org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:20
1)
        at
org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
        at
org.glassfish.grizzly.http.server.HttpHandlerChain.doHandle(HttpHandlerChain
.java:223)
        at
org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilt
er.java:235)
        at
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolve
r.java:119)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFi
lterChain.java:284)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(Defaul
tFilterChain.java:201)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:133)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransp
ort.java:561)
        at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOSt
rategy.java:112)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOS
trategy.java:117)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThr
eadIOStrategy.java:56)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable
.run(WorkerThreadIOStrategy.java:137)
        at
com.kakao.common.util.StatisticKakaoThreadPoolExecutor$StatisticRunnable.run
(StatisticKakaoThreadPoolExecutor.java:278)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.runTask(KakaoThreadPool
Executor.java:950)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.run(KakaoThreadPoolExec
utor.java:987)
        at java.lang.Thread.run(Thread.java:744)
        
8¿ù 01, 2014 9:38:30 ¿ÀÀü org.glassfish.grizzly.ProcessorExecutor execute
°æ°í: GRIZZLY0029: Error during Processor execution.
Connection=TCPNIOConnection{localSocketAddress={/${http_server_IP}:33824},
peerSocketAddress={${custom_server_domain}/${custom_server_IP}:10000}}
ioEvent=READ
processor=org.glassfish.grizzly.filterchain.DefaultFilterChain_at_55ef2704
java.lang.IndexOutOfBoundsException: Index: 6, Size: 2
        at java.util.ArrayList.rangeCheck(ArrayList.java:635)
        at java.util.ArrayList.get(ArrayList.java:411)
        at
org.glassfish.grizzly.filterchain.ListFacadeFilterChain.get(ListFacadeFilter
Chain.java:145)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.throwChain(DefaultFilte
rChain.java:478)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:159)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChai
n.java:413)
        at
org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:407)
        at
org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:371)
        at
com.kakao.common.client.AbstractClient.sendInternal(AbstractClient.java:582)
        at
com.kakao.capri.alfs.client.CapriAlfsClient.increaseAndCheck(CapriAlfsClient
.java:46)
        at
com.kakao.capri.alfs.client.spring.GrizzlyAlfsClientBean.increaseAndCheck(Gr
izzlyAlfsClientBean.java:300)
        at
com.kakao.capri.api.server.alfs.AlfsClientDelegator.increaseAndCheck(AlfsCli
entDelegator.java:178)
        at
com.kakao.capri.api.server.alfs.AlfsClientDelegator.increaseAndCheck(AlfsCli
entDelegator.java:156)
        at
com.kakao.capri.api.server.handler.APIHandler.checkAlfsFilter(APIHandler.jav
a:130)
        at
com.kakao.capri.api.server.handler.APIHandler.beforeService(APIHandler.java:
81)
        at
com.kakao.server.web.handler.GrizzlyHandler.service(GrizzlyHandler.java:127)
        at
org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:20
1)
        at
org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175)
        at
org.glassfish.grizzly.http.server.HttpHandlerChain.doHandle(HttpHandlerChain
.java:223)
        at
org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilt
er.java:235)
        at
org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolve
r.java:119)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFi
lterChain.java:284)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(Defaul
tFilterChain.java:201)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterCh
ain.java:133)
        at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterCh
ain.java:112)
        at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransp
ort.java:561)
        at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOSt
rategy.java:112)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOS
trategy.java:117)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThr
eadIOStrategy.java:56)
        at
org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable
.run(WorkerThreadIOStrategy.java:137)
        at
com.kakao.common.util.StatisticKakaoThreadPoolExecutor$StatisticRunnable.run
(StatisticKakaoThreadPoolExecutor.java:278)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.runTask(KakaoThreadPool
Executor.java:950)
        at
com.kakao.common.util.KakaoThreadPoolExecutor$Worker.run(KakaoThreadPoolExec
utor.java:987)
        at java.lang.Thread.run(Thread.java:744)
===

I don't know whether the thread of stack traces is same or not. But I know
the custom client has only two filter chains but it seems that it tries to
refer to http filters.
I think maybe current FilterChainContext or internalContext was wrong. In
the last exception, peerSocketAddress was correct but DefaultFilterChain
tried to get index 6 and you can see a NPE during getting getFilterChain()
in FilterChainContext.

Unfortunately, it is not easy to reproduce it. And I used grizzly version
v2.3.13.

Could I get some hints about this issue?

Thanks!

Regards,
Bongjae Chang