Hi Alexey,
We are now facing a new problem we haven't seen before [note we have moved to Grizzly 2.3.21] (we have a logger on the Grizzly classes), happing quite a few hours into the load run:
java.lang.NullPointerException
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
at java.lang.Thread.run(Thread.java:745)
Looking at the code (2.3.18) showing the following line:
nextNextAction = executor.execute(currentFilter, ctx);
Which suggests the executor is becoming nullified at some point of the run?...
Note: We see quite a few such messages, most of them are just saying java.lang.NullPointerException without a stack trace...
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Friday, May 29, 2015 9:34 AM
To: users_at_grizzly.java.net
Subject: Re: Exception during FilterChain execution
Hi Tiran,
I'd say StringFilter is probably not the best fit for you, may be it should be a Filter, which works with List of Strings. I'll try to implement something like that, just to check if it fits better.
Thanks.
WBR,
Alexey.
On 28.05.15 14:46, Meltser Tiran wrote:
Hi Alexey,
Didn't mean to rush you ... :)
I'm not sure our IMAP server would look kindly on adding "\r\n" to our commands.
We were able to put sniffer and capture the problem and it seems the problem is with the read operation and not the write - it seems we don't read (don't know yet if this Grizzly code or ours) a response arriving from the IMAP server we are working against.
We are still trying to put the finger on the accurate place for the problem...
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, May 28, 2015 5:59 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Exception during FilterChain execution
Hi Tiran,
On 25.05.15 07:15, Meltser Tiran wrote:
My question is: If I'll pass to the this StringFilter (which is preceded by a TransportFilter) a String consisting of several lines separated by "\r\n" between them, will it function correctly (taking about write operation of course)?
Yes, but during the serialization it will add "\r\n" to the end of the entire String. The "\r\n" symbols inside the String won't be changed.
WBR,
Alexey.
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Friday, May 22, 2015 4:18 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Exception during FilterChain execution
Hi Tiran,
while MiltiLineFilter is a good sample of how to work with Filters, it doesn't work well when you try to write to the same Connection from several threads at the same time.
Do you have a particular reason why you can't serialize entire message and pass it to the next Filter, why you have to use ctx.getInvokeAction(param)?
Thank you.
WBR,
Alexey.
On 21.05.15 11:06, Meltser Tiran wrote:
Hi Alexey,
I'm back on this difficult problem, hoping to crack it this time J
As far as I know (to continue our conversation):
1. The only place we have in the code "ctx.getInvokeAction(state)" is in the MultiLinePacket class which is very similar to the one existing in the Grizzly examples.
2. I've tried to get rid of the "GRIZZLY0013: Exception during FilterChain execution
java.lang.ClassCastException: com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket cannot be cast to org.glassfish.grizzly.Appendable" exception by implementing the Appendable interface, by adding the other object's lines to the current object (the one on which the append has been called on) lines and returning it - not sure if this is the expected behavior (there isn't much documentation on the expected behavior, and especially on when is it required to be implemented)
3. The GRIZZLY0013 errors did disappear from the logger but our original errors remained L
4. I suspect (no proof yet) that the problem is related to this error and that implementing the Appendable Interface didn't quiet do the job
5. It seems that on such cases, the Grizzly engine doesn't send our requests to the IMAP server we are working against (we are a Proxy between IMAP clients and an IMAP server).
You have mentioned some patch that might shed some more light on the issue, could I trouble you with such a patch (or any other hint that would give us any direction, note we are running over Grizzly 2.3.20)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Meltser Tiran [mailto:Tiran.Meltser_at_comverse.com]
Sent: Tuesday, February 17, 2015 2:33 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: RE: Exception during FilterChain execution
Ok, I'll try a few things to see if I can learn more on it, and update.
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, February 12, 2015 10:40 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Exception during FilterChain execution
Hi,
I fear it won't be feasible to create a working sample for you, as the problem happens once every 10,000 transactions and I'm quite sure it will not happen on a sample level complexity.
Before troubling you with a patch, could you explain briefly on what scenarios would such an action happen?
It may happen when you store a Connection related Filter state using either ctx.getStopAction(state) or ctx.getInvokeAction(state).
In your case I'd say could be ctx.getInvokeAction(state); , because first of all you say you don't do that, second is ctx.getStopAction(state) checks if the state is Appendable and if it's not it will throw an Exception right away.
So it's probably ctx.getInvokeAction(state);, or something else, which we'll need to investigate using patch(es).
WBR,
Alexey.
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Wednesday, February 11, 2015 11:22 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Exception during FilterChain execution
Hi Tiran,
is it possible for you to create a testcase so I can try to reproduce the problem locally?
Or I can prepare you a patch and try to find out when exactly MultiLinePacket gets saved to the FilterChain store.
WBR,
Alexey.
On 11.02.15 01:04, Meltser Tiran wrote:
Hi Alexey,
I call filterChainCtx.getStopAction() [note without arguments] in 2 places:
1. In the MultiLinePacket itself, following the same logic of the MultiLinePacket shown in the examples; note it is used for front end and backend filter chains
2. Right after closing the front end client's connection (connection.close()); this also causes the backend server connection to be closed
Please note that the stack trace shows the problem happens when writing a request to my backend server (I'm acting as a client here, in general I'm a proxy so I both a server and a client at the same time, using Grizzly both way).
Hope that sheds some light on the issue.
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Wednesday, February 11, 2015 10:39 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Exception during FilterChain execution
Hi Tiran,
do you ever call something like
filterChainCtx.getStopAction(multiLinePacket)?
WBR,
Alexey.
On 10.02.15 22:55, Meltser Tiran wrote:
Hi,
When running a load test on our applications, a very small percentage of the requests fails with the following exception:
WARNING: GRIZZLY0013: Exception during FilterChain execution
java.lang.ClassCastException: com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket cannot be cast to org.glassfish.grizzly.Appendable
at org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.append(DefaultFilterChain.java:701)
at org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.access$600(DefaultFilterChain.java:640)
at org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.append(DefaultFilterChain.java:634)
at org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.access$200(DefaultFilterChain.java:571)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.checkStoredMessage(DefaultFilterChain.java:510)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:197)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)
at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:418)
at org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:382)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.writeCommandRequestToEndPoint(AbstractProtocolCommandDispatcher.java:611)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:357)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:324)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandRequest(AbstractProtocolCommandDispatcher.java:209)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandResponse(AbstractProtocolCommandDispatcher.java:547)
at com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandResponse(AbstractProtocolCommandDispatcher.java:369)
at com.comverse.mist.app.rte.grizzly.filters.AbstractProtocolEndPointFilter.handleRead(AbstractProtocolEndPointFilter.java:184)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
at java.lang.Thread.run(Thread.java:745)
The object failing to be casted in our derivation of the MultiLinePacket (which doesn't implement Appendable of course, similar to the original MultiLinePacket).
Can someone advise on why are we getting this error and what leads to this flow (after all, the vast majority of the flows pass without a problem)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com. Thank You."