Hi Tiran,
out of curiosity, can you pls. try to use this MultiStringFilter [1]
instead of StringFilter + MultiLineFilter.
Unlike StringFilter, the MultiStringFilter works with List<String>.
Sample client and server are attached.
WBR,
Alexey.
[1]
https://gist.github.com/oleksiys/acc8c4cd0a236f29c013
On 31.05.15 12:19, Meltser Tiran wrote:
>
> Hi Alexey,
>
> We are now facing a new problem we haven’t seen before [note we have
> moved to Grizzly 2.3.21] (we have a logger on the Grizzly classes),
> happing quite a few hours into the load run:
>
> java.lang.NullPointerException
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
>
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)
>
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
>
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
>
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
>
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Looking at the code (2.3.18) showing the following line:
>
> nextNextAction = executor.execute(currentFilter, ctx);
>
> Which suggests the executor is becoming nullified at some point of the
> run?…
>
> Note: We see quite a few such messages, most of them are just saying
> *java.lang.NullPointerException* without a stack trace…
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, May 29, 2015 9:34 AM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> I'd say StringFilter is probably not the best fit for you, may be it
> should be a Filter, which works with List of Strings. I'll try to
> implement something like that, just to check if it fits better.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 28.05.15 14:46, Meltser Tiran wrote:
>
> Hi Alexey,
>
> Didn’t mean to rush you … J
>
> I’m not sure our IMAP server would look kindly on adding “\r\n” to
> our commands.
>
> We were able to put sniffer and capture the problem and it seems
> the problem is with the read operation and not the write – it
> seems we don’t read (don’t know yet if this Grizzly code or ours)
> a response arriving from the IMAP server we are working against.
>
> We are still trying to put the finger on the accurate place for
> the problem…
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, May 28, 2015 5:59 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
>
> On 25.05.15 07:15, Meltser Tiran wrote:
>
> My question is: If I’ll pass to the this StringFilter (which
> is preceded by a TransportFilter) a String consisting of
> several lines separated by “\r\n” between them, will it
> function correctly (taking about write operation of course)?
>
> Yes, but during the serialization it will add "\r\n" to the end of
> the entire String. The "\r\n" symbols inside the String won't be
> changed.
>
> WBR,
> Alexey.
>
>
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this email *
>
> *From:* Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, May 22, 2015 4:18 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> while MiltiLineFilter is a good sample of how to work with
> Filters, it doesn't work well when you try to write to the same
> Connection from several threads at the same time.
> Do you have a particular reason why you can't serialize entire
> message and pass it to the next Filter, why you have to use
> ctx.getInvokeAction(param)?
>
> Thank you.
>
> WBR,
> Alexey.
>
> On 21.05.15 11:06, Meltser Tiran wrote:
>
> Hi Alexey,
>
> I’m back on this difficult problem, hoping to crack it this time J
>
> As far as I know (to continue our conversation):
>
> 1. The only place we have in the code
> “ctx.getInvokeAction(state)” is in the MultiLinePacket class
> which is very similar to the one existing in the Grizzly examples.
>
> 2. I’ve tried to get rid of the “/GRIZZLY0013: Exception
> during FilterChain execution/
>
> */java.lang.ClassCastException:
> com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket
> cannot be cast to org.glassfish.grizzly.Appendable”
> /*exception by implementing the Appendable interface, by
> adding the other object’s lines to the current object (the one
> on which the append has been called on) lines and returning it
> – *not sure if this is the expected behavior* (there isn’t
> much documentation on the expected behavior, and especially on
> when is it required to be implemented)
>
> 3. The *GRIZZLY0013* errors did disappear from the
> logger but our original errors remained L
>
> 4. I suspect (no proof yet) that the problem is related
> to this error and that implementing the Appendable Interface
> didn’t quiet do the job
>
> 5. It seems that on such cases, the Grizzly engine
> doesn’t send our requests to the IMAP server we are working
> against (we are a Proxy between IMAP clients and an IMAP server).
>
> You have mentioned some patch that might shed some more light
> on the issue, could I trouble you with such a patch (or any
> other hint that would give us any direction, note we are
> running over Grizzly 2.3.20)?
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this email *
>
> *From:* Meltser Tiran [mailto:Tiran.Meltser_at_comverse.com]
> *Sent:* Tuesday, February 17, 2015 2:33 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* RE: Exception during FilterChain execution
>
> Ok, I’ll try a few things to see if I can learn more on it,
> and update.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this email *
>
> *From:* Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, February 12, 2015 10:40 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi,
>
>
> I fear it won’t be feasible to create a working sample for
> you, as the problem happens once every 10,000 transactions and
> I’m quite sure it will not happen on a sample level complexity.
>
> Before troubling you with a patch, could you explain briefly
> on what scenarios would such an action happen?
>
> It may happen when you store a Connection related Filter state
> using either ctx.getStopAction(state) or
> ctx.getInvokeAction(state).
>
> In your case I'd say could be ctx.getInvokeAction(state); ,
> because first of all you say you don't do that, second is
> ctx.getStopAction(state) checks if the state is Appendable and
> if it's not it will throw an Exception right away.
> So it's probably ctx.getInvokeAction(state);, or something
> else, which we'll need to investigate using patch(es).
>
> WBR,
> Alexey.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this email *
>
> *From:* Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, February 11, 2015 11:22 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> is it possible for you to create a testcase so I can try to
> reproduce the problem locally?
> Or I can prepare you a patch and try to find out when exactly
> MultiLinePacket gets saved to the FilterChain store.
>
> WBR,
> Alexey.
>
> On 11.02.15 01:04, Meltser Tiran wrote:
>
> Hi Alexey,
>
> I call filterChainCtx.getStopAction() [*note without
> arguments*] in 2 places:
>
> 1. In the MultiLinePacket itself, following the same
> logic of the MultiLinePacket shown in the examples; note
> it is used for front end and backend filter chains
>
> 2. Right after closing the front end client’s
> connection (connection.close()); this also causes the
> backend server connection to be closed
>
> Please note that the stack trace shows the problem happens
> when writing a request to my backend server (I’m acting as
> a client here, in general I’m a proxy so I both a server
> and a client at the same time, using Grizzly both way).
>
> Hope that sheds some light on the issue.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this
> email *
>
> *From:* Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, February 11, 2015 10:39 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> do you ever call something like
> filterChainCtx.getStopAction(multiLinePacket)?
>
> WBR,
> Alexey.
>
> On 10.02.15 22:55, Meltser Tiran wrote:
>
> Hi,
>
> When running a load test on our applications, a very
> small percentage of the requests fails with the
> following exception:
>
> /WARNING: GRIZZLY0013: Exception during FilterChain
> execution/
>
> */java.lang.ClassCastException:
> com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket
> cannot be cast to org.glassfish.grizzly.Appendable/*
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.append(DefaultFilterChain.java:701)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.access$600(DefaultFilterChain.java:640)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.append(DefaultFilterChain.java:634)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.access$200(DefaultFilterChain.java:571)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.checkStoredMessage(DefaultFilterChain.java:510)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:197)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)/
>
> /at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)/
>
> /at
> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:418)/
>
> /at
> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:382)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.writeCommandRequestToEndPoint(AbstractProtocolCommandDispatcher.java:611)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:357)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:324)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandRequest(AbstractProtocolCommandDispatcher.java:209)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandResponse(AbstractProtocolCommandDispatcher.java:547)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandResponse(AbstractProtocolCommandDispatcher.java:369)/
>
> /at
> com.comverse.mist.app.rte.grizzly.filters.AbstractProtocolEndPointFilter.handleRead(AbstractProtocolEndPointFilter.java:184)/
>
> /at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)/
>
> /at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)/
>
> /at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)/
>
> /at
> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)/
>
> /at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)/
>
> /at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)/
>
> /at java.lang.Thread.run(Thread.java:745)/
>
> The object failing to be casted in our derivation of
> the MultiLinePacket (which doesn’t implement
> Appendable of course, similar to the original
> MultiLinePacket).
>
> *Can someone advise on why are we getting this error
> and what leads to this flow (after all, the vast
> majority of the flows pass without a problem)?*
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing
> this email *
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”