users@grizzly.java.net

Re: Exception during FilterChain execution

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Mon, 22 Jun 2015 15:07:13 +0200

Thank you Tiran,
I've fixed the sample using MultiStringFilter.

WBR,
Alexey.

On 09.06.15 18:57, Meltser Tiran wrote:
>
> Thanks Alexey – I’ll keep this option open incase needed, currently
> we’ll stay with the change I made and see where it will take us.
>
> *Kindly observe my note below on the examples…*
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, June 09, 2015 7:19 PM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> glad to hear you solved the problem.
> I just noticed in your email, that you hadn't found the sample file I
> had sent you.
> I hadn't commited it to the repo, it's here
> https://gist.github.com/oleksiys/acc8c4cd0a236f29c013
> <http://cp.mcafee.com/d/2DRPow938wrhoppj76zBAsOrKrjhpKCOyyCYrjhpKCOy-CYrjhpKOyYeupdETKqekmnXKcCVOzIelY-tfy2BPfNF_bCRPfNF_bCNPN17T7T-LOrVEVjKehWZOWrdQ-vp73DCkumKzp5dmUV7BgY-F6lK1FJ4Sqejt-jpoKMC--OCrKr9PCJhbc8OFfycFASO7bVk_OJ95GpJHfmoIxvkSn7kPhPqWb9JKDNEnGoxYgkhz8k53OT2eCtU6CQrTj79CMnWhEw6w-THjBm1Ewgrza14Qg2YzV_4i9Xa1EwYf-PsVlwrojdFZrR>
>
> WBR,
> Alexey.
>
> On 08.06.15 17:02, Meltser Tiran wrote:
>
> Hi Alexey,
>
> I’m glad to inform I’ve finally managed to solve the bug and we
> are now Cruising in 1600 TPS for 72 hours run with 100% success rate.
>
> I’ve changed the *MultiLineFilter#handleWrite()* method from the
> implementation *shown in the examples* to the following:
>
> */// Replace MultiLinePacket message with String message/*
>
> */MultiLinePacket input = (MultiLinePacket)ctx.getMessage();/*
>
> */String messageAsString = input.asStringMessage();/*
>
> */ctx.setMessage(messageAsString);/*
>
> *//*
>
> */return ctx.getInvokeAction();/*
>
> Where the *MultiLinePacket#asStringMessage()* method is (the
> method gets “true” as its *aOmitTerminatingEolMark* parameter and
> its outcome is turned into a String):
>
> */int projectedInitialCapacity = mLines.size() *
> PROJECTED_SINGLE_LINE_LENGTH_FOR_STRING_REPRESENTATION;/*
>
> */StringBuilder sb = new StringBuilder(projectedInitialCapacity);/*
>
> */int linesIterationIndex = 0;/*
>
> *//*
>
> */for (String line : mLines)/*
>
> */{/*
>
> */linesIterationIndex++;/*
>
> *//*
>
> */sb.append(line);/*
>
> *//*
>
> */// Check if need to add EOL and if so, if requested to do so.../*
>
> */if (line.endsWith(ProtocolHandler.EOL_MARK) == false && /*
>
> */ (aOmitTerminatingEolMark == false ||
> (aOmitTerminatingEolMark == true && linesIterationIndex <
> mLines.size())))/*
>
> */{/*
>
> */sb.append(ProtocolHandler.EOL_MARK);/*
>
> */}/*
>
> */}/*
>
> *//*
>
> */return sb;/*
>
> *I would consider changing the example code for the Multiline
> sample as it seems the approach shown there doesn’t scale up well…*
>
> In any case, thanks for your dedicated support on the issueJ
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Meltser Tiran [mailto:Tiran.Meltser_at_comverse.com]
> *Sent:* Wednesday, June 03, 2015 8:40 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* RE: Exception during FilterChain execution
> *Importance:* High
>
> Hi Alexey,
>
> I have downloaded the 2.3.21 sources, but I can’t find the
> MultiStringFilter class (not even in the samples sources) – where
> can I find it?
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, June 03, 2015 12:09 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> out of curiosity, can you pls. try to use this MultiStringFilter
> [1] instead of StringFilter + MultiLineFilter.
> Unlike StringFilter, the MultiStringFilter works with List<String>.
> Sample client and server are attached.
>
> WBR,
> Alexey.
>
> [1] https://gist.github.com/oleksiys/acc8c4cd0a236f29c013
> <http://cp.mcafee.com/d/2DRPow938wrhoppj76zBAsOrKrjhpKCOyyCYrjhpKCOy-CYrjhpKOyYeupdETKqekmnXKcCVOzIelY-tfy2BPfNF_bCRPfNF_bCNPN17T7T-LOrVEVjKehWZOWrdQ-vp73DCkumKzp5dmUV7BgY-F6lK1FJ4Sqejt-jpoKMC--OCrKr9PCJhbc8OFfycFASO7bVk_OJ95GpJHfmoIxvkSn7kPhPqWb9JKDNEnGoxYgkhz8k53OT2eCtU6CQrTj79CMnWhEw6w-THjBm1Ewgrza14Qg2YzV_4i9Xa1EwYf-PsVlwrojdFZrR>
>
> On 31.05.15 12:19, Meltser Tiran wrote:
>
> Hi Alexey,
>
> We are now facing a new problem we haven’t seen before [note
> we have moved to Grizzly 2.3.21] (we have a logger on the
> Grizzly classes), happing quite a few hours into the load run:
>
> java.lang.NullPointerException
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
>
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)
>
> at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
>
> at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
>
> at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
>
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
>
> at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
>
> at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Looking at the code (2.3.18) showing the following line:
>
> nextNextAction = executor.execute(currentFilter, ctx);
>
> Which suggests the executor is becoming nullified at some
> point of the run?…
>
> Note: We see quite a few such messages, most of them are just
> saying *java.lang.NullPointerException* without a stack trace…
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this
> email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, May 29, 2015 9:34 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> I'd say StringFilter is probably not the best fit for you, may
> be it should be a Filter, which works with List of Strings.
> I'll try to implement something like that, just to check if it
> fits better.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 28.05.15 14:46, Meltser Tiran wrote:
>
> Hi Alexey,
>
> Didn’t mean to rush you … J
>
> I’m not sure our IMAP server would look kindly on adding
> “\r\n” to our commands.
>
> We were able to put sniffer and capture the problem and it
> seems the problem is with the read operation and not the
> write – it seems we don’t read (don’t know yet if this
> Grizzly code or ours) a response arriving from the IMAP
> server we are working against.
>
> We are still trying to put the finger on the accurate
> place for the problem…
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this
> email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, May 28, 2015 5:59 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
>
> On 25.05.15 07:15, Meltser Tiran wrote:
>
> My question is: If I’ll pass to the this StringFilter
> (which is preceded by a TransportFilter) a String
> consisting of several lines separated by “\r\n”
> between them, will it function correctly (taking about
> write operation of course)?
>
> Yes, but during the serialization it will add "\r\n" to
> the end of the entire String. The "\r\n" symbols inside
> the String won't be changed.
>
> WBR,
> Alexey.
>
>
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing this
> email *
>
> *From:* Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, May 22, 2015 4:18 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> while MiltiLineFilter is a good sample of how to work with
> Filters, it doesn't work well when you try to write to the
> same Connection from several threads at the same time.
> Do you have a particular reason why you can't serialize
> entire message and pass it to the next Filter, why you
> have to use ctx.getInvokeAction(param)?
>
> Thank you.
>
> WBR,
> Alexey.
>
> On 21.05.15 11:06, Meltser Tiran wrote:
>
> Hi Alexey,
>
> I’m back on this difficult problem, hoping to crack it
> this time J
>
> As far as I know (to continue our conversation):
>
> 1. The only place we have in the code
> “ctx.getInvokeAction(state)” is in the MultiLinePacket
> class which is very similar to the one existing in the
> Grizzly examples.
>
> 2. I’ve tried to get rid of the “/GRIZZLY0013:
> Exception during FilterChain execution/
>
> */java.lang.ClassCastException:
> com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket
> cannot be cast to org.glassfish.grizzly.Appendable”
> /*exception by implementing the Appendable interface,
> by adding the other object’s lines to the current
> object (the one on which the append has been called
> on) lines and returning it – *not sure if this is the
> expected behavior* (there isn’t much documentation on
> the expected behavior, and especially on when is it
> required to be implemented)
>
> 3. The *GRIZZLY0013* errors did disappear from
> the logger but our original errors remained L
>
> 4. I suspect (no proof yet) that the problem is
> related to this error and that implementing the
> Appendable Interface didn’t quiet do the job
>
> 5. It seems that on such cases, the Grizzly
> engine doesn’t send our requests to the IMAP server we
> are working against (we are a Proxy between IMAP
> clients and an IMAP server).
>
> You have mentioned some patch that might shed some
> more light on the issue, could I trouble you with such
> a patch (or any other hint that would give us any
> direction, note we are running over Grizzly 2.3.20)?
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing
> this email *
>
> *From:* Meltser Tiran [mailto:Tiran.Meltser_at_comverse.com]
> *Sent:* Tuesday, February 17, 2015 2:33 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* RE: Exception during FilterChain execution
>
> Ok, I’ll try a few things to see if I can learn more
> on it, and update.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing
> this email *
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, February 12, 2015 10:40 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi,
>
>
> I fear it won’t be feasible to create a working sample
> for you, as the problem happens once every 10,000
> transactions and I’m quite sure it will not happen on
> a sample level complexity.
>
> Before troubling you with a patch, could you explain
> briefly on what scenarios would such an action happen?
>
> It may happen when you store a Connection related
> Filter state using either ctx.getStopAction(state) or
> ctx.getInvokeAction(state).
>
> In your case I'd say could be
> ctx.getInvokeAction(state); , because first of all you
> say you don't do that, second is
> ctx.getStopAction(state) checks if the state is
> Appendable and if it's not it will throw an Exception
> right away.
> So it's probably ctx.getInvokeAction(state);, or
> something else, which we'll need to investigate using
> patch(es).
>
> WBR,
> Alexey.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing
> this email *
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, February 11, 2015 11:22 AM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> is it possible for you to create a testcase so I can
> try to reproduce the problem locally?
> Or I can prepare you a patch and try to find out when
> exactly MultiLinePacket gets saved to the FilterChain
> store.
>
> WBR,
> Alexey.
>
> On 11.02.15 01:04, Meltser Tiran wrote:
>
> Hi Alexey,
>
> I call filterChainCtx.getStopAction() [*note
> without arguments*] in 2 places:
>
> 1. In the MultiLinePacket itself, following
> the same logic of the MultiLinePacket shown in the
> examples; note it is used for front end and
> backend filter chains
>
> 2. Right after closing the front end
> client’s connection (connection.close()); this
> also causes the backend server connection to be closed
>
> Please note that the stack trace shows the problem
> happens when writing a request to my backend
> server (I’m acting as a client here, in general
> I’m a proxy so I both a server and a client at the
> same time, using Grizzly both way).
>
> Hope that sheds some light on the issue.
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before printing
> this email *
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, February 11, 2015 10:39 AM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Exception during FilterChain execution
>
> Hi Tiran,
>
> do you ever call something like
> filterChainCtx.getStopAction(multiLinePacket)?
>
> WBR,
> Alexey.
>
> On 10.02.15 22:55, Meltser Tiran wrote:
>
> Hi,
>
> When running a load test on our applications,
> a very small percentage of the requests fails
> with the following exception:
>
> /WARNING: GRIZZLY0013: Exception during
> FilterChain execution/
>
> */java.lang.ClassCastException:
> com.comverse.mist.app.rte.grizzly.filters.io.payload.MultiLinePacket
> cannot be cast to
> org.glassfish.grizzly.Appendable/*
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.append(DefaultFilterChain.java:701)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FilterStateElement.access$600(DefaultFilterChain.java:640)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.append(DefaultFilterChain.java:634)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState.access$200(DefaultFilterChain.java:571)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.checkStoredMessage(DefaultFilterChain.java:510)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:197)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)/
>
> /at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.write(DefaultFilterChain.java:413)/
>
> /at
> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:418)/
>
> /at
> org.glassfish.grizzly.nio.NIOConnection.write(NIOConnection.java:382)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.writeCommandRequestToEndPoint(AbstractProtocolCommandDispatcher.java:611)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:357)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandRequest(AbstractProtocolCommandDispatcher.java:324)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandRequest(AbstractProtocolCommandDispatcher.java:209)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.doDispatchCommandResponse(AbstractProtocolCommandDispatcher.java:547)/
>
> /at
> com.comverse.mist.app.rte.protocol.command.AbstractProtocolCommandDispatcher.dispatchCommandResponse(AbstractProtocolCommandDispatcher.java:369)/
>
> /at
> com.comverse.mist.app.rte.grizzly.filters.AbstractProtocolEndPointFilter.handleRead(AbstractProtocolEndPointFilter.java:184)/
>
> /at
> org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)/
>
> /at
> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)/
>
> /at
> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)/
>
> /at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)/
>
> /at
> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)/
>
> /at
> org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)/
>
> /at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)/
>
> /at
> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)/
>
> /at java.lang.Thread.run(Thread.java:745)/
>
> The object failing to be casted in our
> derivation of the MultiLinePacket (which
> doesn’t implement Appendable of course,
> similar to the original MultiLinePacket).
>
> *Can someone advise on why are we getting this
> error and what leads to this flow (after all,
> the vast majority of the flows pass without a
> problem)?*
>
> *Tiran Meltser *
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P Please think of the environment before
> printing this email *
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message,
> you are hereby notified that any review, use
> or distribution of this information is
> absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing
> to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of Comverse
> Inc. or its subsidiaries. If you are not the
> intended recipient of this message, you are hereby
> notified that any review, use or distribution of
> this information is absolutely prohibited and we
> request that you delete all copies and contact us
> by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”