Hi Johan,
On 26.05.14 00:29, Johan Maasing wrote:
>
> It would be great with a few more samples showing not specific
> features but rather showing the expected usage / life cycle of the key
> concepts. Like the following (I probably have a few of these wrong -
> being a newbie to grizzly).
>
> When using Grizzly to write a CLIENT to access another system:
> * Transport is expected to be a singleton and be long lived.
> * Connection is expected to be created for each request to the other
> system. Connections can (in some situations) be pooled.
> * The Client should use one or more filters implementing onWrite to
> format the request to the other system.
> * The Client should use one or more filters implementing onRead to
> handle the response from the other system.
> * The onRead filter is called without any context or correlation with
> the request - i.e. it is not possible correlate/know which request
> caused the response this filter is handling.
>
> The last point is what caused me problems. I have "anonymous" requests
> i.e. there is no correlation ID or similar in the protocol. Just a lot
> of threads issuing requests and expecting (eventually) a response to
> that specific request, like in HTTP.
>
> I also tried to use the future, and alternatively use a completion
> listener, for the connection write method. Since the javadoc is a bit
> vague on what that future actually mean "|Future|
> <http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/Future.html?is-external=true>,
> using which it's possible to check the result" - The result of what?
> If the write succeded, if the request was transported, what the
> response was?
Good point, here by the result we mean the Buffer/data reaches the
Socket output buffer. We can't control the data beyond that point.
> This is how I understood the advice on the mailing list:
> * The best way to correlate the request and the response is to set
> attributes on the connection
correct.
> then call write on the connection, then get the connection from the
> FilterChain context to read the attributes when handling a response.
correct.
> This essentially makes the connections unpoolable (or more complicated
> to "clean" them before returning to the pool).
well, as an attribute you can maintain a queue, so with every new
request - you just add the request to the queue and when response is
coming - it should correspond to the first request in the queue.
> The last point is pretty hard to glean from the documentation. A
> multi-threaded HTTP Client might be a good sample to illustrate these
> points, given that they indeed are correct :)
You can take a look at Grizzly HttpClientFilter, it's using queue, as I
described above, to handle pipelined HTTP request.
> What would be the best way to go about contributing this? Mail to this
> list?
We can discuss it here. Or you can directly work on the documentation
git://java.net/grizzly~site
Thanks.
WBR,
Alexey.
>
>
> 2014-05-26 8:43 GMT+02:00 Oleksiy Stashok <oleksiy.stashok_at_oracle.com
> <mailto:oleksiy.stashok_at_oracle.com>>:
>
> Hi Tiran,
>
>
>> Before I dive into the big rework on the code (now that I
>> understand the complete picture), I have an important query that
>> might affect my new design.
>>
>> In case I need to perform a synchronous read operation (meaning
>> writing on a connection and retrieve the response from the server
>> synchronously), is it possible to do it using the read() method
>> of the connection, or is it doable only via interacting with one
>> of the transport’s filters (as I’ve seen in the *filterchain*
>> sample)?
>>
> I'd suggest to go via FilterChain. It's also should be possible to
> use connection.read(), but first you have to disable async read
> events processing (connection.disableIOEvent(IOEvent.READ)), and
> frankly saying we didn't use and test this path very actively.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>> I’ve tried the following code on the *echo* sample but the
>> message returned from the ReadResult getMessage() method is null:
>>
>> */Future<ReadResult<String,SocketAddress>> futureRead =
>> connection.read();/*
>>
>> *//*
>>
>> */Future<WriteResult<String,SocketAddress>> futureWrite =
>> connection.write("This is a test message");/*
>>
>> */futureWrite.get(10,TimeUnit.SECONDS);/*
>>
>> *//*
>>
>> */ReadResult readResult = futureRead.get(10,TimeUnit.SECONDS);/*
>>
>> */assert readResult != null;/*
>>
>> */System.out.println("Result obtained from server is: [" +
>> readResult.getMessage() + "]");/*
>>
>> *Tiran Meltser***
>> System Architect
>> Global Products & Operations
>> *Comverse* – /Making Your Network Smarter/
>>
>> T +972-3-7678381
>> M +972-54-5639381
>> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
>> *www.comverse.com <http://www.comverse.com/> *
>>
>> *P****Please think of the environment before printing this email***
>>
>> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
>> *Sent:* Monday, May 26, 2014 8:34 AM
>> *To:* Meltser Tiran
>> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>';
>> Broide Uri; Shafir Ehud
>> *Subject:* Re: Thread pool peculiar behavior
>>
>> Hi,
>>
>> On 25.05.14 08:45, Meltser Tiran wrote:
>>
>>
>> *Once realized that, I want to reduce the massive threads
>> creation so,*
>>
>> ·Can I use a *single Transport* (*used as a client not as a
>> server*) to create all my clients connections (*meaning is it
>> thread safe?*)?
>>
>> ·Or should I go with an Object Pool (on the *Transport*)?
>>
>> Single Transport and many Connections - it's how it's supposed to
>> work :) It's thread safe - you don't have to worry about
>> synchronization.
>>
>> Pls. let me know if you have more questions.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>
>> *Tiran Meltser***
>> System Architect
>> Global Products & Operations
>> *Comverse* – /Making Your Network Smarter/
>>
>> T +972-3-7678381
>> M +972-54-5639381
>> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
>> *www.comverse.com <http://www.comverse.com/> *
>>
>> *P****Please think of the environment before printing this email***
>>
>> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
>> *Sent:* Thursday, May 22, 2014 12:54 AM
>> *To:* Meltser Tiran
>> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>';
>> Broide Uri; Shafir Ehud
>> *Subject:* Re: Thread pool peculiar behavior
>>
>> Hi,
>>
>> On 21.05.14 02:43, Meltser Tiran wrote:
>>
>> Hi Alexey,
>>
>> Below you can find the output of the run (I ran 500 client
>> with 500ms delay between them).
>>
>> You can clearly see 5 threads are created at startup and no
>> new threads are created afterwards.
>>
>> *This aligns with my claim that the Grizzly ignores this pool
>> completely, as I tracked the run in the VisualVM (via the
>> threads tab) and these 5 threads are not active during all
>> the run (they are on Waiting status) while the Selectors and
>> original/built-in Workers threads (Grizzly-worker(x)) are
>> fully working and being replaced with other threads
>> throughout the run.*
>>
>> *Can you pls. share the testcase?
>>
>> Thank you.
>>
>> WBR,
>> Alexey.
>>
>>
>>
>> *
>>
>> **
>>
>> Also please find below the quote from the JConsole(VM Summary
>> tab) in this short run after it ended:
>>
>> *Live threads: *
>>
>>
>>
>> 38
>>
>> *Peak: *
>>
>>
>>
>> 56
>>
>> *Daemon threads: *
>>
>>
>>
>> 25
>>
>> *Total threads started: *
>>
>>
>>
>> *6,544 *
>>
>>
>>
>> *Current classes loaded: *
>>
>>
>>
>> 3,028
>>
>> *Total classes loaded: *
>>
>>
>>
>> 3,028
>>
>> *Total classes unloaded: *
>>
>>
>>
>> 0
>>
>>
>>
>>
>> _This is the output from the run:_
>>
>> *//*
>>
>> */java.lang.Exception: Creating a new thread with
>> 'experimental-thread-1'/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>>
>> */at
>> org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>>
>> */at
>> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>>
>> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>>
>> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>>
>> */java.lang.Exception: Creating a new thread with
>> 'experimental-thread-2'/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>>
>> */at
>> org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>>
>> */at
>> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>>
>> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>>
>> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>>
>> */java.lang.Exception: Creating a new thread with
>> 'experimental-thread-3'/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>>
>> */at
>> org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>>
>> */at
>> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>>
>> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>>
>> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>>
>> */java.lang.Exception: Creating a new thread with
>> 'experimental-thread-4'/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>>
>> */at
>> org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>>
>> */at
>> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>>
>> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>>
>> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>>
>> */java.lang.Exception: Creating a new thread with
>> 'experimental-thread-5'/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>>
>> */at
>> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>>
>> */at
>> org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>>
>> */at
>> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>>
>> */at
>> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>>
>> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>>
>> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>>
>> *Tiran Meltser***
>> System Architect
>> Global Products & Operations
>> *Comverse* – /Making Your Network Smarter/
>>
>> T +972-3-7678381
>> M +972-54-5639381
>> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
>> *www.comverse.com <http://www.comverse.com/> *
>>
>> *P****Please think of the environment before printing this email***
>>
>> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
>> *Sent:* Wednesday, May 21, 2014 10:37 AM
>> *To:* Meltser Tiran
>> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>';
>> Broide Uri; Shafir Ehud
>> *Subject:* Re: Thread pool peculiar behavior
>>
>> Hi Tiran,
>>
>> On 20.05.14 23:58, Meltser Tiran wrote:
>>
>> Second, let me focus on the problem as it seems I wasn’t
>> clear enough:
>>
>> /Assume for a moment I don’t want to touch the thread pool
>> (we’ll deal with that later on), and we agree on the size of
>> the Kernel and Workers default pools (as you wrote below),/
>>
>> *Why does Grizzly keep creating/renewing the threads of these
>> 2 pools constantly and rapidly (I can see in the JConsole the
>> “Total threads started” keeps growing although live threads
>> remains steady) and doesn’t operate them as pools (it can
>> also be seen very easily by using the VisualVM profiling
>> tool, even with light/moderate load runs)?*
>>
>> Hmm, that's very strange, can you pls. do one experiment?
>> Please override the ThreadFactory like:
>>
>>
>> builder.setWorkerThreadPoolConfig(
>> ThreadPoolConfig.defaultConfig().setThreadFactory(new
>> ThreadFactory() {
>>
>> final AtomicInteger counter = new
>> AtomicInteger();
>>
>> @Override
>> public Thread newThread(Runnable r) {
>> final String name =
>> "experimental-thread-" + counter.incrementAndGet();
>>
>> new Exception("Creating a new thread with
>> '" + name + "'")
>> .printStackTrace(System.out);
>>
>> return new DefaultWorkerThread(
>> AttributeBuilder.DEFAULT_ATTRIBUTE_BUILDER, name,
>> null, r);
>> }
>> })
>> );
>>
>> You'll see the stacktraces of all code paths, which lead to a new
>> thread creation. Would you be able to share that info?
>>
>> Thank you.
>>
>> WBR,
>> Alexey.
>>
>>
>>
>> *Tiran Meltser*
>> System Architect
>> Global Products & Operations
>> *Comverse* – /Making Your Network Smarter/
>>
>> T +972-3-7678381
>> M +972-54-5639381
>> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
>> *www.comverse.com <http://www.comverse.com/> *
>>
>> *P****Please think of the environment before printing this email***
>>
>> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
>> *Sent:* Wednesday, May 21, 2014 3:28 AM
>> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
>> *Subject:* Re: Thread pool peculiar behavior
>>
>> Oh, just realized you don't use HTTP framework... please
>> disregard my HttpHandler-related comments.
>> Please try to simulate long-lasting tasks in FilterChain (one of
>> your Filters can call Thread.sleep(N)) and you have to see worker
>> threads to be busy.
>>
>> If you're concerned about too many threads created - set the core
>> thread pool size like:
>> /workerPoolConfig.setCorePoolSize(0);
>>
>> WBR,
>> Alexey./
>>
>> On 20.05.14 16:50, Oleksiy Stashok wrote:
>>
>> Hi Tiran,
>>
>>
>>
>>
>> Here are the details:
>>
>> 1)I’m working on a 64 bit Windows workstation, 1 CPU with 4
>> cores and 8GB memory, running Gizzly 2.3.12 with Java 7 VM
>> (development machine).
>>
>> 2)I have observed the threads on load runs (heavy and light
>> loads – it doesn’t matter) and found out that *Grizzly keeps
>> replacing (creating new) threads* for both the
>> Selector/Kernel “*/pool/*” (*grizzly-nio-kernel(1)
>> SelectorRunner* to *grizzly-nio-kernel(5) SelectorRunner*)
>>
>> It's expected, number of kernel threads by default is
>> number-of-cores + 1;
>>
>>
>>
>>
>> and the Workers “*/pool/*” (*Grizzly-worker(1)* to
>> *Grizzly-worker(8)*)!
>>
>> also expected, the default number of worker threads is
>> number-of-cores * 2;
>>
>>
>>
>>
>> 3)When I try and define a Workers thread pool, I see the
>> threads group (*Grizzly(1)* to *Grizzly(100)* if the pool is
>> targeted for 100 threads) created and put on *Waiting* state
>> forever!
>>
>> Hmm, why do you think they put on wait forever? Pls. try to
>> implement HttpHandler, which does Thread.sleep(N) inside,
>> this way you'll be able to make a threads snapshot and you
>> have to see the worker thread stacktraces leading to
>> Thread.sleep(N);
>>
>> *The questions:*
>>
>> *1)**How can I cause Gizzly to operate its pools
>> (Selectors/Kernel and Workers) as pools and not as factory
>> (meaning don’t create new threads and if pool and or queue
>> are exhausted refuse new events/requests)?*
>>
>> It's possible to set ExecutorService as worker or kernel
>> thread-pool to a Transport instance directly (not via
>> builder), but I don't really understand what is the problem
>> with the Builder.
>>
>>
>>
>>
>> *2)**After question 1 is solved, how can I configure
>> Grizzly’s pools and getting them to work (and not put the
>> pool’s threads on a Waiting status)?*
>>
>> As I suggested, let's try to create a simple Thread.sleep(N)
>> HttpHandler and see the thread (thread group), which will
>> handle the request.
>>
>>
>>
>>
>>
>>
>> /NIOTransportBuilder transportBuilder =
>> TCPNIOTransportBuilder.newInstance();/
>>
>> //
>>
>> /// Configure workers pool/
>>
>> /ThreadPoolConfig workerPoolConfig =
>> transportBuilder.getWorkerThreadPoolConfig();/
>>
>> */// We shouldn't be here according to Grizzly examples, but
>> still it turns out we do.../*
>>
>> */It's expected in 2.3.12.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>
>>
>>
>>
>>
>> /*
>>
>> /if (workerPoolConfig == null)/
>>
>> /{/
>>
>> /workerPoolConfig = ThreadPoolConfig.defaultConfig();/
>>
>> /transportBuilder.setWorkerThreadPoolConfig(workerPoolConfig);/
>>
>> /}/
>>
>> //
>>
>> /workerPoolConfig.setCorePoolSize(100);/
>>
>> /workerPoolConfig.setMaxPoolSize(100);/
>>
>> /workerPoolConfig.setQueueLimit(1000);/
>>
>> /workerPoolConfig.setKeepAliveTime(-1,TimeUnit.HOURS);/
>>
>> //
>>
>> /NIOTransport transport = transportBuilder.build();/
>>
>> *Tiran****Meltser *
>> System Architect
>> Global Products & Operations
>> *Comverse* – /Making Your Network Smarter/
>>
>> T +972-3-7678381
>> M +972-54-5639381
>> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
>> *www.comverse.com <http://www.comverse.com/> *
>>
>> *P **Please think of the environment before printing this email *
>>
>> ------------------------------------------------------------------------
>>
>> “This e-mail message may contain confidential, commercial or
>> privileged information that constitutes proprietary
>> information of Comverse Inc. or its subsidiaries. If you are
>> not the intended recipient of this message, you are hereby
>> notified that any review, use or distribution of this
>> information is absolutely prohibited and we request that you
>> delete all copies and contact us by e-mailing to:
>> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>>
>> ------------------------------------------------------------------------
>>
>> “This e-mail message may contain confidential, commercial or
>> privileged information that constitutes proprietary information
>> of Comverse Inc. or its subsidiaries. If you are not the intended
>> recipient of this message, you are hereby notified that any
>> review, use or distribution of this information is absolutely
>> prohibited and we request that you delete all copies and contact
>> us by e-mailing to: security_at_comverse.com
>> <mailto:security_at_comverse.com>. Thank You.”
>>
>> ------------------------------------------------------------------------
>>
>> “This e-mail message may contain confidential, commercial or
>> privileged information that constitutes proprietary information
>> of Comverse Inc. or its subsidiaries. If you are not the intended
>> recipient of this message, you are hereby notified that any
>> review, use or distribution of this information is absolutely
>> prohibited and we request that you delete all copies and contact
>> us by e-mailing to: security_at_comverse.com
>> <mailto:security_at_comverse.com>. Thank You.”
>>
>> ------------------------------------------------------------------------
>>
>> “This e-mail message may contain confidential, commercial or
>> privileged information that constitutes proprietary information
>> of Comverse Inc. or its subsidiaries. If you are not the intended
>> recipient of this message, you are hereby notified that any
>> review, use or distribution of this information is absolutely
>> prohibited and we request that you delete all copies and contact
>> us by e-mailing to: security_at_comverse.com
>> <mailto:security_at_comverse.com>. Thank You.”
>>
>>
>> ------------------------------------------------------------------------
>> “This e-mail message may contain confidential, commercial or
>> privileged information that constitutes proprietary information
>> of Comverse Inc. or its subsidiaries. If you are not the intended
>> recipient of this message, you are hereby notified that any
>> review, use or distribution of this information is absolutely
>> prohibited and we request that you delete all copies and contact
>> us by e-mailing to: security_at_comverse.com
>> <mailto:security_at_comverse.com>. Thank You.”
>
>