users@grizzly.java.net

Re: Thread pool peculiar behavior

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Mon, 26 May 2014 01:33:22 -0700

I've just fixed it on trunk and 2.3.x branch.
Anyway the code has to look like:

*/connection.disableIOEvent(IOEvent.READ);/*

*//*

*/Future<WriteResult<String,SocketAddress>> futureWrite =
connection.write("This is a test message");/*

*/futureWrite.get(10,TimeUnit.SECONDS);/*

*//*

*/Future<ReadResult<String,SocketAddress>> futureRead =
connection.read();/**//*

*/ReadResult readResult = futureRead.get(10,TimeUnit.SECONDS);/*

*/assert readResult != null;/*

*//*

*/System.out.println("Read Result obtained from server is: [" +
readResult.getMessage() + "]");/*


because even though connection.read() returns Future - it's always
blocking operation.
I'd anyway suggest to use FilterChain ;)

Thanks.

WBR,
Alexey.

On 26.05.14 00:47, Meltser Tiran wrote:
>
> Understood.
>
> I gave your suggestion a try as it could simplify my code a lot (it is
> needed for my automatic tests), but indeed this doesn't seem to work
> (still getting a null):
>
> *//*
>
> */connection.disableIOEvent(IOEvent.READ);/*
>
> *//*
>
> */Future<ReadResult<String,SocketAddress>> futureRead =
> connection.read();/*
>
> *//*
>
> */Future<WriteResult<String,SocketAddress>> futureWrite =
> connection.write("This is a test message");/*
>
> */futureWrite.get(10,TimeUnit.SECONDS);/*
>
> *//*
>
> */ReadResult readResult = futureRead.get(10,TimeUnit.SECONDS);/*
>
> */assert readResult != null;/*
>
> *//*
>
> */System.out.println("Read Result obtained from server is: [" +
> readResult.getMessage() + "]");/*
>
> I'll take the harder path L
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, May 26, 2014 9:43 AM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: Thread pool peculiar behavior
>
> Hi Tiran,
>
> Before I dive into the big rework on the code (now that I understand
> the complete picture), I have an important query that might affect my
> new design.
>
> In case I need to perform a synchronous read operation (meaning
> writing on a connection and retrieve the response from the server
> synchronously), is it possible to do it using the read() method of the
> connection, or is it doable only via interacting with one of the
> transport's filters (as I've seen in the *filterchain* sample)?
>
> I'd suggest to go via FilterChain. It's also should be possible to use
> connection.read(), but first you have to disable async read events
> processing (connection.disableIOEvent(IOEvent.READ)), and frankly
> saying we didn't use and test this path very actively.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
> I've tried the following code on the *echo* sample but the message
> returned from the ReadResult getMessage() method is null:
>
> */Future<ReadResult<String,SocketAddress>> futureRead =
> connection.read();/*
>
> *//*
>
> */Future<WriteResult<String,SocketAddress>> futureWrite =
> connection.write("This is a test message");/*
>
> */futureWrite.get(10,TimeUnit.SECONDS);/*
>
> *//*
>
> */ReadResult readResult = futureRead.get(10,TimeUnit.SECONDS);/*
>
> */assert readResult != null;/*
>
> */System.out.println("Result obtained from server is: [" +
> readResult.getMessage() + "]");/*
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, May 26, 2014 8:34 AM
> *To:* Meltser Tiran
> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>'; Broide
> Uri; Shafir Ehud
> *Subject:* Re: Thread pool peculiar behavior
>
> Hi,
>
> On 25.05.14 08:45, Meltser Tiran wrote:
>
>
> *Once realized that, I want to reduce the massive threads creation
> so,*
>
> ·Can I use a *single Transport* (*used as a client not as a
> server*) to create all my clients connections (*meaning is it
> thread safe?*)?
>
> ·Or should I go with an Object Pool (on the *Transport*)?
>
> Single Transport and many Connections - it's how it's supposed to work
> :) It's thread safe - you don't have to worry about synchronization.
>
> Pls. let me know if you have more questions.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, May 22, 2014 12:54 AM
> *To:* Meltser Tiran
> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>'; Broide
> Uri; Shafir Ehud
> *Subject:* Re: Thread pool peculiar behavior
>
> Hi,
>
> On 21.05.14 02:43, Meltser Tiran wrote:
>
> Hi Alexey,
>
> Below you can find the output of the run (I ran 500 client with
> 500ms delay between them).
>
> You can clearly see 5 threads are created at startup and no new
> threads are created afterwards.
>
> *This aligns with my claim that the Grizzly ignores this pool
> completely, as I tracked the run in the VisualVM (via the threads
> tab) and these 5 threads are not active during all the run (they
> are on Waiting status) while the Selectors and original/built-in
> Workers threads (Grizzly-worker(x)) are fully working and being
> replaced with other threads throughout the run.*
>
> *Can you pls. share the testcase?
>
> Thank you.
>
> WBR,
> Alexey.
>
>
>
>
> *
>
> **
>
> Also please find below the quote from the JConsole(VM Summary tab) in
> this short run after it ended:
>
> *Live threads: *
>
>
>
> 38
>
> *Peak: *
>
>
>
> 56
>
> *Daemon threads: *
>
>
>
> 25
>
> *Total threads started: *
>
>
>
> *6,544 *
>
>
>
> *Current classes loaded: *
>
>
>
> 3,028
>
> *Total classes loaded: *
>
>
>
> 3,028
>
> *Total classes unloaded: *
>
>
>
> 0
>
>
>
>
> _This is the output from the run:_
>
> *//*
>
> */java.lang.Exception: Creating a new thread with
> 'experimental-thread-1'/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>
> */at
> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>
> */at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>
> */at
> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>
> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>
> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>
> */java.lang.Exception: Creating a new thread with
> 'experimental-thread-2'/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>
> */at
> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>
> */at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>
> */at
> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>
> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>
> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>
> */java.lang.Exception: Creating a new thread with
> 'experimental-thread-3'/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>
> */at
> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>
> */at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>
> */at
> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>
> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>
> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>
> */java.lang.Exception: Creating a new thread with
> 'experimental-thread-4'/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>
> */at
> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>
> */at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>
> */at
> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>
> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>
> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>
> */java.lang.Exception: Creating a new thread with
> 'experimental-thread-5'/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)/*
>
> */at
> org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)/*
>
> */at
> org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)/*
>
> */at
> org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)/*
>
> */at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)/*
>
> */at
> com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)/*
>
> */at
> com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)/*
>
> */at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)/*
>
> */at com.comverse.mist.app.MistApp.main(MistApp.java:82)/*
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, May 21, 2014 10:37 AM
> *To:* Meltser Tiran
> *Cc:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>'; Broide
> Uri; Shafir Ehud
> *Subject:* Re: Thread pool peculiar behavior
>
> Hi Tiran,
>
> On 20.05.14 23:58, Meltser Tiran wrote:
>
> Second, let me focus on the problem as it seems I wasn't clear
> enough:
>
> /Assume for a moment I don't want to touch the thread pool (we'll
> deal with that later on), and we agree on the size of the Kernel
> and Workers default pools (as you wrote below),/
>
> *Why does Grizzly keep creating/renewing the threads of these 2
> pools constantly and rapidly (I can see in the JConsole the "Total
> threads started" keeps growing although live threads remains
> steady) and doesn't operate them as pools (it can also be seen
> very easily by using the VisualVM profiling tool, even with
> light/moderate load runs)?*
>
> Hmm, that's very strange, can you pls. do one experiment?
> Please override the ThreadFactory like:
>
>
> builder.setWorkerThreadPoolConfig(
> ThreadPoolConfig.defaultConfig().setThreadFactory(new ThreadFactory() {
>
> final AtomicInteger counter = new AtomicInteger();
>
> @Override
> public Thread newThread(Runnable r) {
> final String name = "experimental-thread-" +
> counter.incrementAndGet();
>
> new Exception("Creating a new thread with '" +
> name + "'")
> .printStackTrace(System.out);
>
> return new DefaultWorkerThread(
> AttributeBuilder.DEFAULT_ATTRIBUTE_BUILDER, name,
> null, r);
> }
> })
> );
>
> You'll see the stacktraces of all code paths, which lead to a new
> thread creation. Would you be able to share that info?
>
> Thank you.
>
> WBR,
> Alexey.
>
>
>
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, May 21, 2014 3:28 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Thread pool peculiar behavior
>
> Oh, just realized you don't use HTTP framework... please disregard my
> HttpHandler-related comments.
> Please try to simulate long-lasting tasks in FilterChain (one of your
> Filters can call Thread.sleep(N)) and you have to see worker threads
> to be busy.
>
> If you're concerned about too many threads created - set the core
> thread pool size like:
> /workerPoolConfig.setCorePoolSize(0);
>
> WBR,
> Alexey./
>
> On 20.05.14 16:50, Oleksiy Stashok wrote:
>
> Hi Tiran,
>
>
>
>
>
> Here are the details:
>
> 1)I'm working on a 64 bit Windows workstation, 1 CPU with 4 cores
> and 8GB memory, running Gizzly 2.3.12 with Java 7 VM (development
> machine).
>
> 2)I have observed the threads on load runs (heavy and light loads
> -- it doesn't matter) and found out that *Grizzly keeps replacing
> (creating new) threads* for both the Selector/Kernel "*/pool/*"
> (*grizzly-nio-kernel(1) SelectorRunner* to *grizzly-nio-kernel(5)
> SelectorRunner*)
>
> It's expected, number of kernel threads by default is
> number-of-cores + 1;
>
>
>
>
>
> and the Workers "*/pool/*" (*Grizzly-worker(1)* to
> *Grizzly-worker(8)*)!
>
> also expected, the default number of worker threads is
> number-of-cores * 2;
>
>
>
>
>
> 3)When I try and define a Workers thread pool, I see the threads
> group (*Grizzly(1)* to *Grizzly(100)* if the pool is targeted for
> 100 threads) created and put on *Waiting* state forever!
>
> Hmm, why do you think they put on wait forever? Pls. try to
> implement HttpHandler, which does Thread.sleep(N) inside, this way
> you'll be able to make a threads snapshot and you have to see the
> worker thread stacktraces leading to Thread.sleep(N);
>
> *The questions:*
>
> *1)**How can I cause Gizzly to operate its pools (Selectors/Kernel
> and Workers) as pools and not as factory (meaning don't create new
> threads and if pool and or queue are exhausted refuse new
> events/requests)?*
>
> It's possible to set ExecutorService as worker or kernel
> thread-pool to a Transport instance directly (not via builder),
> but I don't really understand what is the problem with the Builder.
>
>
>
>
>
> *2)**After question 1 is solved, how can I configure Grizzly's
> pools and getting them to work (and not put the pool's threads on
> a Waiting status)?*
>
> As I suggested, let's try to create a simple Thread.sleep(N)
> HttpHandler and see the thread (thread group), which will handle
> the request.
>
>
>
>
>
>
>
> /NIOTransportBuilder transportBuilder =
> TCPNIOTransportBuilder.newInstance();/
>
> //
>
> /// Configure workers pool/
>
> /ThreadPoolConfig workerPoolConfig =
> transportBuilder.getWorkerThreadPoolConfig();/
>
> */// We shouldn't be here according to Grizzly examples, but still
> it turns out we do.../*
>
> */It's expected in 2.3.12.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>
>
>
>
>
> /*
>
> /if (workerPoolConfig == null)/
>
> /{/
>
> /workerPoolConfig = ThreadPoolConfig.defaultConfig();/
>
> /transportBuilder.setWorkerThreadPoolConfig(workerPoolConfig);/
>
> /}/
>
> //
>
> /workerPoolConfig.setCorePoolSize(100);/
>
> /workerPoolConfig.setMaxPoolSize(100);/
>
> /workerPoolConfig.setQueueLimit(1000);/
>
> /workerPoolConfig.setKeepAliveTime(-1,TimeUnit.HOURS);/
>
> //
>
> /NIOTransport transport = transportBuilder.build();/
>
> *Tiran****Meltser *
> System Architect
> Global Products & Operations
> *Comverse* -- /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P **Please think of the environment before printing this email *
>
> ------------------------------------------------------------------------
>
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You."
>
> ------------------------------------------------------------------------
>
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You."
>
> ------------------------------------------------------------------------
>
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You."
>
> ------------------------------------------------------------------------
>
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You."
>
> ------------------------------------------------------------------------
>
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You."
>
>
> ------------------------------------------------------------------------
> "This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You."