Hi Alexey,
Below you can find the output of the run (I ran 500 client with 500ms delay between them).
You can clearly see 5 threads are created at startup and no new threads are created afterwards.
This aligns with my claim that the Grizzly ignores this pool completely, as I tracked the run in the VisualVM (via the threads tab) and these 5 threads are not active during all the run (they are on Waiting status) while the Selectors and original/built-in Workers threads (Grizzly-worker(x)) are fully working and being replaced with other threads throughout the run.
Also please find below the quote from the JConsole(VM Summary tab) in this short run after it ended:
Live threads:
38
Peak:
56
Daemon threads:
25
Total threads started:
6,544
Current classes loaded:
3,028
Total classes loaded:
3,028
Total classes unloaded:
0
This is the output from the run:
java.lang.Exception: Creating a new thread with 'experimental-thread-1'
at com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)
at org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)
at org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)
at com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)
at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)
at com.comverse.mist.app.MistApp.main(MistApp.java:82)
java.lang.Exception: Creating a new thread with 'experimental-thread-2'
at com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)
at org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)
at org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)
at com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)
at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)
at com.comverse.mist.app.MistApp.main(MistApp.java:82)
java.lang.Exception: Creating a new thread with 'experimental-thread-3'
at com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)
at org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)
at org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)
at com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)
at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)
at com.comverse.mist.app.MistApp.main(MistApp.java:82)
java.lang.Exception: Creating a new thread with 'experimental-thread-4'
at com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)
at org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)
at org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)
at com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)
at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)
at com.comverse.mist.app.MistApp.main(MistApp.java:82)
java.lang.Exception: Creating a new thread with 'experimental-thread-5'
at com.comverse.mist.app.protocol.AbstractNetworkManager$1.newThread(AbstractNetworkManager.java:514)
at org.glassfish.grizzly.threadpool.AbstractThreadPool.startWorker(AbstractThreadPool.java:180)
at org.glassfish.grizzly.threadpool.SyncThreadPool.startWorker(SyncThreadPool.java:131)
at org.glassfish.grizzly.threadpool.SyncThreadPool.<init>(SyncThreadPool.java:83)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.setImpl(GrizzlyExecutorService.java:104)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.<init>(GrizzlyExecutorService.java:82)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.createInstance(GrizzlyExecutorService.java:78)
at org.glassfish.grizzly.nio.NIOTransport.start(NIOTransport.java:447)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindEntryPoint(AbstractNetworkManager.java:200)
at com.comverse.mist.app.protocol.AbstractNetworkManager.bindProtocol(AbstractNetworkManager.java:103)
at com.comverse.mist.app.MistApp.startupProtocolsListeners(MistApp.java:445)
at com.comverse.mist.app.MistApp.startApp(MistApp.java:145)
at com.comverse.mist.app.MistApp.main(MistApp.java:82)
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Wednesday, May 21, 2014 10:37 AM
To: Meltser Tiran
Cc: 'users_at_grizzly.java.net'; Broide Uri; Shafir Ehud
Subject: Re: Thread pool peculiar behavior
Hi Tiran,
On 20.05.14 23:58, Meltser Tiran wrote:
Second, let me focus on the problem as it seems I wasn't clear enough:
Assume for a moment I don't want to touch the thread pool (we'll deal with that later on), and we agree on the size of the Kernel and Workers default pools (as you wrote below),
Why does Grizzly keep creating/renewing the threads of these 2 pools constantly and rapidly (I can see in the JConsole the "Total threads started" keeps growing although live threads remains steady) and doesn't operate them as pools (it can also be seen very easily by using the VisualVM profiling tool, even with light/moderate load runs)?
Hmm, that's very strange, can you pls. do one experiment?
Please override the ThreadFactory like:
builder.setWorkerThreadPoolConfig(
ThreadPoolConfig.defaultConfig().setThreadFactory(new ThreadFactory() {
final AtomicInteger counter = new AtomicInteger();
@Override
public Thread newThread(Runnable r) {
final String name = "experimental-thread-" + counter.incrementAndGet();
new Exception("Creating a new thread with '" + name + "'")
.printStackTrace(System.out);
return new DefaultWorkerThread(
AttributeBuilder.DEFAULT_ATTRIBUTE_BUILDER, name,
null, r);
}
})
);
You'll see the stacktraces of all code paths, which lead to a new thread creation. Would you be able to share that info?
Thank you.
WBR,
Alexey.
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Wednesday, May 21, 2014 3:28 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: Thread pool peculiar behavior
Oh, just realized you don't use HTTP framework... please disregard my HttpHandler-related comments.
Please try to simulate long-lasting tasks in FilterChain (one of your Filters can call Thread.sleep(N)) and you have to see worker threads to be busy.
If you're concerned about too many threads created - set the core thread pool size like:
workerPoolConfig.setCorePoolSize(0);
WBR,
Alexey.
On 20.05.14 16:50, Oleksiy Stashok wrote:
Hi Tiran,
Here are the details:
1) I'm working on a 64 bit Windows workstation, 1 CPU with 4 cores and 8GB memory, running Gizzly 2.3.12 with Java 7 VM (development machine).
2) I have observed the threads on load runs (heavy and light loads - it doesn't matter) and found out that Grizzly keeps replacing (creating new) threads for both the Selector/Kernel "pool" (grizzly-nio-kernel(1) SelectorRunner to grizzly-nio-kernel(5) SelectorRunner)
It's expected, number of kernel threads by default is number-of-cores + 1;
and the Workers "pool" (Grizzly-worker(1) to Grizzly-worker(8))!
also expected, the default number of worker threads is number-of-cores * 2;
3) When I try and define a Workers thread pool, I see the threads group (Grizzly(1) to Grizzly(100) if the pool is targeted for 100 threads) created and put on Waiting state forever!
Hmm, why do you think they put on wait forever? Pls. try to implement HttpHandler, which does Thread.sleep(N) inside, this way you'll be able to make a threads snapshot and you have to see the worker thread stacktraces leading to Thread.sleep(N);
The questions:
1) How can I cause Gizzly to operate its pools (Selectors/Kernel and Workers) as pools and not as factory (meaning don't create new threads and if pool and or queue are exhausted refuse new events/requests)?
It's possible to set ExecutorService as worker or kernel thread-pool to a Transport instance directly (not via builder), but I don't really understand what is the problem with the Builder.
2) After question 1 is solved, how can I configure Grizzly's pools and getting them to work (and not put the pool's threads on a Waiting status)?
As I suggested, let's try to create a simple Thread.sleep(N) HttpHandler and see the thread (thread group), which will handle the request.
NIOTransportBuilder transportBuilder = TCPNIOTransportBuilder.newInstance();
// Configure workers pool
ThreadPoolConfig workerPoolConfig = transportBuilder.getWorkerThreadPoolConfig();
// We shouldn't be here according to Grizzly examples, but still it turns out we do...
It's expected in 2.3.12.
Thanks.
WBR,
Alexey.
if (workerPoolConfig == null)
{
workerPoolConfig = ThreadPoolConfig.defaultConfig();
transportBuilder.setWorkerThreadPoolConfig(workerPoolConfig);
}
workerPoolConfig.setCorePoolSize(100);
workerPoolConfig.setMaxPoolSize(100);
workerPoolConfig.setQueueLimit(1000);
workerPoolConfig.setKeepAliveTime(-1,TimeUnit.HOURS);
NIOTransport transport = transportBuilder.build();
Tiran Meltser
System Architect
Global Products & Operations
Comverse - Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<
http://www.comverse.com/>
P Please think of the environment before printing this email
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You."
________________________________
"This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com. Thank You."