users@grizzly.java.net

Re: Grizzly Thread Pool Monitoring

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Sun, 24 May 2015 23:32:16 -0700

It's been just released.

WBR,
Alexey.


On 24.05.15 23:30, Broide Uri wrote:
>
> Thanks Alexey.
>
> Can you share with us when 2.3.21 shall be ready?
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, May 22, 2015 3:11 AM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> it's a problem with SyncThreadPool [1].
> I've just fixed it.
>
> Thank you for reporting the problem!
>
> WBR,
> Alexey.
>
> [1] https://java.net/jira/browse/GRIZZLY-1773
> <http://cp.mcafee.com/d/2DRPoA820scCQm6kQTSm1RPtPqqbdQSkkkTzqqbdQSknQTzqqbdSknxPP9J6ZTT4S4QTPrspsgzmi7wFsPYqvOVJsPYqvOVJV5wtXXX_nV4sUO-MeWZOWq9EVLPbOdQSjhPORQr8EGThvVkffGhBrwqrhdI6XYyUqekSrzD3hOCrKr9PCJhbc4tcXMdMhB7s8hGoF-185xE4wi5genh73CjrtfzgLkN3UwEz6gEa7BK4tcXMddIIefEL6MnWhEw2kp2wEumd4099N_k3h05GRfd40Tm4FRza14SOMrvcJRj4vtDIIX>
>
> On 21.05.15 06:19, Broide Uri wrote:
>
> Hi Alexey,
>
> Following your recommendation we configured the
> *keepAliveTimeout*, ran load and have noticed that there is no
> releasing of IDLE Threads from the
> *thread-pool-allocated-thread-count*
>
> Note that we are using the latest Grizzly 2.3.20
>
> Any suggestions why?
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, April 21, 2015 3:46 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> releasing... sorry for the delay.
>
> WBR,
> Alexey.
>
> On 20.04.15 04:48, Broide Uri wrote:
>
> Hi Alexey,
>
> Any update regarding the latest formal Grizzly release
> (*2.3.20*) ?
>
> Thanks
>
> Uri
>
> *From:*Broide Uri
> *Sent:* Tuesday, April 14, 2015 3:50 PM
> *To:* 'users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>'
> *Subject:* RE: Grizzly Thread Pool Monitoring
>
> Hi Alexey,
>
> Embedded
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, April 13, 2015 10:47 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> On 13.04.15 08:32, Broide Uri wrote:
>
> 1.Can you please share with us when the latest formal
> Grizzly release (*2.3.20*) is planned?
>
> The end of this week, is it ok?*/[Broide Uri] /* Of course,
> whenever you can
>
>
>
> 2.We have done a load run to see the behavior of the new
> attribute you have (*thread-pool-busy-thread-count*) added and
> have noticed that is very low (in average) comparing to the
> *thread-pool-allocated-thread-count.*
>
> Have you ever considered implementing some kind of pool
> reduction mechanism (attached a screenshot of last 8 hours of
> our 31 hours load run on our server), unless there is such a
> mechanism we are not aware of?
>
> What kind of pool configuration are you using? I suspect core
> and max sizes are different, aren't they? In that case you can
> specify keepAliveTimeout so the thread pool could be shrink
> down to its core pool size.*/[Broide Uri] /* OK, We will try
> to configure it (keepAliveTimeout) when getting the official
> Grizzly
>
> 3.why is the tasks queue (*thread-pool-queued-task-count*) is
> active despite of the fact the pool never reached its maximum
> limit (according to the allocated threads count)?
>
> If you're actively loading the thread-pool, its pools are not
> able to keep up, because it takes some time to pass the task
> to a thread and it takes even more time to spawn a new
> thread... so the queue is growing.*/[Broide Uri] /* Thanks
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>
> Thanks Uri.
>
> imap://oleksiy%2Estashok%40oracle%2Ecom_at_stbeehive.oracle.com:993/fetch%3EUID%3E/Java.Net/Grizzly/users%3E1860?header=quotebody&part=1.2&filename=image003.jpg
>
> *From:*Broide Uri
> *Sent:* Thursday, April 02, 2015 10:16 AM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Cc:* Tsukerman Dana
> *Subject:* RE: Grizzly Thread Pool Monitoring
>
> Thanks allot Alexey,
>
> I will and keep you updated
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, April 01, 2015 7:34 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> it's done, please check the latest 2.3.20-SNAPSHOT
>
> WBR,
> Alexey.
>
> On 31.03.15 01:32, Broide Uri wrote:
>
> Hi Alexey,
>
> Thanks allot, I opened *GRIZZLY-1756* in Jira as you
> requested.
>
> *GRIZZLY-1755*was opened by mistake by me, please discard
> it (I didn't find out how to do it myself)
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, March 30, 2015 8:11 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi,
>
> sorry, I've just come back from vacation.
> After quick check it looks like there is no attribute
> showing the number of active threads in the pool (the
> ones, which are currently processing tasks).
> Can you pls. file an RFE - I'll add this feature today and
> let you know.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 30.03.15 09:43, Broide Uri wrote:
>
> Alexey?
>
> *From:*Broide Uri
> *Sent:* Wednesday, March 25, 2015 7:41 AM
> *To:* 'users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>'
> *Subject:* RE: Grizzly Thread Pool Monitoring
>
> Hi Alexey,
>
> Kind reminder
>
> Thanks
>
> Uri
>
> *From:*Broide Uri
> *Sent:* Sunday, March 22, 2015 10:10 AM
> *To:* 'users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>'
> *Subject:* RE: Grizzly Thread Pool Monitoring
>
> Thanks Alexey,
>
> We have looked for this attribute and didn't find.
>
> Can share the exact attribute name with us?
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, March 19, 2015 6:08 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* RE: Grizzly Thread Pool Monitoring
>
> Hi Uri,
> allocated thread count represents the number of
> threads in the thread pool, which could be either idle
> or active. There should be another value (i'm writing
> from the phone, so cant check the exact name), that
> represents the number of threads, which are busy
> processing tasks.
>
> On Mar 19, 2015 3:49 AM, Broide Uri
> <Uri.Broide_at_comverse.com
> <mailto:Uri.Broide_at_comverse.com>> wrote:
>
> Hi Alexey,
>
> It looks that your 2.3.20-SNAPSHOTfixed the problem
> (by mistake I took only grizzly-framework.jar without
> the grizzly-framework-monitoring.jar).
>
> One more clarification, is the
> *thread-pool-allocated-thread-count* is indeed the
> current active threads count ?
>
> Alexey, thanks again your quick response
>
> Uri
>
> cid:image006.jpg_at_01D06229.E7BFEB00
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, March 16, 2015 6:27 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Ok, let's do this, there is a monitoring sample in our
> docs [1]
> Can you pls. run it with the snapshot and tell me if
> the values are correct for you?
>
> Thank you.
>
> WBR,
> Alexey.
>
> public static void main(String[] args) {
>
>
>
> final GrizzlyJmxManager manager = GrizzlyJmxManager.instance();
>
> final TCPNIOTransport transport1 = TCPNIOTransportBuilder.newInstance().build();
>
> final TCPNIOTransport transport2 = TCPNIOTransportBuilder.newInstance().build();
>
> try {
>
> Object jmxTransportObject1 =
>
> transport1.getMonitoringConfig().createManagementObject();
>
>
>
> Object jmxTransportObject2 =
>
> transport2.getMonitoringConfig().createManagementObject();
>
>
>
> manager.registerAtRoot(jmxTransportObject1, "Transport1");
>
> manager.registerAtRoot(jmxTransportObject2, "Transport2");
>
> transport1.start();
>
> transport1.bind(9999);
>
> System.out.println("Press any key to stop the example...");
>
> System.in.read();
>
> } catch (IOException ioe) {
>
> ioe.printStackTrace();
>
> System.exit(1);
>
> } finally {
>
> try {
>
> transport1.shutdownNow();
>
> } catch (IOException ignored) {}
>
> }
>
> }
>
>
>
>
> [1] https://grizzly.java.net/monitoring.html
> <http://cp.mcafee.com/d/1jWVIp4x0SyMCU-COCeKrKrjhpKCOyyCYrjhpKCOy-CYrjhpKOyYeupdETK-UCMCC-rrzby4qOgY5bCvzj-ndHCvzj-ndKzCbzHLLZvAnHCzCVEVjWZOWqpEVvVdOWpEVd7bfbnjIyCHss_OEuvkzaT0QSyrpdTV55YsYqem1P8UTsSjDdqymohz8k53OT2eCtU6UHm4FRza17wsrrFYq5WC8v454oO51gYJMzFDu1FJBMszsQsCMnWhEw2kp2wEumd4099N_k3h05GRfd40Tm4FRza14SOMr4DMJ>
>
> On 16.03.15 07:35, Broide Uri wrote:
>
> Hi Alexey,
>
> Unfortunately it would be very difficult to supply
> a test case due to the complexity of our
> application. Is there anything else we can open
> (Grizzly loggers, etc ..)?
>
> Can you enter some System.out in order for us to
> make sure that we are working with the right
> version (although I am pretty sure we do,
> according to the grizzly-framework.jar Manifest).
>
> Anyhow, note that :
>
> 1.the *thread-pool-allocated-thread-count* value
> is _always_ stabilized with
> *thread-pool-max-num-threads* minus
> *thread-pool-core-pool-size*
>
> 2.*thread-pool-core-pool-size*is always set to 12
>
> Uri
>
> cid:image003.jpg_at_01D06229.26AAE6E0
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, March 16, 2015 9:33 AM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hmm,
>
> can you pls. provide the testcase, because I can't
> reproduce the problem after the change I made.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 16.03.15 00:30, Broide Uri wrote:
>
> Hi Alexey,
>
> Right, even started is reported wrong as you
> can see from the screenshot below (that was
> taken NOT under load).
>
> And yes we verified that it is the right
> version (2.3.20-SNAPSHOT)
>
> Thanks
>
> Uri
>
> cid:image004.jpg_at_01D06229.26AAE6E0
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Sunday, March 15, 2015 5:21 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> you mean even "started" is reported wrong?
> It should be something wrong with the version
> you picked up.
>
> Did you use this snapshots repository:
> https://maven.java.net/content/repositories/snapshots/
> <http://cp.mcafee.com/d/5fHCNEg6gUqdEI9KccFL6zB4TsSCyPtdB55dUSCyPtdB5ZdUSCyPtB5UsYOrhLtZNdxddYST6n48RAxUanc_6DYKrnc_6DYKrs7cW_tZ_HYy-OqekNPfnKnjjupVdYttVMQsYJt6OaaJS6ul3PWApmU6CQjqpK_9IInojvvpjdTdAVPmEBCaWp-72eCtU6VqI9_2uhZqJ9jH6nQOAdmEjHeC6SWv6xuFy7N1h6cxgkfbs8WpTwqrhu79FLCMnWhEw2kp2wEumd4099N_k3h05GRfd40Tm4FRza14SOMrnVhOYlH0>
>
> Thank you.
>
> WBR,
> Alexey.
>
> On 15.03.15 01:47, Broide Uri wrote:
>
> Hi Alexey,
>
> Unfortunately the fix doesn’t work. Can we
> gather some more information on this?
>
> Please note that we have a protection
> mechanism that monitors the usage of the
> thread pools, and currently it blocks
> traffic based on wrong reporting in the
> thread pool JMX objects.
>
> Thanks
>
> Uri
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, March 11, 2015 9:22 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
>
>
> On 11.03.15 07:54, Broide Uri wrote:
>
> 1.We opened the requested BUG
> (GRIZZLY-1748
> <http://cp.mcafee.com/d/avndzgQrhojshKye7c6XCQQmrFIEEFL6QQmrFIELFL6QQmrIEL3DCjqdXLK9I9FLCSUOUx6IAf1iVDUQ_BPqVDUQ_BPrZT3hPuXX_nV5wsMzvHTbFTovVdVBNV7BHFShhlhKPOEuvkzaT0QSyrhdTV5MQsFIT7e6zBcTsSjDdqymo8WpTwrwzaeUgzkNjY2gb3g90AawsKyeoudJQ-d2Zj4fy2ycp2wEumUhQPL0QSyZtB5MS2_id40iz8k53ONEw19efWwq80JmFVEw6WMBeIpg8CSm3qCJWcYkr>).
>
> 2.Can you answer my Questions below
>
> The questions are as follows:
>
> 1.Are we looking on the right MBean
> attributes?
>
> Yes.
>
>
>
>
> 2.Assuming 1 is true, then why for the
> same load model the thread pool is
> reaching this high number (max = core +
> allocated)?
>
> It's because the probe values reported
> based on events coming from the thread
> pool and at the time of probe registration
> some events are already missed.
>
>
>
>
> 3.Why the *thread-pool-started* attribute
> is false?
>
> same as above.
>
> I think I've fixed the issue, can you pls.
> check the latest 2.3.20-SNAPSHOT?
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>
>
>
> *From:* Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, March 10, 2015 8:15 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> looks like a bug to me, 12 core threads
> are not counted as allocated.
> Can you pls. file an issue?
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 10.03.15 10:26, Broide Uri wrote:
>
> Hi
>
> We are performing Stability load run
> against our Grizzly based application.
>
> We are trying to determine how to
> monitor the Thread Pool usage via JMX.
>
> We are looking at the ThreadPool MBean
> (org.glassfish.grizzly:pp=/gmbal-root/TCPNIOTransport[EndPoint.IMAP.NON_SECURED],type=ThreadPool,name=ThreadPool),
> on the following attributes:
>
> ·thread-pool-max-num-threads
>
> ·thread-pool-allocated-thread-count
>
> ·thread-pool-core-pool-size
>
> We have conducted 2 load runs (with
> *_exactly_* the same load model):
>
> 1.max threads was configured to be
> *200* (verified against
> thread-pool-max-num-threads)
>
> ofrom some point of the load run the
> *thread-pool-allocated-thread-count*
> was stabilized on 188
>
> o*thread-pool-core-pool-size* was
> fixed on 12
>
> 2.max threads was configured to be
> *300* (verified against
> thread-pool-max-num-threads)
>
> ofrom some point of the load run the
> *thread-pool-allocated-thread-count*
> was stabilized on 288
>
> o*thread-pool-core-pool-size* was
> fixed on 12
>
> The questions are as follows:
>
> 1.Are we looking on the right MBean
> attributes?
>
> 2.Assuming 1 is true, then why for the
> same load model the thread pool is
> reaching this high number (max = core
> + allocated)?
>
> 3.Why the *thread-pool-started*
> attribute is false?
>
> Attaching the JConsole print screen of
> the second run
>
> cid:image001.png_at_01D05B52.9D7A53B0
>
> Thanks
>
> Uri
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain
> confidential, commercial or privileged
> information that constitutes
> proprietary information of Comverse
> Inc. or its subsidiaries. If you are
> not the intended recipient of this
> message, you are hereby notified that
> any review, use or distribution of
> this information is absolutely
> prohibited and we request that you
> delete all copies and contact us by
> e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank
> You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain
> confidential, commercial or privileged
> information that constitutes proprietary
> information of Comverse Inc. or its
> subsidiaries. If you are not the intended
> recipient of this message, you are hereby
> notified that any review, use or
> distribution of this information is
> absolutely prohibited and we request that
> you delete all copies and contact us by
> e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain
> confidential, commercial or privileged
> information that constitutes proprietary
> information of Comverse Inc. or its
> subsidiaries. If you are not the intended
> recipient of this message, you are hereby
> notified that any review, use or
> distribution of this information is
> absolutely prohibited and we request that
> you delete all copies and contact us by
> e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message,
> you are hereby notified that any review, use
> or distribution of this information is
> absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing
> to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of Comverse
> Inc. or its subsidiaries. If you are not the
> intended recipient of this message, you are hereby
> notified that any review, use or distribution of
> this information is absolutely prohibited and we
> request that you delete all copies and contact us
> by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”