users@grizzly.java.net

Re: Grizzly Thread Pool Monitoring

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Mon, 16 Mar 2015 09:27:11 -0700

Ok, let's do this, there is a monitoring sample in our docs [1]
Can you pls. run it with the snapshot and tell me if the values are
correct for you?

Thank you.

WBR,
Alexey.

public static void main(String[] args) {

         final GrizzlyJmxManager manager = GrizzlyJmxManager.instance();
         final TCPNIOTransport transport1 = TCPNIOTransportBuilder.newInstance().build();
         final TCPNIOTransport transport2 = TCPNIOTransportBuilder.newInstance().build();
         try {
             Object jmxTransportObject1 =
                     transport1.getMonitoringConfig().createManagementObject();

             Object jmxTransportObject2 =
                     transport2.getMonitoringConfig().createManagementObject();

             manager.registerAtRoot(jmxTransportObject1,"Transport1");
             manager.registerAtRoot(jmxTransportObject2,"Transport2");
             transport1.start();
             transport1.bind(9999);
             System.out.println("Press any key to stop the example...");
             System.in.read();
         }catch (IOException ioe) {
             ioe.printStackTrace();
             System.exit(1);
         }finally {
             try {
                 transport1.shutdownNow();
             }catch (IOException ignored) {}
         }
}


[1] https://grizzly.java.net/monitoring.html

On 16.03.15 07:35, Broide Uri wrote:
>
> Hi Alexey,
>
> Unfortunately it would be very difficult to supply a test case due to
> the complexity of our application. Is there anything else we can open
> (Grizzly loggers, etc ..)?
>
> Can you enter some System.out in order for us to make sure that we are
> working with the right version (although I am pretty sure we do,
> according to the grizzly-framework.jar Manifest).
>
> Anyhow, note that :
>
> 1.the *thread-pool-allocated-thread-count* value is _always_
> stabilized with *thread-pool-max-num-threads* minus
> *thread-pool-core-pool-size*
>
> 2.*thread-pool-core-pool-size*is always set to 12
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Monday, March 16, 2015 9:33 AM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hmm,
>
> can you pls. provide the testcase, because I can't reproduce the
> problem after the change I made.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 16.03.15 00:30, Broide Uri wrote:
>
> Hi Alexey,
>
> Right, even started is reported wrong as you can see from the
> screenshot below (that was taken NOT under load).
>
> And yes we verified that it is the right version (2.3.20-SNAPSHOT)
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Sunday, March 15, 2015 5:21 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> you mean even "started" is reported wrong?
> It should be something wrong with the version you picked up.
>
> Did you use this snapshots repository:
> https://maven.java.net/content/repositories/snapshots/
> <http://cp.mcafee.com/d/5fHCNEg6gUqdEI9KccFL6zB4TsSCyPtdB55dUSCyPtdB5ZdUSCyPtB5UsYOrhLtZNdxddYST6n48RAxUanc_6DYKrnc_6DYKrs7cW_tZ_HYy-OqekNPfnKnjjupVdYttVMQsYJt6OaaJS6ul3PWApmU6CQjqpK_9IInojvvpjdTdAVPmEBCaWp-72eCtU6VqI9_2uhZqJ9jH6nQOAdmEjHeC6SWv6xuFy7N1h6cxgkfbs8WpTwqrhu79FLCMnWhEw2kp2wEumd4099N_k3h05GRfd40Tm4FRza14SOMrnVhOYlH0>
>
> Thank you.
>
> WBR,
> Alexey.
>
>
> On 15.03.15 01:47, Broide Uri wrote:
>
> Hi Alexey,
>
> Unfortunately the fix doesn’t work. Can we gather some more
> information on this?
>
> Please note that we have a protection mechanism that monitors
> the usage of the thread pools, and currently it blocks traffic
> based on wrong reporting in the thread pool JMX objects.
>
> Thanks
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Wednesday, March 11, 2015 9:22 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
>
>
> On 11.03.15 07:54, Broide Uri wrote:
>
> 1.We opened the requested BUG (GRIZZLY-1748
> <http://cp.mcafee.com/d/avndzgQrhojshKye7c6XCQQmrFIEEFL6QQmrFIELFL6QQmrIEL3DCjqdXLK9I9FLCSUOUx6IAf1iVDUQ_BPqVDUQ_BPrZT3hPuXX_nV5wsMzvHTbFTovVdVBNV7BHFShhlhKPOEuvkzaT0QSyrhdTV5MQsFIT7e6zBcTsSjDdqymo8WpTwrwzaeUgzkNjY2gb3g90AawsKyeoudJQ-d2Zj4fy2ycp2wEumUhQPL0QSyZtB5MS2_id40iz8k53ONEw19efWwq80JmFVEw6WMBeIpg8CSm3qCJWcYkr>).
>
> 2.Can you answer my Questions below
>
> The questions are as follows:
>
> 1.Are we looking on the right MBean attributes?
>
> Yes.
>
>
>
>
> 2.Assuming 1 is true, then why for the same load model the
> thread pool is reaching this high number (max = core + allocated)?
>
> It's because the probe values reported based on events coming
> from the thread pool and at the time of probe registration
> some events are already missed.
>
>
>
>
> 3.Why the *thread-pool-started* attribute is false?
>
> same as above.
>
> I think I've fixed the issue, can you pls. check the latest
> 2.3.20-SNAPSHOT?
>
> Thanks.
>
> WBR,
> Alexey.
>
>
>
>
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, March 10, 2015 8:15 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: Grizzly Thread Pool Monitoring
>
> Hi Uri,
>
> looks like a bug to me, 12 core threads are not counted as
> allocated.
> Can you pls. file an issue?
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 10.03.15 10:26, Broide Uri wrote:
>
> Hi
>
> We are performing Stability load run against our Grizzly
> based application.
>
> We are trying to determine how to monitor the Thread Pool
> usage via JMX.
>
> We are looking at the ThreadPool MBean
> (org.glassfish.grizzly:pp=/gmbal-root/TCPNIOTransport[EndPoint.IMAP.NON_SECURED],type=ThreadPool,name=ThreadPool),
> on the following attributes:
>
> ·thread-pool-max-num-threads
>
> ·thread-pool-allocated-thread-count
>
> ·thread-pool-core-pool-size
>
> We have conducted 2 load runs (with *_exactly_* the same
> load model):
>
> 1.max threads was configured to be *200* (verified against
> thread-pool-max-num-threads)
>
> ofrom some point of the load run the
> *thread-pool-allocated-thread-count* was stabilized on 188
>
> o*thread-pool-core-pool-size* was fixed on 12
>
> 2.max threads was configured to be *300* (verified against
> thread-pool-max-num-threads)
>
> ofrom some point of the load run the
> *thread-pool-allocated-thread-count* was stabilized on 288
>
> o*thread-pool-core-pool-size* was fixed on 12
>
> The questions are as follows:
>
> 1.Are we looking on the right MBean attributes?
>
> 2.Assuming 1 is true, then why for the same load model the
> thread pool is reaching this high number (max = core +
> allocated)?
>
> 3.Why the *thread-pool-started* attribute is false?
>
> Attaching the JConsole print screen of the second run
>
> cid:image001.png_at_01D05B52.9D7A53B0
>
> Thanks
>
> Uri
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”