users@grizzly.java.net

Re: CPU Usage Profiling

From: Oleksiy Stashok <oleksiy.stashok_at_oracle.com>
Date: Sun, 14 Jun 2015 22:53:29 +0200

Hi Uri,

looks like there is no selector spinning issue, so I suspect what you
observe is normal... however if you want to investigate it more - you
can profile your project including Java core classes, so we may see what
exactly inside Grizzly select consumes CPU, I bet it's gonna be
java.nio.Selector, but we can double check. Also you may want to run the
same test on different jdk version, may be it will show you different
result.

Thanks.

WBR,
Alexey.

On 14.06.15 13:24, Broide Uri wrote:
>
> Hi Alexey,
>
> The following is the profile data snapshot after stopping the load and
> wait much more time.
>
> You can see that the select() method CPU consumption got back to 0.
>
> Thanks
>
> Uri
>
> cid:image004.jpg_at_01D0A6AD.5FC54060
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, June 12, 2015 1:38 AM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: CPU Usage Profiling
>
> Hi Uri,
>
> I saw you were using 1.7, just wanted to double check.
> After stopping the load, can you check what is the CPU consumption
> rate of the server process?
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 11.06.15 19:13, Broide Uri wrote:
>
> Hi Alexey,
>
> As requested, the following is the profile data snapshot after
> stopping the load and waiting till all connections were closed
>
> BTW, we are running with *JDK7* and not 6
>
> Uri
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, June 11, 2015 7:26 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Cc:* Hasid Ronen; Tsukerman Dana
> *Subject:* Re: CPU Usage Profiling
>
> Hmm,
>
> interesting, one more question, could you pls. check how the
> profile data look like when you stop loading the server.
> So first you load the server, make sure that
> DefaultSelectorHandler consumes its 60%+ of CPU and then stop the
> load.
>
> Can you pls. also double check that CPU consumption in general is
> low after you stop the server. Just want to make sure it's not
> related to selector spin problem observed with JDK 6.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 11.06.15 18:10, Broide Uri wrote:
>
> As requested.
>
> As you can see in the screenshot the select method on idle
> process is idle
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, June 11, 2015 12:21 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: CPU Usage Profiling
>
> Hi Uri,
>
> can you pls. profile the server without loading at all to see
> the diff?
> As I mentioned before, Selector.select(...) usually is
> reported as a hotspot, so I don't think your report is
> suspicious, but let's double check.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 11.06.15 09:22, Broide Uri wrote:
>
> Hi Alexey,
>
> Any update?
>
> Thanks
>
> Uri
>
> *From:*Meltser Tiran [mailto:Tiran.Meltser_at_comverse.com]
> *Sent:* Tuesday, June 09, 2015 7:56 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* RE: CPU Usage Profiling
> *Importance:* High
>
> Hi Alexey,
>
> We don’t think this is the case as (I hope) you can see
> the graph is sorted by *Self Time (CPU)*.
>
> *Tiran Meltser***
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381
> M +972-54-5639381
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P****Please think of the environment before printing this
> email***
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, June 09, 2015 7:21 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: CPU Usage Profiling
>
> Hi Uri,
>
> I suspect what you see is the total time spent inside the
> method, but most of the time it's in the idle state
> waiting for I/O events, so it shouldn't consume CPU.
> Please let me know if it's not the case.
>
> WBR,
> Alexey.
>
> On 09.06.15 16:29, Broide Uri wrote:
>
> Hi Alexey,
>
> After stabilizing on 100% Success Rate in heavy load
> on our application we started to perform CPU Usage
> profiling.
>
> We have noticed (you can see the attached screenshot)
> that we are spending 66% from the CPU time in the
> *org.glassfish.nio.DefaultSelectorHandler.select()*
> method.
>
> We are running the load with the following environment:
>
> ·Grizzly 2.3.21
>
> ·JDK 1.7.0_72-b14
>
> ·RHEL 5.X
>
> Any advice on how can we improve the high CPU consumption?
>
> cid:image003.jpg_at_01D0A2D6.6204FEE0
>
> Thanks
>
> Uri
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”