You could assign dedicated accepting selector thread, it may help accept
clients even if other I/O threads busy with SSL handshake. You can also
try to use worker thread strategy to offload I/O threads, or use same
thread strategy, but delegate handshake operation to custom thread pool,
so you can always control number of handshaking connections and don't
allow them to consume all CPUs.
Thanks.
WBR,
Alexey.
On 20.07.15 09:57, Ben-Yosef Efrat wrote:
>
> The test wasn’t the one who crashed it was our application since we
> run on a very high load and handshakes every second. I got no cores
> but the CPU was so high that it caused long response times and new
> users also couldn’t connect
>
> *From:*Riyad Kalla [mailto:rkalla_at_gmail.com]
> *Sent:* Thursday, July 16, 2015 9:05 PM
> *To:* users_at_grizzly.java.net
> *Subject:* Re: High CPU Usage on SSL utils
>
> The performance gap could be narrowed down simply to handshake (takes
> quite a bit longer), but Efrat's point about the test crashing after
> 20mins is something worthy of investigation it seems...
>
> Efrat, did you get an exception or core dump from the VM? Wondering if
> Grizzly is leaking resources (Exception) or if something more native
> and horrible is happening (VM core dump).
>
> On Mon, Jul 13, 2015 at 8:36 AM, Ben-Yosef Efrat
> <Efrat.Ben-Yosef_at_comverse.com <mailto:Efrat.Ben-Yosef_at_comverse.com>>
> wrote:
>
> Hi Alexey,
>
> Attached is the tester you asked for (change the suffix to zip after
> downloading).
>
> The results are relatively the same as the previous ones .
>
> BTW – we don’t know what is the key size – the tester is using the
> grizzly samples jks (I guess you can change it if necessary),
> reminding you that we see the problem with 2048 size key.
>
> Thanks
>
> Efrat and Tiran
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com
> <mailto:oleksiy.stashok_at_oracle.com>]
> *Sent:* Friday, July 10, 2015 1:35 AM
>
>
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: High CPU Usage on SSL utils
>
> Hi Tiran,
>
> can you pls. create a simple client/server reproducer for the problem,
> which includes SSL configuration, so I know I have the same scenario
> as you do?
> IMO we have to understand what exactly JSSE doing that takes so long time.
>
> Thank you.
>
> WBR,
> Alexey.
>
> On 07.07.15 06:49, Meltser Tiran wrote:
>
> Hi Alexey,
>
> In theory you are right.
>
> Unfortunately we don’t control the IMAP clients and from what we
> see in the field, they perform short sessions…
>
> Do you know perhaps on any special settings in Grizzly/Java that
> can optimize the SSL handshake (SSL acceleration)?
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381 <tel:%2B972-3-7678381>
> M +972-54-5639381 <tel:%2B972-54-5639381>
> Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P**Please think of the environment before printing this email*
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Tuesday, July 07, 2015 4:32 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: High CPU Usage on SSL utils
>
> Sorry, I meant IMAP. AFAIU there is keep-alive mechanism IMAP
> IDLE, which could be used to keep connection open and receive
> server notification once there is a new email.
> I believe the main problem you have is initial SSL handshake,
> which really could be expensive, and I think you can rid off, or
> reduce the effect of the expensive SSL handshake by using IMAP
> keep-alive mechanism.
>
> WBR,
> Alexey.
>
> On 06.07.15 14:53, Ben-Yosef Efrat wrote:
>
> Hi Alexey,
>
> We are not using SMTP, we are using IMAP Protocol.
>
> We also noticed that this issue does not occur if we reduce
> the key size (at first it was 2048, and we reduced it to 512).
>
> Of course we cannot run with 512. Any Ideas?
>
> Thanks
>
> Efrat
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, July 02, 2015 6:44 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: High CPU Usage on SSL utils
>
> Hi guys,
>
> IMO what you're currently measuring is SSL handshake
> performance vs. no-handshake in plain connection.
> I quickly googled for SMTP protocol and looks like there is
> possibility to keep connection alive, IMO it's essential,
> especially for secured use case.
>
> To be more fair, try to run Echo benchmark, but not with 10
> messages, but for example 100K or more, it should remove
> impact of SSL handshake.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 30.06.15 16:05, Ben-Yosef Efrat wrote:
>
> Sorry, couldn’t attach the file. Please see it in
> ftp://transfer.comverse.com/outgoing/SSLVsPlain/SSLVsPlain.zip
>
>
> *From:*Ben-Yosef Efrat [mailto:Efrat.Ben-Yosef_at_comverse.com]
> *Sent:* Tuesday, June 30, 2015 4:41 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* RE: High CPU Usage on SSL utils
>
> Hi Alexey,
>
> We spent that last week on profiling this issue. We did
> the following:
> 1. Run our application on plain vs. Ssl when the
> application basically does nothing (respond with hard
> coded OK)
>
> 2. Run Grizzly SSL/Plain echo example under load
>
> 3. run Jprofiler, VisualVm and netbeans profiler
>
> All showed the same:
>
> * CPU increases dramatically – over 60% on some cases
>
> * memory increase by 40% and more on some cases
>
> *
> org.glassfish.grizzly.ssl.SSlUtils.executeDelegatedTask()
> is the most problematic methods CPU wise
>
> We cannot run our application this way at all. On ssl
> everything crashes after 20 min.
>
> Attached are some screenshots and the code for the
> ssl/Plain echo load test we run.
>
> Please advise
>
> Efrat
>
> *From:*Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Friday, June 19, 2015 8:18 PM
> *To:* users_at_grizzly.java.net <mailto:users_at_grizzly.java.net>
> *Subject:* Re: High CPU Usage on SSL utils
>
> Hi Tiran,
>
> I see your concern, but IMO it's similar to what you found
> w.r.t. "selector", if you enable "core classes" profiling
> - you'll see that it's core SSL implementation consumes
> CPU, not Grizzly itself. Once you have the profiling
> results with core classes, we can check what exactly
> consumes that much time.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On 18.06.15 20:07, Meltser Tiran wrote:
>
> Hi Alexey (working with Efrat and Uri) J,
>
> First we don’t keep the connections open as this is an
> IMAP server (or proxy).
>
> Second, I agree that SSL channel should have some
> performance price but what we experience is far beyond
> a reasonable price, after making another check we see
> we have response times slower *600* times than plain
> communication…
>
> Third, we are running on a 16 cores machine in which
> SSL should hardly be noticed.
>
> Forth, we have just compared the SSL experience to our
> previous server written in C++ some more than 10 years
> ago and operated on machines with about ¼ CPU power
> (or even less), and load results show response time
> was doubled at max (*not 600 times worse*).
>
> Can you have a look on the hot method the profiler is
> showing and see if something can be done there (or
> maybe give us some guideline)?
>
> *Tiran Meltser*
> System Architect
> Global Products & Operations
> *Comverse* – /Making Your Network Smarter/
>
> T +972-3-7678381 <tel:%2B972-3-7678381>
> M +972-54-5639381 <tel:%2B972-54-5639381>
> Tiran.Meltser_at_comverse.com
> <mailto:Tiran.Meltser_at_comverse.com>
> *www.comverse.com <http://www.comverse.com/> *
>
> *P**Please think of the environment before printing
> this email*
>
> *From:*Oleksiy Stashok
> [mailto:oleksiy.stashok_at_oracle.com]
> *Sent:* Thursday, June 18, 2015 8:16 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Subject:* Re: High CPU Usage on SSL utils
>
> Hi Efrat,
>
> do you keep connections alive?
> In general SSL is not for free and it's expected to be
> slower than plain communication.
>
> WBR,
> Alexey.
>
> On 18.06.15 18:14, Ben-Yosef Efrat wrote:
>
> Also, we see higher response times then before
> (around 50 times longer)
>
> *From:*Ben-Yosef Efrat
> [mailto:Efrat.Ben-Yosef_at_comverse.com]
> *Sent:* Thursday, June 18, 2015 6:54 PM
> *To:* users_at_grizzly.java.net
> <mailto:users_at_grizzly.java.net>
> *Cc:* Broide Uri; Meltser Tiran
> *Subject:* High CPU Usage on SSL utils
>
> Hi Alexey,
>
> Continuing Uri’s query about the CPU, we moved to
> java8 profiler and as you said, indeed not the
> select() was the issue.
>
> Now, we added the SSL to the picture (utilizing
> Grizzly’s SSLfilter) and as you can see in the
> attached screenshot, the executeDelegatedTask()
> method consumes a lot of CPU (we sort by self-tile
> CPU).
>
> Please advise,
>
> Thanks,
>
> Efrat
>
> cid:image001.png_at_01D0A9F5.776D0720
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of Comverse
> Inc. or its subsidiaries. If you are not the
> intended recipient of this message, you are hereby
> notified that any review, use or distribution of
> this information is absolutely prohibited and we
> request that you delete all copies and contact us
> by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that
> constitutes proprietary information of Comverse
> Inc. or its subsidiaries. If you are not the
> intended recipient of this message, you are hereby
> notified that any review, use or distribution of
> this information is absolutely prohibited and we
> request that you delete all copies and contact us
> by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential,
> commercial or privileged information that constitutes
> proprietary information of Comverse Inc. or its
> subsidiaries. If you are not the intended recipient of
> this message, you are hereby notified that any review,
> use or distribution of this information is absolutely
> prohibited and we request that you delete all copies
> and contact us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial
> or privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you
> are not the intended recipient of this message, you are
> hereby notified that any review, use or distribution of
> this information is absolutely prohibited and we request
> that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>.
> Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary
> information of Comverse Inc. or its subsidiaries. If you are
> not the intended recipient of this message, you are hereby
> notified that any review, use or distribution of this
> information is absolutely prohibited and we request that you
> delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any
> review, use or distribution of this information is absolutely
> prohibited and we request that you delete all copies and contact
> us by e-mailing to: security_at_comverse.com
> <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
>
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com <mailto:security_at_comverse.com>. Thank You.”
>
> ------------------------------------------------------------------------
> “This e-mail message may contain confidential, commercial or
> privileged information that constitutes proprietary information of
> Comverse Inc. or its subsidiaries. If you are not the intended
> recipient of this message, you are hereby notified that any review,
> use or distribution of this information is absolutely prohibited and
> we request that you delete all copies and contact us by e-mailing to:
> security_at_comverse.com. Thank You.”