Date: Wed, 22 Jul 2015 08:47:19 +0000
Hi Alexey,
Your suggestion looks quite interesting and we will try to give it a try and keep you updated
Thanks
Uri
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Tuesday, July 21, 2015 8:09 AM
To: users_at_grizzly.java.net
Subject: Re: High CPU Usage on SSL utils
You could assign dedicated accepting selector thread, it may help accept clients even if other I/O threads busy with SSL handshake. You can also try to use worker thread strategy to offload I/O threads, or use same thread strategy, but delegate handshake operation to custom thread pool, so you can always control number of handshaking connections and don't allow them to consume all CPUs.
Thanks.
WBR,
Alexey.
On 20.07.15 09:57, Ben-Yosef Efrat wrote:
The test wasn’t the one who crashed it was our application since we run on a very high load and handshakes every second. I got no cores but the CPU was so high that it caused long response times and new users also couldn’t connect
From: Riyad Kalla [mailto:rkalla_at_gmail.com]
Sent: Thursday, July 16, 2015 9:05 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
The performance gap could be narrowed down simply to handshake (takes quite a bit longer), but Efrat's point about the test crashing after 20mins is something worthy of investigation it seems...
Efrat, did you get an exception or core dump from the VM? Wondering if Grizzly is leaking resources (Exception) or if something more native and horrible is happening (VM core dump).
On Mon, Jul 13, 2015 at 8:36 AM, Ben-Yosef Efrat <Efrat.Ben-Yosef_at_comverse.com<mailto:Efrat.Ben-Yosef_at_comverse.com>> wrote:
Hi Alexey,
Attached is the tester you asked for (change the suffix to zip after downloading).
The results are relatively the same as the previous ones .
BTW – we don’t know what is the key size – the tester is using the grizzly samples jks (I guess you can change it if necessary), reminding you that we see the problem with 2048 size key.
Thanks
Efrat and Tiran
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com<mailto:oleksiy.stashok_at_oracle.com>]
Sent: Friday, July 10, 2015 1:35 AM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi Tiran,
can you pls. create a simple client/server reproducer for the problem, which includes SSL configuration, so I know I have the same scenario as you do?
IMO we have to understand what exactly JSSE doing that takes so long time.
Thank you.
WBR,
Alexey.
On 07.07.15 06:49, Meltser Tiran wrote:
Hi Alexey,
In theory you are right.
Unfortunately we don’t control the IMAP clients and from what we see in the field, they perform short sessions…
Do you know perhaps on any special settings in Grizzly/Java that can optimize the SSL handshake (SSL acceleration)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse – Making Your Network Smarter
T +972-3-7678381<tel:%2B972-3-7678381>
M +972-54-5639381<tel:%2B972-54-5639381>
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Tuesday, July 07, 2015 4:32 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Sorry, I meant IMAP. AFAIU there is keep-alive mechanism IMAP IDLE, which could be used to keep connection open and receive server notification once there is a new email.
I believe the main problem you have is initial SSL handshake, which really could be expensive, and I think you can rid off, or reduce the effect of the expensive SSL handshake by using IMAP keep-alive mechanism.
WBR,
Alexey.
On 06.07.15 14:53, Ben-Yosef Efrat wrote:
Hi Alexey,
We are not using SMTP, we are using IMAP Protocol.
We also noticed that this issue does not occur if we reduce the key size (at first it was 2048, and we reduced it to 512).
Of course we cannot run with 512. Any Ideas?
Thanks
Efrat
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, July 02, 2015 6:44 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi guys,
IMO what you're currently measuring is SSL handshake performance vs. no-handshake in plain connection.
I quickly googled for SMTP protocol and looks like there is possibility to keep connection alive, IMO it's essential, especially for secured use case.
To be more fair, try to run Echo benchmark, but not with 10 messages, but for example 100K or more, it should remove impact of SSL handshake.
Thanks.
WBR,
Alexey.
On 30.06.15 16:05, Ben-Yosef Efrat wrote:
Sorry, couldn’t attach the file. Please see it in ftp://transfer.comverse.com/outgoing/SSLVsPlain/SSLVsPlain.zip
From: Ben-Yosef Efrat [mailto:Efrat.Ben-Yosef_at_comverse.com]
Sent: Tuesday, June 30, 2015 4:41 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: RE: High CPU Usage on SSL utils
Hi Alexey,
We spent that last week on profiling this issue. We did the following:
1. Run our application on plain vs. Ssl when the application basically does nothing (respond with hard coded OK)
2. Run Grizzly SSL/Plain echo example under load
3. run Jprofiler, VisualVm and netbeans profiler
All showed the same:
* CPU increases dramatically – over 60% on some cases
* memory increase by 40% and more on some cases
* org.glassfish.grizzly.ssl.SSlUtils.executeDelegatedTask() is the most problematic methods CPU wise
We cannot run our application this way at all. On ssl everything crashes after 20 min.
Attached are some screenshots and the code for the ssl/Plain echo load test we run.
Please advise
Efrat
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Friday, June 19, 2015 8:18 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi Tiran,
I see your concern, but IMO it's similar to what you found w.r.t. "selector", if you enable "core classes" profiling - you'll see that it's core SSL implementation consumes CPU, not Grizzly itself. Once you have the profiling results with core classes, we can check what exactly consumes that much time.
Thanks.
WBR,
Alexey.
On 18.06.15 20:07, Meltser Tiran wrote:
Hi Alexey (working with Efrat and Uri) ☺,
First we don’t keep the connections open as this is an IMAP server (or proxy).
Second, I agree that SSL channel should have some performance price but what we experience is far beyond a reasonable price, after making another check we see we have response times slower 600 times than plain communication…
Third, we are running on a 16 cores machine in which SSL should hardly be noticed.
Forth, we have just compared the SSL experience to our previous server written in C++ some more than 10 years ago and operated on machines with about ¼ CPU power (or even less), and load results show response time was doubled at max (not 600 times worse).
Can you have a look on the hot method the profiler is showing and see if something can be done there (or maybe give us some guideline)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse – Making Your Network Smarter
T +972-3-7678381<tel:%2B972-3-7678381>
M +972-54-5639381<tel:%2B972-54-5639381>
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, June 18, 2015 8:16 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi Efrat,
do you keep connections alive?
In general SSL is not for free and it's expected to be slower than plain communication.
WBR,
Alexey.
On 18.06.15 18:14, Ben-Yosef Efrat wrote:
Also, we see higher response times then before (around 50 times longer)
From: Ben-Yosef Efrat [mailto:Efrat.Ben-Yosef_at_comverse.com]
Sent: Thursday, June 18, 2015 6:54 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Cc: Broide Uri; Meltser Tiran
Subject: High CPU Usage on SSL utils
Hi Alexey,
Continuing Uri’s query about the CPU, we moved to java8 profiler and as you said, indeed not the select() was the issue.
Now, we added the SSL to the picture (utilizing Grizzly’s SSLfilter) and as you can see in the attached screenshot, the executeDelegatedTask() method consumes a lot of CPU (we sort by self-tile CPU).
Please advise,
Thanks,
Efrat
[cid:image001.png_at_01D0A9F5.776D0720]
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com. Thank You.”
(image/png attachment: image001.png)