Date: Tue, 7 Jul 2015 13:49:47 +0000
Hi Alexey,
In theory you are right.
Unfortunately we don’t control the IMAP clients and from what we see in the field, they perform short sessions…
Do you know perhaps on any special settings in Grizzly/Java that can optimize the SSL handshake (SSL acceleration)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse – Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Tuesday, July 07, 2015 4:32 PM
To: users_at_grizzly.java.net
Subject: Re: High CPU Usage on SSL utils
Sorry, I meant IMAP. AFAIU there is keep-alive mechanism IMAP IDLE, which could be used to keep connection open and receive server notification once there is a new email.
I believe the main problem you have is initial SSL handshake, which really could be expensive, and I think you can rid off, or reduce the effect of the expensive SSL handshake by using IMAP keep-alive mechanism.
WBR,
Alexey.
On 06.07.15 14:53, Ben-Yosef Efrat wrote:
Hi Alexey,
We are not using SMTP, we are using IMAP Protocol.
We also noticed that this issue does not occur if we reduce the key size (at first it was 2048, and we reduced it to 512).
Of course we cannot run with 512. Any Ideas?
Thanks
Efrat
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, July 02, 2015 6:44 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi guys,
IMO what you're currently measuring is SSL handshake performance vs. no-handshake in plain connection.
I quickly googled for SMTP protocol and looks like there is possibility to keep connection alive, IMO it's essential, especially for secured use case.
To be more fair, try to run Echo benchmark, but not with 10 messages, but for example 100K or more, it should remove impact of SSL handshake.
Thanks.
WBR,
Alexey.
On 30.06.15 16:05, Ben-Yosef Efrat wrote:
Sorry, couldn’t attach the file. Please see it in ftp://transfer.comverse.com/outgoing/SSLVsPlain/SSLVsPlain.zip
From: Ben-Yosef Efrat [mailto:Efrat.Ben-Yosef_at_comverse.com]
Sent: Tuesday, June 30, 2015 4:41 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: RE: High CPU Usage on SSL utils
Hi Alexey,
We spent that last week on profiling this issue. We did the following:
1. Run our application on plain vs. Ssl when the application basically does nothing (respond with hard coded OK)
2. Run Grizzly SSL/Plain echo example under load
3. run Jprofiler, VisualVm and netbeans profiler
All showed the same:
* CPU increases dramatically – over 60% on some cases
* memory increase by 40% and more on some cases
* org.glassfish.grizzly.ssl.SSlUtils.executeDelegatedTask() is the most problematic methods CPU wise
We cannot run our application this way at all. On ssl everything crashes after 20 min.
Attached are some screenshots and the code for the ssl/Plain echo load test we run.
Please advise
Efrat
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Friday, June 19, 2015 8:18 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi Tiran,
I see your concern, but IMO it's similar to what you found w.r.t. "selector", if you enable "core classes" profiling - you'll see that it's core SSL implementation consumes CPU, not Grizzly itself. Once you have the profiling results with core classes, we can check what exactly consumes that much time.
Thanks.
WBR,
Alexey.
On 18.06.15 20:07, Meltser Tiran wrote:
Hi Alexey (working with Efrat and Uri) :),
First we don’t keep the connections open as this is an IMAP server (or proxy).
Second, I agree that SSL channel should have some performance price but what we experience is far beyond a reasonable price, after making another check we see we have response times slower 600 times than plain communication…
Third, we are running on a 16 cores machine in which SSL should hardly be noticed.
Forth, we have just compared the SSL experience to our previous server written in C++ some more than 10 years ago and operated on machines with about ¼ CPU power (or even less), and load results show response time was doubled at max (not 600 times worse).
Can you have a look on the hot method the profiler is showing and see if something can be done there (or maybe give us some guideline)?
Tiran Meltser
System Architect
Global Products & Operations
Comverse – Making Your Network Smarter
T +972-3-7678381
M +972-54-5639381
Tiran.Meltser_at_comverse.com <mailto:Tiran.Meltser_at_comverse.com>
www.comverse.com<http://www.comverse.com/>
P Please think of the environment before printing this email
From: Oleksiy Stashok [mailto:oleksiy.stashok_at_oracle.com]
Sent: Thursday, June 18, 2015 8:16 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Subject: Re: High CPU Usage on SSL utils
Hi Efrat,
do you keep connections alive?
In general SSL is not for free and it's expected to be slower than plain communication.
WBR,
Alexey.
On 18.06.15 18:14, Ben-Yosef Efrat wrote:
Also, we see higher response times then before (around 50 times longer)
From: Ben-Yosef Efrat [mailto:Efrat.Ben-Yosef_at_comverse.com]
Sent: Thursday, June 18, 2015 6:54 PM
To: users_at_grizzly.java.net<mailto:users_at_grizzly.java.net>
Cc: Broide Uri; Meltser Tiran
Subject: High CPU Usage on SSL utils
Hi Alexey,
Continuing Uri’s query about the CPU, we moved to java8 profiler and as you said, indeed not the select() was the issue.
Now, we added the SSL to the picture (utilizing Grizzly’s SSLfilter) and as you can see in the attached screenshot, the executeDelegatedTask() method consumes a lot of CPU (we sort by self-tile CPU).
Please advise,
Thanks,
Efrat
[cid:image001.png_at_01D0A9F5.776D0720]
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com<mailto:security_at_comverse.com>. Thank You.”
________________________________
“This e-mail message may contain confidential, commercial or privileged information that constitutes proprietary information of Comverse Inc. or its subsidiaries. If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies and contact us by e-mailing to: security_at_comverse.com. Thank You.”
(image/png attachment: image001.png)