Sure. We have (or will have) a server sitting on a port on the Internet.
Several (could be hundreds) clients make long lived SSL connections (the
server sends various kinds of commands to them). The server sits behind a
load balancer so that we have many instances of the server. The load
balancer uses TCP connection (an open followed by a close) as an "are you
alive" check (what I've been calling a health check). We noticed that when
under the balancer that the SSL connections were getting dropped (the
clients automatically reconnect) far more quickly than should be (5 - 15
minutes for connections that should last hours). This behavior does not
occur when TCP health check is not used (either another kind of health check
or not under the balancer). When we began to investigate we noticed that the
Grizzly logging in the SSL handshake was complaining (in the form of a stack
trace) that it was getting connections that were not SSL (the health check).
I do not know if this stack trace has any relationship to the dropped
connections, but it is the only evidence I have to go on. As I noted
earlier, the real concern here is that a malicious program could send TCP
connections to our server and severely disrupt it.
Thanks again.
Brandon
Oleksiy Stashok wrote:
>
> Hmm, probably I missed something in your usecase :)
>
>> I'm not understanding you're first statement. Do you think the TCP
>> connection will interfere with existing SSL connections? I've been
>> looking
>> though the Grizzly code and couldn't see how this was possible.
> I was just thinking, that you will have single connection, where you'd
> like to send SSL packets together with raw TCP "health" packets.
> Isn't it the case?
>
> If not - please provide more info how it should work.
>
> Thank you.
>
> WBR,
> Alexey.
>
>>
>> Thanks.
>> Brandon
>>
>>
>>
>> Oleksiy Stashok wrote:
>>>
>>> Hi Brandon,
>>>
>>>
>>>> Thanks for the reply. You are correct the health check is a regular
>>>> TCP
>>>> connection and should not go to SSLReadFilter. I thought to write a
>>>> simple
>>>> example that showed the issue. But after a day, my "simple" example
>>>> does not
>>>> show the issue. I've attached the relevant bits from our server
>>>> (hope you
>>>> can wade through it).
>>> Strange, I'd say once you send any "health" packet - SSL connection
>>> will fail. May be in "simple" example you don't send "health"
>>> packets?
>>> IMO we have 2 possibilities here to solve the issue:
>>> 1) Add additional filters to filter-chain before SSLReadFilter, to
>>> handle "health packets".
>>> Doable, but not the easiest solution.
>>>
>>> 2) Send "health" packets via SSL.
>>> More consistent way, easier to implement, but more expensive from
>>> perf. side.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>>
>>>> Regards
>>>> Brandon
>>>>
>>>> http://www.nabble.com/file/p26033264/
>>>> brivo_grizzly_initialization.zip
>>>> brivo_grizzly_initialization.zip
>>>>
>>>>
>>>>
>>>> Oleksiy Stashok wrote:
>>>>>
>>>>> Hi Brandon,
>>>>>
>>>>>> I'm working on a Grizzly server (using 1.9.18). I have a number of
>>>>>> long
>>>>>> lived SSL connected clients. I'm trying to run this configuration
>>>>>> under a
>>>>>> load balancer. The balancer uses a health check that consists of a
>>>>>> TCP
>>>>>> open-close cycle. It seems that after a number (exact count
>>>>>> undetermined) of
>>>>>> these health check cycles one (or more - undetermined) of the SSL
>>>>>> connected
>>>>>> clients gets its connection closed. The health check hits the same
>>>>>> port as
>>>>>> the SSL client. I'm using the SSLReadFilter and the health check
>>>>>> causes the
>>>>>> filter to start an SSL handshake which fails (and a stack trace is
>>>>>> thrown -
>>>>>> see below).
>>>>>>
>>>>>> Any pointers to how to diagnose this issue would be greatly
>>>>>> appreciated. In
>>>>>> particular, is there a way to put a filter in front of the
>>>>>> SSLReadFilter
>>>>>> that would catch the health check and throw it away?
>>>>> Just to understand the usecase better... Health check packets are
>>>>> not
>>>>> secured, right, so they shouldn't come to SSLReadFilter?
>>>>> Can you pls. share the Grizzly code, how you initialize the server?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Brandon
>>>>>>
>>>>>> Exception thrown when TCP open-close cycle hits the SSLReadFilter:
>>>>>>
>>>>>> javax.net.ssl.SSLException: Inbound closed before receiving peer's
>>>>>> close_notify: possible truncation attack?
>>>>>> at
>>>>>> com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:
>>>>>> 166)
>>>>>> at
>>>>>> com
>>>>>> .sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:
>>>>>> 1356)
>>>>>> at
>>>>>> com
>>>>>> .sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:
>>>>>> 1324)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .net
>>>>>> .ssl.internal.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:
>>>>>> 1263)
>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:426)
>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:390)
>>>>>> at
>>>>>> com
>>>>>> .sun.grizzly.filter.SSLReadFilter.doHandshake(SSLReadFilter.java:
>>>>>> 242)
>>>>>> at
>>>>>> com.sun.grizzly.filter.SSLReadFilter.execute(SSLReadFilter.java:
>>>>>> 173)
>>>>>> at
>>>>>> com
>>>>>> .brivo
>>>>>> .commandserv
>>>>>> .communication
>>>>>> .EnhancedSSLReadFilter.execute(EnhancedSSLReadFilter.java:40)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .DefaultProtocolChain
>>>>>> .executeProtocolFilter(DefaultProtocolChain.java:135)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 102)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 88)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:
>>>>>> 57)
>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>> at
>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>> $BasicWorker.dowork(FixedThreadPool.java:379)
>>>>>> at
>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>> $BasicWorker.run(FixedThreadPool.java:360)
>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>> Oct 22, 2009 1:52:43 PM com.sun.grizzly.filter.SSLReadFilter log
>>>>>> FINE: doHandshake
>>>>>> java.io.EOFException: Connection closed
>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:433)
>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:390)
>>>>>> at
>>>>>> com
>>>>>> .sun.grizzly.filter.SSLReadFilter.doHandshake(SSLReadFilter.java:
>>>>>> 242)
>>>>>> at
>>>>>> com.sun.grizzly.filter.SSLReadFilter.execute(SSLReadFilter.java:
>>>>>> 173)
>>>>>> at
>>>>>> com
>>>>>> .brivo
>>>>>> .commandserv
>>>>>> .communication
>>>>>> .EnhancedSSLReadFilter.execute(EnhancedSSLReadFilter.java:40)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .DefaultProtocolChain
>>>>>> .executeProtocolFilter(DefaultProtocolChain.java:135)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 102)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>> 88)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:
>>>>>> 57)
>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>> at
>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>> $BasicWorker.dowork(FixedThreadPool.java:379)
>>>>>> at
>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>> $BasicWorker.run(FixedThreadPool.java:360)
>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26016355.html
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26033264.html
>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26061227.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
--
View this message in context: http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26061792.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.