Hi Alexy
Your filter is working well. Thanks again.
I feel like I should be cleaning up the health connections. Should I? What's
the best way to do that? I'm currently closing the channel and canceling the
key.
Regards
Brandon
Oleksiy Stashok wrote:
>
>
>> There are two reasons which lead me to believe that the health check
>> was
>> related to the stack trace. The first is that I set up a local test
>> where I
>> run less watching the Grizzly log and run a program which does the TCP
>> open-close on a regular interval (5 seconds). I could see that every
>> time
>> the TCP open-close fired I would get a stack trace. The second
>> reason is
>> that the stack trace occurs on line 426 of SSLUtils (near the top of
>> doHandshake). The doRead method is called and this returns -1. This
>> causes
>> sslEngine.closeInbound() (which appears to cause one of the
>> exceptions) then
>> we hit throw new EOFException("Connection closed"), which is the other
>> exception.
> I see. However this exception is related to current TCP health
> connection, and shouldn't affect any other existing connection.
>
>> What's a good way to get started writing a filter to sit in front of
>> SSLReadFilter?
> The question here is how we can recognize, that this is not SSL, but
> "health" connection?
> Now, IMO, every connection, which doesn't have initialized SSL session
> and returns -1 on first read attempt is "health" connection, right?
>
> So the very first filter should check if processing connection is new
> (doesn't have specific attachment) and read returns -1 - then skip
> further processing.
> Ok, it will work good if it's really health connection, but if not -
> we need to make SSLReadFilter use data, which has been read already on
> first read. To avoid copying data from one buffer to another, we need
> our first filter to read data in the same buffer, which will be used
> by SSLReadFilter. So we will need to borrow some logic from
> SSLReadFilter.
>
> Here I'll try to show the filter implementation how I see it, but you
> can change the implementation for sure (I have created custom
> SSLReadFilter to have access to some protected fields and methods, so
> you can use this class instead regular SSLReadFilter implementation).
>
> Anyway, it's just an idea.
>
> Thanks.
>
> WBR,
> Alexey.
>
>
> import com.sun.grizzly.Context;
> import com.sun.grizzly.ProtocolFilter;
> import com.sun.grizzly.filter.SSLReadFilter;
> import com.sun.grizzly.util.SSLUtils;
> import com.sun.grizzly.util.ThreadAttachment;
> import com.sun.grizzly.util.WorkerThread;
> import java.io.IOException;
> import java.nio.ByteBuffer;
> import java.nio.channels.SelectionKey;
> import java.nio.channels.SocketChannel;
> import javax.net.ssl.SSLEngine;
>
> /**
> *
> */
> public class CustomFilter implements ProtocolFilter {
> private final CustomSSLReadFilter sslFilter;
>
> public CustomFilter(CustomSSLReadFilter sslFilter) {
> this.sslFilter = sslFilter;
> }
>
> public boolean execute(Context ctx) throws IOException {
> // Check if there is SSL session open on the connection
> if (!isSSLSessionEstablished(ctx)) {
> final SelectionKey selectionKey = ctx.getSelectionKey();
>
> // Create SSL Engine to be able to initialize SSL buffers
> sslFilter.obtainSSLEngine(selectionKey);
>
> // Initialize SSL buffers to read data directly there
> SSLUtils.allocateThreadBuffers(sslFilter.getInputBBSize());
>
> final WorkerThread workerThread = (WorkerThread)
> Thread.currentThread();
> ByteBuffer inputBB = workerThread.getInputBB();
>
> int count;
> try {
> count = ((SocketChannel)
> selectionKey.channel()).read(inputBB);
> } catch(IOException e) {
> count = -1;
> }
>
> if (count == -1) {
> // This is health connection, stop filter chain
> return false;
> }
>
> // Let SSLReadFilter process handshake
> return true;
> }
>
> // Let SSLReadFilter process handshake
> return true;
> }
>
>
>
> public boolean postExecute(Context ctx) throws IOException {
> return true;
> }
>
> /**
> * Checks if SSL session was established
> *
> * @param ctx
> * @return
> */
> private boolean isSSLSessionEstablished(Context ctx) {
> final WorkerThread workerThread = (WorkerThread)
> Thread.currentThread();
> if (workerThread.getSSLEngine() != null) {
> return true;
> }
>
> final SelectionKey key = ctx.getSelectionKey();
> if (key.attachment() instanceof ThreadAttachment) {
> if (((ThreadAttachment) key.attachment()).getSSLEngine() !
> = null) {
> return true;
> }
> }
>
> return false;
> }
>
> public class CustomSSLReadFilter extends SSLReadFilter {
> public int getInputBBSize() {
> return inputBBSize;
> }
>
> @Override
> public SSLEngine obtainSSLEngine(SelectionKey key) {
> final SSLEngine sslEngine = super.obtainSSLEngine(key);
>
> final WorkerThread workerThread = (WorkerThread)
> Thread.currentThread();
> workerThread.setSSLEngine(sslEngine);
>
> final ThreadAttachment attachment =
> workerThread.updateAttachment(
> ThreadAttachment.Mode.SSL_ENGINE);
> key.attach(attachment);
>
> return sslEngine;
> }
>
>
> }
> }
>
>
>>
>> Thanks again.
>>
>> Regards
>> Brandon
>>
>>
>>
>> Oleksiy Stashok wrote:
>>>
>>>> Sure. We have (or will have) a server sitting on a port on the
>>>> Internet.
>>>> Several (could be hundreds) clients make long lived SSL connections
>>>> (the
>>>> server sends various kinds of commands to them). The server sits
>>>> behind a
>>>> load balancer so that we have many instances of the server. The load
>>>> balancer uses TCP connection (an open followed by a close) as an
>>>> "are you
>>>> alive" check (what I've been calling a health check). We noticed
>>>> that when
>>>> under the balancer that the SSL connections were getting dropped
>>>> (the
>>>> clients automatically reconnect) far more quickly than should be (5
>>>> - 15
>>>> minutes for connections that should last hours). This behavior does
>>>> not
>>>> occur when TCP health check is not used (either another kind of
>>>> health check
>>>> or not under the balancer). When we began to investigate we noticed
>>>> that the
>>>> Grizzly logging in the SSL handshake was complaining (in the form of
>>>> a stack
>>>> trace) that it was getting connections that were not SSL (the health
>>>> check).
>>>> I do not know if this stack trace has any relationship to the
>>>> dropped
>>>> connections, but it is the only evidence I have to go on. As I noted
>>>> earlier, the real concern here is that a malicious program could
>>>> send TCP
>>>> connections to our server and severely disrupt it.
>>> It shouldn't be the case here. Newly accepted TCP connection should
>>> not affect any existing SSL connections.
>>> If your health check is just open/close, and you don't send any
>>> data -
>>> you'd never see any handshake problem, because you actually don't
>>> send
>>> any data.
>>> So, IMO, the stacktrace you saw wasn't related to health check.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>>
>>>> Thanks again.
>>>> Brandon
>>>>
>>>>
>>>>
>>>> Oleksiy Stashok wrote:
>>>>>
>>>>> Hmm, probably I missed something in your usecase :)
>>>>>
>>>>>> I'm not understanding you're first statement. Do you think the TCP
>>>>>> connection will interfere with existing SSL connections? I've been
>>>>>> looking
>>>>>> though the Grizzly code and couldn't see how this was possible.
>>>>> I was just thinking, that you will have single connection, where
>>>>> you'd
>>>>> like to send SSL packets together with raw TCP "health" packets.
>>>>> Isn't it the case?
>>>>>
>>>>> If not - please provide more info how it should work.
>>>>>
>>>>> Thank you.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>> Brandon
>>>>>>
>>>>>>
>>>>>>
>>>>>> Oleksiy Stashok wrote:
>>>>>>>
>>>>>>> Hi Brandon,
>>>>>>>
>>>>>>>
>>>>>>>> Thanks for the reply. You are correct the health check is a
>>>>>>>> regular
>>>>>>>> TCP
>>>>>>>> connection and should not go to SSLReadFilter. I thought to
>>>>>>>> write a
>>>>>>>> simple
>>>>>>>> example that showed the issue. But after a day, my "simple"
>>>>>>>> example
>>>>>>>> does not
>>>>>>>> show the issue. I've attached the relevant bits from our server
>>>>>>>> (hope you
>>>>>>>> can wade through it).
>>>>>>> Strange, I'd say once you send any "health" packet - SSL
>>>>>>> connection
>>>>>>> will fail. May be in "simple" example you don't send "health"
>>>>>>> packets?
>>>>>>> IMO we have 2 possibilities here to solve the issue:
>>>>>>> 1) Add additional filters to filter-chain before SSLReadFilter,
>>>>>>> to
>>>>>>> handle "health packets".
>>>>>>> Doable, but not the easiest solution.
>>>>>>>
>>>>>>> 2) Send "health" packets via SSL.
>>>>>>> More consistent way, easier to implement, but more expensive
>>>>>>> from
>>>>>>> perf. side.
>>>>>>>
>>>>>>> WBR,
>>>>>>> Alexey.
>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Brandon
>>>>>>>>
>>>>>>>> http://www.nabble.com/file/p26033264/
>>>>>>>> brivo_grizzly_initialization.zip
>>>>>>>> brivo_grizzly_initialization.zip
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Oleksiy Stashok wrote:
>>>>>>>>>
>>>>>>>>> Hi Brandon,
>>>>>>>>>
>>>>>>>>>> I'm working on a Grizzly server (using 1.9.18). I have a
>>>>>>>>>> number of
>>>>>>>>>> long
>>>>>>>>>> lived SSL connected clients. I'm trying to run this
>>>>>>>>>> configuration
>>>>>>>>>> under a
>>>>>>>>>> load balancer. The balancer uses a health check that consists
>>>>>>>>>> of a
>>>>>>>>>> TCP
>>>>>>>>>> open-close cycle. It seems that after a number (exact count
>>>>>>>>>> undetermined) of
>>>>>>>>>> these health check cycles one (or more - undetermined) of the
>>>>>>>>>> SSL
>>>>>>>>>> connected
>>>>>>>>>> clients gets its connection closed. The health check hits the
>>>>>>>>>> same
>>>>>>>>>> port as
>>>>>>>>>> the SSL client. I'm using the SSLReadFilter and the health
>>>>>>>>>> check
>>>>>>>>>> causes the
>>>>>>>>>> filter to start an SSL handshake which fails (and a stack
>>>>>>>>>> trace is
>>>>>>>>>> thrown -
>>>>>>>>>> see below).
>>>>>>>>>>
>>>>>>>>>> Any pointers to how to diagnose this issue would be greatly
>>>>>>>>>> appreciated. In
>>>>>>>>>> particular, is there a way to put a filter in front of the
>>>>>>>>>> SSLReadFilter
>>>>>>>>>> that would catch the health check and throw it away?
>>>>>>>>> Just to understand the usecase better... Health check packets
>>>>>>>>> are
>>>>>>>>> not
>>>>>>>>> secured, right, so they shouldn't come to SSLReadFilter?
>>>>>>>>> Can you pls. share the Grizzly code, how you initialize the
>>>>>>>>> server?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> WBR,
>>>>>>>>> Alexey.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Brandon
>>>>>>>>>>
>>>>>>>>>> Exception thrown when TCP open-close cycle hits the
>>>>>>>>>> SSLReadFilter:
>>>>>>>>>>
>>>>>>>>>> javax.net.ssl.SSLException: Inbound closed before receiving
>>>>>>>>>> peer's
>>>>>>>>>> close_notify: possible truncation attack?
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:
>>>>>>>>>> 166)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:
>>>>>>>>>> 1356)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:
>>>>>>>>>> 1324)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .net
>>>>>>>>>> .ssl
>>>>>>>>>> .internal.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:
>>>>>>>>>> 1263)
>>>>>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:
>>>>>>>>>> 426)
>>>>>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:
>>>>>>>>>> 390)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly.filter.SSLReadFilter.doHandshake(SSLReadFilter.java:
>>>>>>>>>> 242)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun.grizzly.filter.SSLReadFilter.execute(SSLReadFilter.java:
>>>>>>>>>> 173)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .brivo
>>>>>>>>>> .commandserv
>>>>>>>>>> .communication
>>>>>>>>>> .EnhancedSSLReadFilter.execute(EnhancedSSLReadFilter.java:40)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain
>>>>>>>>>> .executeProtocolFilter(DefaultProtocolChain.java:135)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>>>>>> 102)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>>>>>> 88)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .ProtocolChainContextTask
>>>>>>>>>> .doCall(ProtocolChainContextTask.java:
>>>>>>>>>> 53)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:
>>>>>>>>>> 57)
>>>>>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>>>>>> at
>>>>>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>>>>>> $BasicWorker.dowork(FixedThreadPool.java:379)
>>>>>>>>>> at
>>>>>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>>>>>> $BasicWorker.run(FixedThreadPool.java:360)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>>>>>> Oct 22, 2009 1:52:43 PM com.sun.grizzly.filter.SSLReadFilter
>>>>>>>>>> log
>>>>>>>>>> FINE: doHandshake
>>>>>>>>>> java.io.EOFException: Connection closed
>>>>>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:
>>>>>>>>>> 433)
>>>>>>>>>> at com.sun.grizzly.util.SSLUtils.doHandshake(SSLUtils.java:
>>>>>>>>>> 390)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly.filter.SSLReadFilter.doHandshake(SSLReadFilter.java:
>>>>>>>>>> 242)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun.grizzly.filter.SSLReadFilter.execute(SSLReadFilter.java:
>>>>>>>>>> 173)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .brivo
>>>>>>>>>> .commandserv
>>>>>>>>>> .communication
>>>>>>>>>> .EnhancedSSLReadFilter.execute(EnhancedSSLReadFilter.java:40)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain
>>>>>>>>>> .executeProtocolFilter(DefaultProtocolChain.java:135)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>>>>>> 102)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .DefaultProtocolChain.execute(DefaultProtocolChain.java:
>>>>>>>>>> 88)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .ProtocolChainContextTask
>>>>>>>>>> .doCall(ProtocolChainContextTask.java:
>>>>>>>>>> 53)
>>>>>>>>>> at
>>>>>>>>>> com
>>>>>>>>>> .sun
>>>>>>>>>> .grizzly
>>>>>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:
>>>>>>>>>> 57)
>>>>>>>>>> at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
>>>>>>>>>> at
>>>>>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>>>>>> $BasicWorker.dowork(FixedThreadPool.java:379)
>>>>>>>>>> at
>>>>>>>>>> com.sun.grizzly.util.FixedThreadPool
>>>>>>>>>> $BasicWorker.run(FixedThreadPool.java:360)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:595)
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26016355.html
>>>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>>>> Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26033264.html
>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26061227.html
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26061792.html
>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26063806.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
--
View this message in context: http://www.nabble.com/SSL-connection-closed-after-repeated-TCP-open-close-cycles-tp26016355p26082788.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.