While implementing StartTLS with Grizzly 2.0 M3, I seem to have uncovered a
potential bug with the handshaking code:
I started a TCPNIOTransport using the DefaultFilterChain with the
TransportFilter and our own LDAPFilter. After successful negotiation of the
StartTLS operation, I insert the SSLFilter between the TransportFilter and
the LDAPFilter and initiated the handshake (this is the client side).
However, the handshake gets stuck at the NEED_WRAP state because the
underlying TCPNIOStreamWriter's buffer is too small for the SSLEngine.
The default buffer size for the TCPNIOStreamWriter is 4k but the SSLEngine
requires a 16k destination buffer when calling SSLEngine.wrap. The call to
SSLStreamWriter.checkBuffers sets the new buffer size and then flushes the
smaller buffer. However, since the buffer was empty, the flush didn't
actually do anything. When SSLEngine.wrap() is called, its given a 4k
destination buffer instead of the 16k and the wrap produces no network data
and stays in the NEED_WRAP state. This whole process repeats in a loop.
Stack trace of the failed buffer resize attempt:
AbstractStreamWriter.overflow(..) : 124
AbstractStreamWriter.flush(..) : 153
AbstractStreamWriter.flush() : 145
SSLStreamWriter.checkBuffers() : 106
SSLStreamWriter.flush0(..) : 124
SSLStreamWriter.handshakeWrap(..) : 88
BlockingSSLHandshaker.handshake(..) : 128
Should I open an issue?
Thanks
Bo