Hello Alexey,
My apologies for the delay in responding. Regarding ...
1) Thanks for the confirmation. Getting hold of async writer from the
selector handler is the right way to go for me.
2) Just using a SSLWritePreprocessor isn't helping me. Have tested with the
re-handshaking code and it works for me.
Also, now that I am stress testing the server I am running into a few
problems. I would very much appreciate your suggestions/comments on these
issues. The server in question currently is capable of catering to about
4000 clients during peak times and uses a C library for SSL communication
with the clients. We are in the process of moving it to an NIO based
framework with minimal impact on existing design. My implementation of
Grizzly is as below ...
--> Server initalizes the controller as follows ...
SSLReadFilter sslReadFilter = new MySSLReadFilter(gameServer,
USE_GRIZZLY_ASYNC_READS);
sslReadFilter.setSSLContext(context);
final ProtocolChain protocolChain = new DefaultProtocolChain();
protocolChain.addFilter(sslReadFilter);
ProtocolChainInstanceHandler chainInstanceHandler = new
DefaultProtocolChainInstanceHandler() {
public ProtocolChain poll() {
return protocolChain;
}
};
Controller controller = new Controller();
controller.setHandleReadWriteConcurrently(true);
SSLSelectorHandler selectorHandler = new MySSLSelectorHandler();
selectorHandler.setPort(port);
controller.setSelectorHandler(selectorHandler);
controller.setProtocolChainInstanceHandler(chainInstanceHandler);
controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
Currently set to 0
OutputWriter.setDefaultWriteTimeout(5000);
controller.start();
--> MySSLReadFilter.execute() method does ...
super.execute(context);
engine = workerThread.getSSLEngine();
session = engine.getSession();
Object obj = session.getValue(ServerConHandler.class.getName());
ServerConHandler conHandler;
if (obj == null || !(obj instanceof ServerConHandler)) {
SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
context.getSelectorHandler();
conHandler = new ServerConHandler(selectorHandler, selectionKey,
(SocketChannel) selectionKey.channel(), engine, user);
session.putValue(ServerConHandler.class.getName(), conHandler);
} else {
conHandler = (ServerConHandler) obj;
}
conHandler.read(workerThread.getByteBuffer());
--> MySSLSelectorHandler does ...
public void preSelect(Context ctx) throws IOException {
if (asyncQueueWriter == null) {
asyncQueueWriter = new MySSLAsyncQueueWriter(this);
}
super.preSelect(ctx);
}
--> ServerConHandler on receiving a write request does ...
try {
MySSLAsyncQueueWriter writer = (MySSLAsyncQueueWriter)
selectorHandler.getAsyncQueueWriter();
writer.write(selectionKey, ByteBuffer.wrap(b1),
sslEngine);
} catch (ClosedChannelException e) {
logger.error(e.getMessage());
} catch (IOException e) {
logger.error(e.getMessage());
}
--> MySSLAsyncQueueWriter does ...
public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
sslEngine) throws IOException {
ChannelRegistrationRecord record =
channelRegistrationMap.putIfAbsent(key.channel(), new
ChannelRegistrationRecord(sslEngine));
if (record != null) {
record.writeCount.incrementAndGet();
}
super.write(key, buffer);
}
protected void doWrite(WritableByteChannel channel, SocketAddress
dstAddress, ByteBuffer byteBuffer) throws IOException {
SSLEngine sslEngine;
ChannelRegistrationRecord record =
channelRegistrationMap.get(channel);
if (record == null) {
logger.error("Channel is not registered for an async write");
return;
}
sslEngine = record.sslEngine;
if (sslEngine == null) {
logger.error("SSLEngine is null.");
return;
}
if (record.writeCount.intValue() > 200) {
((SocketChannel)channel).socket().close();
channel.close();
channelRegistrationMap.remove(channel);
return;
}
if (record.writeCount.decrementAndGet() == 0) {
channelRegistrationMap.remove(channel);
}
SSLOutputWriter.flushChannel((SelectableChannel) channel,
byteBuffer, ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
sslEngine.getSession().getPacketBufferSize())), sslEngine);
}
The issues we are observing are ...
--> As the number of concurrent connections shoots up to a number above 200
the stress clients start facing disconnections and following exceptions are
seen on the server side.
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:105)
at
com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:73)
at
com.sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:100)
at
com.my.server.grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:70)
at
com.sun.grizzly.async.AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:374)
at
com.sun.grizzly.async.AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:291)
at
com.sun.grizzly.async.AsyncQueueWriterContextTask.doCall(AsyncQueueWriterContextTask.java:86)
at
com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
at
com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:169)
--> The CPU load on the 4 CPU machine shoots up to a 100% consistently.
--> The memory graph shows no decline.
I think it must be something that I am doing wrong. Also, I wanted to know
if Grizzly has been tested to scale up to 4000 ssl connections and beyond if
required.
Please feel free to revert in case you need some more information.
Thanks & Regards,
Phani
Oleksiy Stashok wrote:
>
>>
>
>
>> 1) I think I should have given you a bit more info on this, this is
>> what
>> happens on the server ...
>>
>> --> Grizzly's SSLReadFilter provides my server Handler with a
>> ByteBuffer
>> --> Server handler parses the ByteBuffer and hands it over to a
>> business
>> processing tier (using custom server threads).
>> --> Meanwhile, Grizzly context is recycled
>> --> A thread from the Business processing tier (BPT) digests the
>> message
>> asynchronously and produces a response
>> --> The BPT turns to Grizzly for handing over the response
>> bytebuffer in an
>> async write.
>> --> The processes of parsing messages on read byte buffers, processing
>> parsed messages and writing them back to the network framework are
>> independent of each other. Eg. the server could be parsing a message
>> M2 in
>> one BPT while writing M1 in another BPT.
>>
>> Rephrasing the question, can I use the solution mentioned in
>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>> while preserving my server's luxury of writing back to grizzly using a
>> context.getAsyncQueueWriter.write(byteBuffer) at its own time without
>> suspending the reads and other business processing?
> Absolutely. Then, in your case I'm not sure you need to use
> SuspendFilter, as basically you don't want to suspend the connection -
> but keep it process other requests.
> Once your custom thread will complete the task execution - context
> will not be accessible anymore, as it was recycled by Grizzly worker
> thread, so you'll need to access async queue writer different way like:
> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
> byteBuffer);
>
>
>> 2) Yes, I did mean the re-handshaking. It is done for every chunk of
>> the
>> bytearray written synchronously using SSLOutputWriter.flushChannel()
>> but,
>> the same is not done if I try to write asynchronously using an
>> AsyncQueueWriter.write() with SSL as the final write is handled by
>> TCPAsyncQueueWriter which passes the bytebuffer through the
>> WritePreprocessor and tries writing the resultant bytebuffer contents
>> directly to the channel without any checks for re-handshaking. I
>> wanted to
>> know if it is left to the user to implement it in his/her own code
>> for now?
> you mean this part of SSLOutputWriter.flushChannel()?
> //--------------- Cut --------------------------
> case OK:
> if (result.getHandshakeStatus() ==
> HandshakeStatus.NEED_TASK) {
> SSLUtils.executeDelegatedTask(sslEngine);
> }
> break;
> //----------------------------------------------
> If yes, can I ask you to add the same lines to your SSL preprocessor
> to check whether it works for you - if yes - we'll fix that in Grizzly
> examples.
>
> Thanks.
>
> WBR,
> Alexey.
>
>>
>>
>> Thanks & Regards,
>> Phani
>>
>>
>> Oleksiy Stashok wrote:
>>>
>>> Hello Phani,
>>>
>>>> I am working on using Grizzly for our server application which uses
>>>> SSLv3
>>>> and a custom binary application level protocol for client-server
>>>> talk. My
>>>> server side implementation of Grizzly and the problems I am facing
>>>> are
>>>> similar to the ones in the thread below ...
>>>>
>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>
>>>> However I have some doubts before I implement the solution
>>>> recommended in
>>>> the thread ...
>>>>
>>>> I am going to implement both Async reads and writes on the server
>>>> side and
>>>> the problem with adopting the solution for this is that once I
>>>> suspend
>>>> execution of a ProtocolChain using the SuspendableFilter a call to
>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>> correct me
>>>> if I am wrong but, as a result of this reads won't happen on the
>>>> channel. Is
>>>> there a way out for this? I do not want reads to be waiting on
>>>> writes
>>>> caused by the previous read. I want reads and writes to be
>>>> independent of
>>>> each other.
>>> You're right. In your case you don't need to have
>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just let
>>> channel to be re-registered for OP_READ.
>>>
>>>
>>>> Also, I see that in case of Async writes for SSL / TLS the final
>>>> channel
>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>> channel.write(byteBuffer) call. I am supplying a write pre processor
>>>> to wrap
>>>> my bytes before sending them out but, the part to redo the broken
>>>> handshakes
>>>> before sending every chunk of bytes out to the channel as in
>>>> SSLOutputWriter.flushChannel() will be missing in
>>>> TCPAsyncQueueWriter.
>>>> Please advice on how to work around this problem without drifting
>>>> away from
>>>> the framework code and classes. For now, I am thinking of
>>>> implementing a new
>>>> SSLAsyncQueueWriter.
>>> Can you pls. provide details what part, you think, is missed in
>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>> If it's some common problem, we have to implement that in default
>>> Grizzly implementation and your help will be very appreciated!
>>>
>>> Thank you.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>>
>>>>
>>>> Any help would be very much appreciated.
>>>>
>>>> Thanks & Regards,
>>>> Phani
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
--
View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.