Hello Phani,
can you pls. send the complete java sources of Grizzly related
classes, it could be easier to read the code.
Thanks.
WBR,
Alexey.
On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>
> Hello Alexey,
>
> My apologies for the delay in responding. Regarding ...
>
> 1) Thanks for the confirmation. Getting hold of async writer from the
> selector handler is the right way to go for me.
> 2) Just using a SSLWritePreprocessor isn't helping me. Have tested
> with the
> re-handshaking code and it works for me.
>
> Also, now that I am stress testing the server I am running into a few
> problems. I would very much appreciate your suggestions/comments on
> these
> issues. The server in question currently is capable of catering to
> about
> 4000 clients during peak times and uses a C library for SSL
> communication
> with the clients. We are in the process of moving it to an NIO based
> framework with minimal impact on existing design. My implementation of
> Grizzly is as below ...
>
> --> Server initalizes the controller as follows ...
>
> SSLReadFilter sslReadFilter = new
> MySSLReadFilter(gameServer,
> USE_GRIZZLY_ASYNC_READS);
> sslReadFilter.setSSLContext(context);
> final ProtocolChain protocolChain = new
> DefaultProtocolChain();
> protocolChain.addFilter(sslReadFilter);
> ProtocolChainInstanceHandler chainInstanceHandler = new
> DefaultProtocolChainInstanceHandler() {
> public ProtocolChain poll() {
> return protocolChain;
> }
> };
> Controller controller = new Controller();
> controller.setHandleReadWriteConcurrently(true);
> SSLSelectorHandler selectorHandler = new
> MySSLSelectorHandler();
> selectorHandler.setPort(port);
> controller.setSelectorHandler(selectorHandler);
>
> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>
> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
> Currently set to 0
> OutputWriter.setDefaultWriteTimeout(5000);
> controller.start();
>
> --> MySSLReadFilter.execute() method does ...
>
> super.execute(context);
> engine = workerThread.getSSLEngine();
> session = engine.getSession();
> Object obj =
> session.getValue(ServerConHandler.class.getName());
> ServerConHandler conHandler;
> if (obj == null || !(obj instanceof ServerConHandler)) {
> SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
> context.getSelectorHandler();
> conHandler = new ServerConHandler(selectorHandler,
> selectionKey,
> (SocketChannel) selectionKey.channel(), engine, user);
> session.putValue(ServerConHandler.class.getName(),
> conHandler);
> } else {
> conHandler = (ServerConHandler) obj;
> }
> conHandler.read(workerThread.getByteBuffer());
>
> --> MySSLSelectorHandler does ...
>
> public void preSelect(Context ctx) throws IOException {
> if (asyncQueueWriter == null) {
> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
> }
> super.preSelect(ctx);
> }
>
> --> ServerConHandler on receiving a write request does ...
>
> try {
> MySSLAsyncQueueWriter writer =
> (MySSLAsyncQueueWriter)
> selectorHandler.getAsyncQueueWriter();
> writer.write(selectionKey, ByteBuffer.wrap(b1),
> sslEngine);
> } catch (ClosedChannelException e) {
> logger.error(e.getMessage());
> } catch (IOException e) {
> logger.error(e.getMessage());
> }
>
> --> MySSLAsyncQueueWriter does ...
>
> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
> sslEngine) throws IOException {
> ChannelRegistrationRecord record =
> channelRegistrationMap.putIfAbsent(key.channel(), new
> ChannelRegistrationRecord(sslEngine));
> if (record != null) {
> record.writeCount.incrementAndGet();
> }
> super.write(key, buffer);
> }
>
> protected void doWrite(WritableByteChannel channel, SocketAddress
> dstAddress, ByteBuffer byteBuffer) throws IOException {
> SSLEngine sslEngine;
> ChannelRegistrationRecord record =
> channelRegistrationMap.get(channel);
> if (record == null) {
> logger.error("Channel is not registered for an async
> write");
> return;
> }
> sslEngine = record.sslEngine;
> if (sslEngine == null) {
> logger.error("SSLEngine is null.");
> return;
> }
> if (record.writeCount.intValue() > 200) {
> ((SocketChannel)channel).socket().close();
> channel.close();
> channelRegistrationMap.remove(channel);
> return;
> }
> if (record.writeCount.decrementAndGet() == 0) {
> channelRegistrationMap.remove(channel);
> }
> SSLOutputWriter.flushChannel((SelectableChannel) channel,
> byteBuffer,
> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
> sslEngine.getSession().getPacketBufferSize())), sslEngine);
> }
>
>
> The issues we are observing are ...
>
> --> As the number of concurrent connections shoots up to a number
> above 200
> the stress clients start facing disconnections and following
> exceptions are
> seen on the server side.
>
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
> 125)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
> 294)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:105)
> at
> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:73)
> at
> com
> .sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
> 100)
> at
> com
> .my
> .server
> .grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:70)
> at
> com
> .sun
> .grizzly
> .async
> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:374)
> at
> com
> .sun
> .grizzly
> .async
> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:291)
> at
> com
> .sun
> .grizzly
> .async
> .AsyncQueueWriterContextTask.doCall(AsyncQueueWriterContextTask.java:
> 86)
> at
> com
> .sun
> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
> at
> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:169)
>
> --> The CPU load on the 4 CPU machine shoots up to a 100%
> consistently.
> --> The memory graph shows no decline.
>
> I think it must be something that I am doing wrong. Also, I wanted
> to know
> if Grizzly has been tested to scale up to 4000 ssl connections and
> beyond if
> required.
>
> Please feel free to revert in case you need some more information.
>
> Thanks & Regards,
> Phani
>
>
>
> Oleksiy Stashok wrote:
>>
>>>
>>
>>
>>> 1) I think I should have given you a bit more info on this, this is
>>> what
>>> happens on the server ...
>>>
>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>> ByteBuffer
>>> --> Server handler parses the ByteBuffer and hands it over to a
>>> business
>>> processing tier (using custom server threads).
>>> --> Meanwhile, Grizzly context is recycled
>>> --> A thread from the Business processing tier (BPT) digests the
>>> message
>>> asynchronously and produces a response
>>> --> The BPT turns to Grizzly for handing over the response
>>> bytebuffer in an
>>> async write.
>>> --> The processes of parsing messages on read byte buffers,
>>> processing
>>> parsed messages and writing them back to the network framework are
>>> independent of each other. Eg. the server could be parsing a message
>>> M2 in
>>> one BPT while writing M1 in another BPT.
>>>
>>> Rephrasing the question, can I use the solution mentioned in
>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>> while preserving my server's luxury of writing back to grizzly
>>> using a
>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>> without
>>> suspending the reads and other business processing?
>> Absolutely. Then, in your case I'm not sure you need to use
>> SuspendFilter, as basically you don't want to suspend the
>> connection -
>> but keep it process other requests.
>> Once your custom thread will complete the task execution - context
>> will not be accessible anymore, as it was recycled by Grizzly worker
>> thread, so you'll need to access async queue writer different way
>> like:
>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>> byteBuffer);
>>
>>
>>> 2) Yes, I did mean the re-handshaking. It is done for every chunk of
>>> the
>>> bytearray written synchronously using SSLOutputWriter.flushChannel()
>>> but,
>>> the same is not done if I try to write asynchronously using an
>>> AsyncQueueWriter.write() with SSL as the final write is handled by
>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>> WritePreprocessor and tries writing the resultant bytebuffer
>>> contents
>>> directly to the channel without any checks for re-handshaking. I
>>> wanted to
>>> know if it is left to the user to implement it in his/her own code
>>> for now?
>> you mean this part of SSLOutputWriter.flushChannel()?
>> //--------------- Cut --------------------------
>> case OK:
>> if (result.getHandshakeStatus() ==
>> HandshakeStatus.NEED_TASK) {
>> SSLUtils.executeDelegatedTask(sslEngine);
>> }
>> break;
>> //----------------------------------------------
>> If yes, can I ask you to add the same lines to your SSL preprocessor
>> to check whether it works for you - if yes - we'll fix that in
>> Grizzly
>> examples.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>>
>>>
>>> Thanks & Regards,
>>> Phani
>>>
>>>
>>> Oleksiy Stashok wrote:
>>>>
>>>> Hello Phani,
>>>>
>>>>> I am working on using Grizzly for our server application which
>>>>> uses
>>>>> SSLv3
>>>>> and a custom binary application level protocol for client-server
>>>>> talk. My
>>>>> server side implementation of Grizzly and the problems I am facing
>>>>> are
>>>>> similar to the ones in the thread below ...
>>>>>
>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>
>>>>> However I have some doubts before I implement the solution
>>>>> recommended in
>>>>> the thread ...
>>>>>
>>>>> I am going to implement both Async reads and writes on the server
>>>>> side and
>>>>> the problem with adopting the solution for this is that once I
>>>>> suspend
>>>>> execution of a ProtocolChain using the SuspendableFilter a call to
>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>>> correct me
>>>>> if I am wrong but, as a result of this reads won't happen on the
>>>>> channel. Is
>>>>> there a way out for this? I do not want reads to be waiting on
>>>>> writes
>>>>> caused by the previous read. I want reads and writes to be
>>>>> independent of
>>>>> each other.
>>>> You're right. In your case you don't need to have
>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just
>>>> let
>>>> channel to be re-registered for OP_READ.
>>>>
>>>>
>>>>> Also, I see that in case of Async writes for SSL / TLS the final
>>>>> channel
>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>> processor
>>>>> to wrap
>>>>> my bytes before sending them out but, the part to redo the broken
>>>>> handshakes
>>>>> before sending every chunk of bytes out to the channel as in
>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>> TCPAsyncQueueWriter.
>>>>> Please advice on how to work around this problem without drifting
>>>>> away from
>>>>> the framework code and classes. For now, I am thinking of
>>>>> implementing a new
>>>>> SSLAsyncQueueWriter.
>>>> Can you pls. provide details what part, you think, is missed in
>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>> If it's some common problem, we have to implement that in default
>>>> Grizzly implementation and your help will be very appreciated!
>>>>
>>>> Thank you.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>>
>>>>>
>>>>> Any help would be very much appreciated.
>>>>>
>>>>> Thanks & Regards,
>>>>> Phani
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>