Hi Alexey,
Looking back at MySSLReadFilter I think I have found the problem - it is not
exactly thread safe.
I will fix it and get back to you in case I have some real issues. Sorry
about the false alarm.
Thanks!
Phani
PhaKuDi wrote:
>
> Hi Alexey,
>
> Please find the source files in the zip file attached.
>
> http://www.nabble.com/file/p18503031/src.zip src.zip
>
> Regards,
> Phani
>
>
> Oleksiy Stashok wrote:
>>
>> Hello Phani,
>>
>> can you pls. send the complete java sources of Grizzly related
>> classes, it could be easier to read the code.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>> On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>>
>>>
>>> Hello Alexey,
>>>
>>> My apologies for the delay in responding. Regarding ...
>>>
>>> 1) Thanks for the confirmation. Getting hold of async writer from the
>>> selector handler is the right way to go for me.
>>> 2) Just using a SSLWritePreprocessor isn't helping me. Have tested
>>> with the
>>> re-handshaking code and it works for me.
>>>
>>> Also, now that I am stress testing the server I am running into a few
>>> problems. I would very much appreciate your suggestions/comments on
>>> these
>>> issues. The server in question currently is capable of catering to
>>> about
>>> 4000 clients during peak times and uses a C library for SSL
>>> communication
>>> with the clients. We are in the process of moving it to an NIO based
>>> framework with minimal impact on existing design. My implementation of
>>> Grizzly is as below ...
>>>
>>> --> Server initalizes the controller as follows ...
>>>
>>> SSLReadFilter sslReadFilter = new
>>> MySSLReadFilter(gameServer,
>>> USE_GRIZZLY_ASYNC_READS);
>>> sslReadFilter.setSSLContext(context);
>>> final ProtocolChain protocolChain = new
>>> DefaultProtocolChain();
>>> protocolChain.addFilter(sslReadFilter);
>>> ProtocolChainInstanceHandler chainInstanceHandler = new
>>> DefaultProtocolChainInstanceHandler() {
>>> public ProtocolChain poll() {
>>> return protocolChain;
>>> }
>>> };
>>> Controller controller = new Controller();
>>> controller.setHandleReadWriteConcurrently(true);
>>> SSLSelectorHandler selectorHandler = new
>>> MySSLSelectorHandler();
>>> selectorHandler.setPort(port);
>>> controller.setSelectorHandler(selectorHandler);
>>>
>>> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>>>
>>> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
>>> Currently set to 0
>>> OutputWriter.setDefaultWriteTimeout(5000);
>>> controller.start();
>>>
>>> --> MySSLReadFilter.execute() method does ...
>>>
>>> super.execute(context);
>>> engine = workerThread.getSSLEngine();
>>> session = engine.getSession();
>>> Object obj =
>>> session.getValue(ServerConHandler.class.getName());
>>> ServerConHandler conHandler;
>>> if (obj == null || !(obj instanceof ServerConHandler)) {
>>> SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
>>> context.getSelectorHandler();
>>> conHandler = new ServerConHandler(selectorHandler,
>>> selectionKey,
>>> (SocketChannel) selectionKey.channel(), engine, user);
>>> session.putValue(ServerConHandler.class.getName(),
>>> conHandler);
>>> } else {
>>> conHandler = (ServerConHandler) obj;
>>> }
>>> conHandler.read(workerThread.getByteBuffer());
>>>
>>> --> MySSLSelectorHandler does ...
>>>
>>> public void preSelect(Context ctx) throws IOException {
>>> if (asyncQueueWriter == null) {
>>> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
>>> }
>>> super.preSelect(ctx);
>>> }
>>>
>>> --> ServerConHandler on receiving a write request does ...
>>>
>>> try {
>>> MySSLAsyncQueueWriter writer =
>>> (MySSLAsyncQueueWriter)
>>> selectorHandler.getAsyncQueueWriter();
>>> writer.write(selectionKey, ByteBuffer.wrap(b1),
>>> sslEngine);
>>> } catch (ClosedChannelException e) {
>>> logger.error(e.getMessage());
>>> } catch (IOException e) {
>>> logger.error(e.getMessage());
>>> }
>>>
>>> --> MySSLAsyncQueueWriter does ...
>>>
>>> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
>>> sslEngine) throws IOException {
>>> ChannelRegistrationRecord record =
>>> channelRegistrationMap.putIfAbsent(key.channel(), new
>>> ChannelRegistrationRecord(sslEngine));
>>> if (record != null) {
>>> record.writeCount.incrementAndGet();
>>> }
>>> super.write(key, buffer);
>>> }
>>>
>>> protected void doWrite(WritableByteChannel channel, SocketAddress
>>> dstAddress, ByteBuffer byteBuffer) throws IOException {
>>> SSLEngine sslEngine;
>>> ChannelRegistrationRecord record =
>>> channelRegistrationMap.get(channel);
>>> if (record == null) {
>>> logger.error("Channel is not registered for an async
>>> write");
>>> return;
>>> }
>>> sslEngine = record.sslEngine;
>>> if (sslEngine == null) {
>>> logger.error("SSLEngine is null.");
>>> return;
>>> }
>>> if (record.writeCount.intValue() > 200) {
>>> ((SocketChannel)channel).socket().close();
>>> channel.close();
>>> channelRegistrationMap.remove(channel);
>>> return;
>>> }
>>> if (record.writeCount.decrementAndGet() == 0) {
>>> channelRegistrationMap.remove(channel);
>>> }
>>> SSLOutputWriter.flushChannel((SelectableChannel) channel,
>>> byteBuffer,
>>> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
>>> sslEngine.getSession().getPacketBufferSize())), sslEngine);
>>> }
>>>
>>>
>>> The issues we are observing are ...
>>>
>>> --> As the number of concurrent connections shoots up to a number
>>> above 200
>>> the stress clients start facing disconnections and following
>>> exceptions are
>>> seen on the server side.
>>>
>>> java.nio.channels.ClosedChannelException
>>> at
>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
>>> 125)
>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
>>> 294)
>>> at
>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:105)
>>> at
>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:73)
>>> at
>>> com
>>> .sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
>>> 100)
>>> at
>>> com
>>> .my
>>> .server
>>> .grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:70)
>>> at
>>> com
>>> .sun
>>> .grizzly
>>> .async
>>> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:374)
>>> at
>>> com
>>> .sun
>>> .grizzly
>>> .async
>>> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:291)
>>> at
>>> com
>>> .sun
>>> .grizzly
>>> .async
>>> .AsyncQueueWriterContextTask.doCall(AsyncQueueWriterContextTask.java:
>>> 86)
>>> at
>>> com
>>> .sun
>>> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
>>> at
>>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:169)
>>>
>>> --> The CPU load on the 4 CPU machine shoots up to a 100%
>>> consistently.
>>> --> The memory graph shows no decline.
>>>
>>> I think it must be something that I am doing wrong. Also, I wanted
>>> to know
>>> if Grizzly has been tested to scale up to 4000 ssl connections and
>>> beyond if
>>> required.
>>>
>>> Please feel free to revert in case you need some more information.
>>>
>>> Thanks & Regards,
>>> Phani
>>>
>>>
>>>
>>> Oleksiy Stashok wrote:
>>>>
>>>>>
>>>>
>>>>
>>>>> 1) I think I should have given you a bit more info on this, this is
>>>>> what
>>>>> happens on the server ...
>>>>>
>>>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>>>> ByteBuffer
>>>>> --> Server handler parses the ByteBuffer and hands it over to a
>>>>> business
>>>>> processing tier (using custom server threads).
>>>>> --> Meanwhile, Grizzly context is recycled
>>>>> --> A thread from the Business processing tier (BPT) digests the
>>>>> message
>>>>> asynchronously and produces a response
>>>>> --> The BPT turns to Grizzly for handing over the response
>>>>> bytebuffer in an
>>>>> async write.
>>>>> --> The processes of parsing messages on read byte buffers,
>>>>> processing
>>>>> parsed messages and writing them back to the network framework are
>>>>> independent of each other. Eg. the server could be parsing a message
>>>>> M2 in
>>>>> one BPT while writing M1 in another BPT.
>>>>>
>>>>> Rephrasing the question, can I use the solution mentioned in
>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>> while preserving my server's luxury of writing back to grizzly
>>>>> using a
>>>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>>>> without
>>>>> suspending the reads and other business processing?
>>>> Absolutely. Then, in your case I'm not sure you need to use
>>>> SuspendFilter, as basically you don't want to suspend the
>>>> connection -
>>>> but keep it process other requests.
>>>> Once your custom thread will complete the task execution - context
>>>> will not be accessible anymore, as it was recycled by Grizzly worker
>>>> thread, so you'll need to access async queue writer different way
>>>> like:
>>>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>>>> byteBuffer);
>>>>
>>>>
>>>>> 2) Yes, I did mean the re-handshaking. It is done for every chunk of
>>>>> the
>>>>> bytearray written synchronously using SSLOutputWriter.flushChannel()
>>>>> but,
>>>>> the same is not done if I try to write asynchronously using an
>>>>> AsyncQueueWriter.write() with SSL as the final write is handled by
>>>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>>>> WritePreprocessor and tries writing the resultant bytebuffer
>>>>> contents
>>>>> directly to the channel without any checks for re-handshaking. I
>>>>> wanted to
>>>>> know if it is left to the user to implement it in his/her own code
>>>>> for now?
>>>> you mean this part of SSLOutputWriter.flushChannel()?
>>>> //--------------- Cut --------------------------
>>>> case OK:
>>>> if (result.getHandshakeStatus() ==
>>>> HandshakeStatus.NEED_TASK) {
>>>> SSLUtils.executeDelegatedTask(sslEngine);
>>>> }
>>>> break;
>>>> //----------------------------------------------
>>>> If yes, can I ask you to add the same lines to your SSL preprocessor
>>>> to check whether it works for you - if yes - we'll fix that in
>>>> Grizzly
>>>> examples.
>>>>
>>>> Thanks.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>>>
>>>>>
>>>>> Thanks & Regards,
>>>>> Phani
>>>>>
>>>>>
>>>>> Oleksiy Stashok wrote:
>>>>>>
>>>>>> Hello Phani,
>>>>>>
>>>>>>> I am working on using Grizzly for our server application which
>>>>>>> uses
>>>>>>> SSLv3
>>>>>>> and a custom binary application level protocol for client-server
>>>>>>> talk. My
>>>>>>> server side implementation of Grizzly and the problems I am facing
>>>>>>> are
>>>>>>> similar to the ones in the thread below ...
>>>>>>>
>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>>
>>>>>>> However I have some doubts before I implement the solution
>>>>>>> recommended in
>>>>>>> the thread ...
>>>>>>>
>>>>>>> I am going to implement both Async reads and writes on the server
>>>>>>> side and
>>>>>>> the problem with adopting the solution for this is that once I
>>>>>>> suspend
>>>>>>> execution of a ProtocolChain using the SuspendableFilter a call to
>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>>>>> correct me
>>>>>>> if I am wrong but, as a result of this reads won't happen on the
>>>>>>> channel. Is
>>>>>>> there a way out for this? I do not want reads to be waiting on
>>>>>>> writes
>>>>>>> caused by the previous read. I want reads and writes to be
>>>>>>> independent of
>>>>>>> each other.
>>>>>> You're right. In your case you don't need to have
>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just
>>>>>> let
>>>>>> channel to be re-registered for OP_READ.
>>>>>>
>>>>>>
>>>>>>> Also, I see that in case of Async writes for SSL / TLS the final
>>>>>>> channel
>>>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>>>> processor
>>>>>>> to wrap
>>>>>>> my bytes before sending them out but, the part to redo the broken
>>>>>>> handshakes
>>>>>>> before sending every chunk of bytes out to the channel as in
>>>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>>>> TCPAsyncQueueWriter.
>>>>>>> Please advice on how to work around this problem without drifting
>>>>>>> away from
>>>>>>> the framework code and classes. For now, I am thinking of
>>>>>>> implementing a new
>>>>>>> SSLAsyncQueueWriter.
>>>>>> Can you pls. provide details what part, you think, is missed in
>>>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>>>> If it's some common problem, we have to implement that in default
>>>>>> Grizzly implementation and your help will be very appreciated!
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> WBR,
>>>>>> Alexey.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help would be very much appreciated.
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> Phani
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18505329.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.