Hi Alexey,
I have made the ProtocolFilter thread safe. Yet the connection failures
persist. It is probably something I am missing. Please find the latest
source code of my Grizzly related classes in the attached zip file. All help
will be appreciated.
http://www.nabble.com/file/p18587446/src.zip src.zip
Thanks & Regards,
Phani
Oleksiy Stashok wrote:
>
> Ok, Phani,
>
> just let us know, once you'll have any results.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On Jul 17, 2008, at 12:11 , PhaKuDi wrote:
>
>>
>> Hi Alexey,
>>
>> Looking back at MySSLReadFilter I think I have found the problem -
>> it is not
>> exactly thread safe.
>>
>> I will fix it and get back to you in case I have some real issues.
>> Sorry
>> about the false alarm.
>>
>> Thanks!
>> Phani
>>
>>
>> PhaKuDi wrote:
>>>
>>> Hi Alexey,
>>>
>>> Please find the source files in the zip file attached.
>>>
>>> http://www.nabble.com/file/p18503031/src.zip src.zip
>>>
>>> Regards,
>>> Phani
>>>
>>>
>>> Oleksiy Stashok wrote:
>>>>
>>>> Hello Phani,
>>>>
>>>> can you pls. send the complete java sources of Grizzly related
>>>> classes, it could be easier to read the code.
>>>>
>>>> Thanks.
>>>>
>>>> WBR,
>>>> Alexey.
>>>>
>>>> On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>>>>
>>>>>
>>>>> Hello Alexey,
>>>>>
>>>>> My apologies for the delay in responding. Regarding ...
>>>>>
>>>>> 1) Thanks for the confirmation. Getting hold of async writer from
>>>>> the
>>>>> selector handler is the right way to go for me.
>>>>> 2) Just using a SSLWritePreprocessor isn't helping me. Have tested
>>>>> with the
>>>>> re-handshaking code and it works for me.
>>>>>
>>>>> Also, now that I am stress testing the server I am running into a
>>>>> few
>>>>> problems. I would very much appreciate your suggestions/comments on
>>>>> these
>>>>> issues. The server in question currently is capable of catering to
>>>>> about
>>>>> 4000 clients during peak times and uses a C library for SSL
>>>>> communication
>>>>> with the clients. We are in the process of moving it to an NIO
>>>>> based
>>>>> framework with minimal impact on existing design. My
>>>>> implementation of
>>>>> Grizzly is as below ...
>>>>>
>>>>> --> Server initalizes the controller as follows ...
>>>>>
>>>>> SSLReadFilter sslReadFilter = new
>>>>> MySSLReadFilter(gameServer,
>>>>> USE_GRIZZLY_ASYNC_READS);
>>>>> sslReadFilter.setSSLContext(context);
>>>>> final ProtocolChain protocolChain = new
>>>>> DefaultProtocolChain();
>>>>> protocolChain.addFilter(sslReadFilter);
>>>>> ProtocolChainInstanceHandler chainInstanceHandler = new
>>>>> DefaultProtocolChainInstanceHandler() {
>>>>> public ProtocolChain poll() {
>>>>> return protocolChain;
>>>>> }
>>>>> };
>>>>> Controller controller = new Controller();
>>>>> controller.setHandleReadWriteConcurrently(true);
>>>>> SSLSelectorHandler selectorHandler = new
>>>>> MySSLSelectorHandler();
>>>>> selectorHandler.setPort(port);
>>>>> controller.setSelectorHandler(selectorHandler);
>>>>>
>>>>> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>>>>>
>>>>> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
>>>>> Currently set to 0
>>>>> OutputWriter.setDefaultWriteTimeout(5000);
>>>>> controller.start();
>>>>>
>>>>> --> MySSLReadFilter.execute() method does ...
>>>>>
>>>>> super.execute(context);
>>>>> engine = workerThread.getSSLEngine();
>>>>> session = engine.getSession();
>>>>> Object obj =
>>>>> session.getValue(ServerConHandler.class.getName());
>>>>> ServerConHandler conHandler;
>>>>> if (obj == null || !(obj instanceof ServerConHandler)) {
>>>>> SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
>>>>> context.getSelectorHandler();
>>>>> conHandler = new ServerConHandler(selectorHandler,
>>>>> selectionKey,
>>>>> (SocketChannel) selectionKey.channel(), engine, user);
>>>>> session.putValue(ServerConHandler.class.getName(),
>>>>> conHandler);
>>>>> } else {
>>>>> conHandler = (ServerConHandler) obj;
>>>>> }
>>>>> conHandler.read(workerThread.getByteBuffer());
>>>>>
>>>>> --> MySSLSelectorHandler does ...
>>>>>
>>>>> public void preSelect(Context ctx) throws IOException {
>>>>> if (asyncQueueWriter == null) {
>>>>> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
>>>>> }
>>>>> super.preSelect(ctx);
>>>>> }
>>>>>
>>>>> --> ServerConHandler on receiving a write request does ...
>>>>>
>>>>> try {
>>>>> MySSLAsyncQueueWriter writer =
>>>>> (MySSLAsyncQueueWriter)
>>>>> selectorHandler.getAsyncQueueWriter();
>>>>> writer.write(selectionKey, ByteBuffer.wrap(b1),
>>>>> sslEngine);
>>>>> } catch (ClosedChannelException e) {
>>>>> logger.error(e.getMessage());
>>>>> } catch (IOException e) {
>>>>> logger.error(e.getMessage());
>>>>> }
>>>>>
>>>>> --> MySSLAsyncQueueWriter does ...
>>>>>
>>>>> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
>>>>> sslEngine) throws IOException {
>>>>> ChannelRegistrationRecord record =
>>>>> channelRegistrationMap.putIfAbsent(key.channel(), new
>>>>> ChannelRegistrationRecord(sslEngine));
>>>>> if (record != null) {
>>>>> record.writeCount.incrementAndGet();
>>>>> }
>>>>> super.write(key, buffer);
>>>>> }
>>>>>
>>>>> protected void doWrite(WritableByteChannel channel, SocketAddress
>>>>> dstAddress, ByteBuffer byteBuffer) throws IOException {
>>>>> SSLEngine sslEngine;
>>>>> ChannelRegistrationRecord record =
>>>>> channelRegistrationMap.get(channel);
>>>>> if (record == null) {
>>>>> logger.error("Channel is not registered for an async
>>>>> write");
>>>>> return;
>>>>> }
>>>>> sslEngine = record.sslEngine;
>>>>> if (sslEngine == null) {
>>>>> logger.error("SSLEngine is null.");
>>>>> return;
>>>>> }
>>>>> if (record.writeCount.intValue() > 200) {
>>>>> ((SocketChannel)channel).socket().close();
>>>>> channel.close();
>>>>> channelRegistrationMap.remove(channel);
>>>>> return;
>>>>> }
>>>>> if (record.writeCount.decrementAndGet() == 0) {
>>>>> channelRegistrationMap.remove(channel);
>>>>> }
>>>>> SSLOutputWriter.flushChannel((SelectableChannel) channel,
>>>>> byteBuffer,
>>>>> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
>>>>> sslEngine.getSession().getPacketBufferSize())), sslEngine);
>>>>> }
>>>>>
>>>>>
>>>>> The issues we are observing are ...
>>>>>
>>>>> --> As the number of concurrent connections shoots up to a number
>>>>> above 200
>>>>> the stress clients start facing disconnections and following
>>>>> exceptions are
>>>>> seen on the server side.
>>>>>
>>>>> java.nio.channels.ClosedChannelException
>>>>> at
>>>>> sun
>>>>> .nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
>>>>> 125)
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
>>>>> 294)
>>>>> at
>>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>>> 105)
>>>>> at
>>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>>> 73)
>>>>> at
>>>>> com
>>>>> .sun
>>>>> .grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
>>>>> 100)
>>>>> at
>>>>> com
>>>>> .my
>>>>> .server
>>>>> .grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:
>>>>> 70)
>>>>> at
>>>>> com
>>>>> .sun
>>>>> .grizzly
>>>>> .async
>>>>> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:
>>>>> 374)
>>>>> at
>>>>> com
>>>>> .sun
>>>>> .grizzly
>>>>> .async
>>>>> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:
>>>>> 291)
>>>>> at
>>>>> com
>>>>> .sun
>>>>> .grizzly
>>>>> .async
>>>>> .AsyncQueueWriterContextTask
>>>>> .doCall(AsyncQueueWriterContextTask.java:
>>>>> 86)
>>>>> at
>>>>> com
>>>>> .sun
>>>>> .grizzly
>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
>>>>> at
>>>>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:
>>>>> 169)
>>>>>
>>>>> --> The CPU load on the 4 CPU machine shoots up to a 100%
>>>>> consistently.
>>>>> --> The memory graph shows no decline.
>>>>>
>>>>> I think it must be something that I am doing wrong. Also, I wanted
>>>>> to know
>>>>> if Grizzly has been tested to scale up to 4000 ssl connections and
>>>>> beyond if
>>>>> required.
>>>>>
>>>>> Please feel free to revert in case you need some more information.
>>>>>
>>>>> Thanks & Regards,
>>>>> Phani
>>>>>
>>>>>
>>>>>
>>>>> Oleksiy Stashok wrote:
>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>> 1) I think I should have given you a bit more info on this,
>>>>>>> this is
>>>>>>> what
>>>>>>> happens on the server ...
>>>>>>>
>>>>>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>>>>>> ByteBuffer
>>>>>>> --> Server handler parses the ByteBuffer and hands it over to a
>>>>>>> business
>>>>>>> processing tier (using custom server threads).
>>>>>>> --> Meanwhile, Grizzly context is recycled
>>>>>>> --> A thread from the Business processing tier (BPT) digests the
>>>>>>> message
>>>>>>> asynchronously and produces a response
>>>>>>> --> The BPT turns to Grizzly for handing over the response
>>>>>>> bytebuffer in an
>>>>>>> async write.
>>>>>>> --> The processes of parsing messages on read byte buffers,
>>>>>>> processing
>>>>>>> parsed messages and writing them back to the network framework
>>>>>>> are
>>>>>>> independent of each other. Eg. the server could be parsing a
>>>>>>> message
>>>>>>> M2 in
>>>>>>> one BPT while writing M1 in another BPT.
>>>>>>>
>>>>>>> Rephrasing the question, can I use the solution mentioned in
>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>> while preserving my server's luxury of writing back to grizzly
>>>>>>> using a
>>>>>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>>>>>> without
>>>>>>> suspending the reads and other business processing?
>>>>>> Absolutely. Then, in your case I'm not sure you need to use
>>>>>> SuspendFilter, as basically you don't want to suspend the
>>>>>> connection -
>>>>>> but keep it process other requests.
>>>>>> Once your custom thread will complete the task execution - context
>>>>>> will not be accessible anymore, as it was recycled by Grizzly
>>>>>> worker
>>>>>> thread, so you'll need to access async queue writer different way
>>>>>> like:
>>>>>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>>>>>> byteBuffer);
>>>>>>
>>>>>>
>>>>>>> 2) Yes, I did mean the re-handshaking. It is done for every
>>>>>>> chunk of
>>>>>>> the
>>>>>>> bytearray written synchronously using
>>>>>>> SSLOutputWriter.flushChannel()
>>>>>>> but,
>>>>>>> the same is not done if I try to write asynchronously using an
>>>>>>> AsyncQueueWriter.write() with SSL as the final write is handled
>>>>>>> by
>>>>>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>>>>>> WritePreprocessor and tries writing the resultant bytebuffer
>>>>>>> contents
>>>>>>> directly to the channel without any checks for re-handshaking. I
>>>>>>> wanted to
>>>>>>> know if it is left to the user to implement it in his/her own
>>>>>>> code
>>>>>>> for now?
>>>>>> you mean this part of SSLOutputWriter.flushChannel()?
>>>>>> //--------------- Cut --------------------------
>>>>>> case OK:
>>>>>> if (result.getHandshakeStatus() ==
>>>>>> HandshakeStatus.NEED_TASK) {
>>>>>> SSLUtils.executeDelegatedTask(sslEngine);
>>>>>> }
>>>>>> break;
>>>>>> //----------------------------------------------
>>>>>> If yes, can I ask you to add the same lines to your SSL
>>>>>> preprocessor
>>>>>> to check whether it works for you - if yes - we'll fix that in
>>>>>> Grizzly
>>>>>> examples.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> WBR,
>>>>>> Alexey.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> Phani
>>>>>>>
>>>>>>>
>>>>>>> Oleksiy Stashok wrote:
>>>>>>>>
>>>>>>>> Hello Phani,
>>>>>>>>
>>>>>>>>> I am working on using Grizzly for our server application which
>>>>>>>>> uses
>>>>>>>>> SSLv3
>>>>>>>>> and a custom binary application level protocol for client-
>>>>>>>>> server
>>>>>>>>> talk. My
>>>>>>>>> server side implementation of Grizzly and the problems I am
>>>>>>>>> facing
>>>>>>>>> are
>>>>>>>>> similar to the ones in the thread below ...
>>>>>>>>>
>>>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>>>>
>>>>>>>>> However I have some doubts before I implement the solution
>>>>>>>>> recommended in
>>>>>>>>> the thread ...
>>>>>>>>>
>>>>>>>>> I am going to implement both Async reads and writes on the
>>>>>>>>> server
>>>>>>>>> side and
>>>>>>>>> the problem with adopting the solution for this is that once I
>>>>>>>>> suspend
>>>>>>>>> execution of a ProtocolChain using the SuspendableFilter a
>>>>>>>>> call to
>>>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>>>>>>> correct me
>>>>>>>>> if I am wrong but, as a result of this reads won't happen on
>>>>>>>>> the
>>>>>>>>> channel. Is
>>>>>>>>> there a way out for this? I do not want reads to be waiting on
>>>>>>>>> writes
>>>>>>>>> caused by the previous read. I want reads and writes to be
>>>>>>>>> independent of
>>>>>>>>> each other.
>>>>>>>> You're right. In your case you don't need to have
>>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just
>>>>>>>> let
>>>>>>>> channel to be re-registered for OP_READ.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also, I see that in case of Async writes for SSL / TLS the
>>>>>>>>> final
>>>>>>>>> channel
>>>>>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>>>>>> processor
>>>>>>>>> to wrap
>>>>>>>>> my bytes before sending them out but, the part to redo the
>>>>>>>>> broken
>>>>>>>>> handshakes
>>>>>>>>> before sending every chunk of bytes out to the channel as in
>>>>>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>>>>>> TCPAsyncQueueWriter.
>>>>>>>>> Please advice on how to work around this problem without
>>>>>>>>> drifting
>>>>>>>>> away from
>>>>>>>>> the framework code and classes. For now, I am thinking of
>>>>>>>>> implementing a new
>>>>>>>>> SSLAsyncQueueWriter.
>>>>>>>> Can you pls. provide details what part, you think, is missed in
>>>>>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>>>>>> If it's some common problem, we have to implement that in
>>>>>>>> default
>>>>>>>> Grizzly implementation and your help will be very appreciated!
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> WBR,
>>>>>>>> Alexey.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help would be very much appreciated.
>>>>>>>>>
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Phani
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>>> Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SSL-ARP-tp18381476p18505329.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
--
View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18587446.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.