Hello Phani,
I'll take a look, could you also share the stress tester code?
Thanks.
WBR,
Alexey.
On Jul 22, 2008, at 14:42 , PhaKuDi wrote:
>
> Hi Alexey,
>
> I have made the ProtocolFilter thread safe. Yet the connection
> failures
> persist. It is probably something I am missing. Please find the latest
> source code of my Grizzly related classes in the attached zip file.
> All help
> will be appreciated.
>
> http://www.nabble.com/file/p18587446/src.zip src.zip
>
> Thanks & Regards,
> Phani
>
>
> Oleksiy Stashok wrote:
>>
>> Ok, Phani,
>>
>> just let us know, once you'll have any results.
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>> On Jul 17, 2008, at 12:11 , PhaKuDi wrote:
>>
>>>
>>> Hi Alexey,
>>>
>>> Looking back at MySSLReadFilter I think I have found the problem -
>>> it is not
>>> exactly thread safe.
>>>
>>> I will fix it and get back to you in case I have some real issues.
>>> Sorry
>>> about the false alarm.
>>>
>>> Thanks!
>>> Phani
>>>
>>>
>>> PhaKuDi wrote:
>>>>
>>>> Hi Alexey,
>>>>
>>>> Please find the source files in the zip file attached.
>>>>
>>>> http://www.nabble.com/file/p18503031/src.zip src.zip
>>>>
>>>> Regards,
>>>> Phani
>>>>
>>>>
>>>> Oleksiy Stashok wrote:
>>>>>
>>>>> Hello Phani,
>>>>>
>>>>> can you pls. send the complete java sources of Grizzly related
>>>>> classes, it could be easier to read the code.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>> On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>>>>>
>>>>>>
>>>>>> Hello Alexey,
>>>>>>
>>>>>> My apologies for the delay in responding. Regarding ...
>>>>>>
>>>>>> 1) Thanks for the confirmation. Getting hold of async writer from
>>>>>> the
>>>>>> selector handler is the right way to go for me.
>>>>>> 2) Just using a SSLWritePreprocessor isn't helping me. Have
>>>>>> tested
>>>>>> with the
>>>>>> re-handshaking code and it works for me.
>>>>>>
>>>>>> Also, now that I am stress testing the server I am running into a
>>>>>> few
>>>>>> problems. I would very much appreciate your suggestions/
>>>>>> comments on
>>>>>> these
>>>>>> issues. The server in question currently is capable of catering
>>>>>> to
>>>>>> about
>>>>>> 4000 clients during peak times and uses a C library for SSL
>>>>>> communication
>>>>>> with the clients. We are in the process of moving it to an NIO
>>>>>> based
>>>>>> framework with minimal impact on existing design. My
>>>>>> implementation of
>>>>>> Grizzly is as below ...
>>>>>>
>>>>>> --> Server initalizes the controller as follows ...
>>>>>>
>>>>>> SSLReadFilter sslReadFilter = new
>>>>>> MySSLReadFilter(gameServer,
>>>>>> USE_GRIZZLY_ASYNC_READS);
>>>>>> sslReadFilter.setSSLContext(context);
>>>>>> final ProtocolChain protocolChain = new
>>>>>> DefaultProtocolChain();
>>>>>> protocolChain.addFilter(sslReadFilter);
>>>>>> ProtocolChainInstanceHandler chainInstanceHandler = new
>>>>>> DefaultProtocolChainInstanceHandler() {
>>>>>> public ProtocolChain poll() {
>>>>>> return protocolChain;
>>>>>> }
>>>>>> };
>>>>>> Controller controller = new Controller();
>>>>>> controller.setHandleReadWriteConcurrently(true);
>>>>>> SSLSelectorHandler selectorHandler = new
>>>>>> MySSLSelectorHandler();
>>>>>> selectorHandler.setPort(port);
>>>>>> controller.setSelectorHandler(selectorHandler);
>>>>>>
>>>>>> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>>>>>>
>>>>>> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
>>>>>> Currently set to 0
>>>>>> OutputWriter.setDefaultWriteTimeout(5000);
>>>>>> controller.start();
>>>>>>
>>>>>> --> MySSLReadFilter.execute() method does ...
>>>>>>
>>>>>> super.execute(context);
>>>>>> engine = workerThread.getSSLEngine();
>>>>>> session = engine.getSession();
>>>>>> Object obj =
>>>>>> session.getValue(ServerConHandler.class.getName());
>>>>>> ServerConHandler conHandler;
>>>>>> if (obj == null || !(obj instanceof ServerConHandler)) {
>>>>>> SSLSelectorHandler selectorHandler =
>>>>>> (SSLSelectorHandler)
>>>>>> context.getSelectorHandler();
>>>>>> conHandler = new ServerConHandler(selectorHandler,
>>>>>> selectionKey,
>>>>>> (SocketChannel) selectionKey.channel(), engine, user);
>>>>>> session.putValue(ServerConHandler.class.getName(),
>>>>>> conHandler);
>>>>>> } else {
>>>>>> conHandler = (ServerConHandler) obj;
>>>>>> }
>>>>>> conHandler.read(workerThread.getByteBuffer());
>>>>>>
>>>>>> --> MySSLSelectorHandler does ...
>>>>>>
>>>>>> public void preSelect(Context ctx) throws IOException {
>>>>>> if (asyncQueueWriter == null) {
>>>>>> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
>>>>>> }
>>>>>> super.preSelect(ctx);
>>>>>> }
>>>>>>
>>>>>> --> ServerConHandler on receiving a write request does ...
>>>>>>
>>>>>> try {
>>>>>> MySSLAsyncQueueWriter writer =
>>>>>> (MySSLAsyncQueueWriter)
>>>>>> selectorHandler.getAsyncQueueWriter();
>>>>>> writer.write(selectionKey, ByteBuffer.wrap(b1),
>>>>>> sslEngine);
>>>>>> } catch (ClosedChannelException e) {
>>>>>> logger.error(e.getMessage());
>>>>>> } catch (IOException e) {
>>>>>> logger.error(e.getMessage());
>>>>>> }
>>>>>>
>>>>>> --> MySSLAsyncQueueWriter does ...
>>>>>>
>>>>>> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
>>>>>> sslEngine) throws IOException {
>>>>>> ChannelRegistrationRecord record =
>>>>>> channelRegistrationMap.putIfAbsent(key.channel(), new
>>>>>> ChannelRegistrationRecord(sslEngine));
>>>>>> if (record != null) {
>>>>>> record.writeCount.incrementAndGet();
>>>>>> }
>>>>>> super.write(key, buffer);
>>>>>> }
>>>>>>
>>>>>> protected void doWrite(WritableByteChannel channel,
>>>>>> SocketAddress
>>>>>> dstAddress, ByteBuffer byteBuffer) throws IOException {
>>>>>> SSLEngine sslEngine;
>>>>>> ChannelRegistrationRecord record =
>>>>>> channelRegistrationMap.get(channel);
>>>>>> if (record == null) {
>>>>>> logger.error("Channel is not registered for an async
>>>>>> write");
>>>>>> return;
>>>>>> }
>>>>>> sslEngine = record.sslEngine;
>>>>>> if (sslEngine == null) {
>>>>>> logger.error("SSLEngine is null.");
>>>>>> return;
>>>>>> }
>>>>>> if (record.writeCount.intValue() > 200) {
>>>>>> ((SocketChannel)channel).socket().close();
>>>>>> channel.close();
>>>>>> channelRegistrationMap.remove(channel);
>>>>>> return;
>>>>>> }
>>>>>> if (record.writeCount.decrementAndGet() == 0) {
>>>>>> channelRegistrationMap.remove(channel);
>>>>>> }
>>>>>> SSLOutputWriter.flushChannel((SelectableChannel) channel,
>>>>>> byteBuffer,
>>>>>> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
>>>>>> sslEngine.getSession().getPacketBufferSize())), sslEngine);
>>>>>> }
>>>>>>
>>>>>>
>>>>>> The issues we are observing are ...
>>>>>>
>>>>>> --> As the number of concurrent connections shoots up to a number
>>>>>> above 200
>>>>>> the stress clients start facing disconnections and following
>>>>>> exceptions are
>>>>>> seen on the server side.
>>>>>>
>>>>>> java.nio.channels.ClosedChannelException
>>>>>> at
>>>>>> sun
>>>>>> .nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
>>>>>> 125)
>>>>>> at
>>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
>>>>>> 294)
>>>>>> at
>>>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>>>> 105)
>>>>>> at
>>>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>>>> 73)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
>>>>>> 100)
>>>>>> at
>>>>>> com
>>>>>> .my
>>>>>> .server
>>>>>> .grizzly
>>>>>> .MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:
>>>>>> 70)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .async
>>>>>> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:
>>>>>> 374)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .async
>>>>>> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:
>>>>>> 291)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .async
>>>>>> .AsyncQueueWriterContextTask
>>>>>> .doCall(AsyncQueueWriterContextTask.java:
>>>>>> 86)
>>>>>> at
>>>>>> com
>>>>>> .sun
>>>>>> .grizzly
>>>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
>>>>>> at
>>>>>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:
>>>>>> 169)
>>>>>>
>>>>>> --> The CPU load on the 4 CPU machine shoots up to a 100%
>>>>>> consistently.
>>>>>> --> The memory graph shows no decline.
>>>>>>
>>>>>> I think it must be something that I am doing wrong. Also, I
>>>>>> wanted
>>>>>> to know
>>>>>> if Grizzly has been tested to scale up to 4000 ssl connections
>>>>>> and
>>>>>> beyond if
>>>>>> required.
>>>>>>
>>>>>> Please feel free to revert in case you need some more
>>>>>> information.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> Phani
>>>>>>
>>>>>>
>>>>>>
>>>>>> Oleksiy Stashok wrote:
>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 1) I think I should have given you a bit more info on this,
>>>>>>>> this is
>>>>>>>> what
>>>>>>>> happens on the server ...
>>>>>>>>
>>>>>>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>>>>>>> ByteBuffer
>>>>>>>> --> Server handler parses the ByteBuffer and hands it over to a
>>>>>>>> business
>>>>>>>> processing tier (using custom server threads).
>>>>>>>> --> Meanwhile, Grizzly context is recycled
>>>>>>>> --> A thread from the Business processing tier (BPT) digests
>>>>>>>> the
>>>>>>>> message
>>>>>>>> asynchronously and produces a response
>>>>>>>> --> The BPT turns to Grizzly for handing over the response
>>>>>>>> bytebuffer in an
>>>>>>>> async write.
>>>>>>>> --> The processes of parsing messages on read byte buffers,
>>>>>>>> processing
>>>>>>>> parsed messages and writing them back to the network framework
>>>>>>>> are
>>>>>>>> independent of each other. Eg. the server could be parsing a
>>>>>>>> message
>>>>>>>> M2 in
>>>>>>>> one BPT while writing M1 in another BPT.
>>>>>>>>
>>>>>>>> Rephrasing the question, can I use the solution mentioned in
>>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>>> while preserving my server's luxury of writing back to grizzly
>>>>>>>> using a
>>>>>>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>>>>>>> without
>>>>>>>> suspending the reads and other business processing?
>>>>>>> Absolutely. Then, in your case I'm not sure you need to use
>>>>>>> SuspendFilter, as basically you don't want to suspend the
>>>>>>> connection -
>>>>>>> but keep it process other requests.
>>>>>>> Once your custom thread will complete the task execution -
>>>>>>> context
>>>>>>> will not be accessible anymore, as it was recycled by Grizzly
>>>>>>> worker
>>>>>>> thread, so you'll need to access async queue writer different
>>>>>>> way
>>>>>>> like:
>>>>>>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>>>>>>> byteBuffer);
>>>>>>>
>>>>>>>
>>>>>>>> 2) Yes, I did mean the re-handshaking. It is done for every
>>>>>>>> chunk of
>>>>>>>> the
>>>>>>>> bytearray written synchronously using
>>>>>>>> SSLOutputWriter.flushChannel()
>>>>>>>> but,
>>>>>>>> the same is not done if I try to write asynchronously using an
>>>>>>>> AsyncQueueWriter.write() with SSL as the final write is handled
>>>>>>>> by
>>>>>>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>>>>>>> WritePreprocessor and tries writing the resultant bytebuffer
>>>>>>>> contents
>>>>>>>> directly to the channel without any checks for re-
>>>>>>>> handshaking. I
>>>>>>>> wanted to
>>>>>>>> know if it is left to the user to implement it in his/her own
>>>>>>>> code
>>>>>>>> for now?
>>>>>>> you mean this part of SSLOutputWriter.flushChannel()?
>>>>>>> //--------------- Cut --------------------------
>>>>>>> case OK:
>>>>>>> if (result.getHandshakeStatus() ==
>>>>>>> HandshakeStatus.NEED_TASK) {
>>>>>>> SSLUtils.executeDelegatedTask(sslEngine);
>>>>>>> }
>>>>>>> break;
>>>>>>> //----------------------------------------------
>>>>>>> If yes, can I ask you to add the same lines to your SSL
>>>>>>> preprocessor
>>>>>>> to check whether it works for you - if yes - we'll fix that in
>>>>>>> Grizzly
>>>>>>> examples.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> WBR,
>>>>>>> Alexey.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> Phani
>>>>>>>>
>>>>>>>>
>>>>>>>> Oleksiy Stashok wrote:
>>>>>>>>>
>>>>>>>>> Hello Phani,
>>>>>>>>>
>>>>>>>>>> I am working on using Grizzly for our server application
>>>>>>>>>> which
>>>>>>>>>> uses
>>>>>>>>>> SSLv3
>>>>>>>>>> and a custom binary application level protocol for client-
>>>>>>>>>> server
>>>>>>>>>> talk. My
>>>>>>>>>> server side implementation of Grizzly and the problems I am
>>>>>>>>>> facing
>>>>>>>>>> are
>>>>>>>>>> similar to the ones in the thread below ...
>>>>>>>>>>
>>>>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>>>>>
>>>>>>>>>> However I have some doubts before I implement the solution
>>>>>>>>>> recommended in
>>>>>>>>>> the thread ...
>>>>>>>>>>
>>>>>>>>>> I am going to implement both Async reads and writes on the
>>>>>>>>>> server
>>>>>>>>>> side and
>>>>>>>>>> the problem with adopting the solution for this is that
>>>>>>>>>> once I
>>>>>>>>>> suspend
>>>>>>>>>> execution of a ProtocolChain using the SuspendableFilter a
>>>>>>>>>> call to
>>>>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is
>>>>>>>>>> made,
>>>>>>>>>> correct me
>>>>>>>>>> if I am wrong but, as a result of this reads won't happen on
>>>>>>>>>> the
>>>>>>>>>> channel. Is
>>>>>>>>>> there a way out for this? I do not want reads to be
>>>>>>>>>> waiting on
>>>>>>>>>> writes
>>>>>>>>>> caused by the previous read. I want reads and writes to be
>>>>>>>>>> independent of
>>>>>>>>>> each other.
>>>>>>>>> You're right. In your case you don't need to have
>>>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but
>>>>>>>>> just
>>>>>>>>> let
>>>>>>>>> channel to be re-registered for OP_READ.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Also, I see that in case of Async writes for SSL / TLS the
>>>>>>>>>> final
>>>>>>>>>> channel
>>>>>>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>>>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>>>>>>> processor
>>>>>>>>>> to wrap
>>>>>>>>>> my bytes before sending them out but, the part to redo the
>>>>>>>>>> broken
>>>>>>>>>> handshakes
>>>>>>>>>> before sending every chunk of bytes out to the channel as in
>>>>>>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>>>>>>> TCPAsyncQueueWriter.
>>>>>>>>>> Please advice on how to work around this problem without
>>>>>>>>>> drifting
>>>>>>>>>> away from
>>>>>>>>>> the framework code and classes. For now, I am thinking of
>>>>>>>>>> implementing a new
>>>>>>>>>> SSLAsyncQueueWriter.
>>>>>>>>> Can you pls. provide details what part, you think, is missed
>>>>>>>>> in
>>>>>>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>>>>>>> If it's some common problem, we have to implement that in
>>>>>>>>> default
>>>>>>>>> Grizzly implementation and your help will be very appreciated!
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>> WBR,
>>>>>>>>> Alexey.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Any help would be very much appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Phani
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>>>> Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-
>>>>>>>>>> unsubscribe_at_grizzly.dev.java.net
>>>>>>>>>> For additional commands, e-mail: users-
>>>>>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-
>>>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/SSL-ARP-tp18381476p18505329.html
>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18587446.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>