Ok, Phani,
just let us know, once you'll have any results.
Thanks.
WBR,
Alexey.
On Jul 17, 2008, at 12:11 , PhaKuDi wrote:
>
> Hi Alexey,
>
> Looking back at MySSLReadFilter I think I have found the problem -
> it is not
> exactly thread safe.
>
> I will fix it and get back to you in case I have some real issues.
> Sorry
> about the false alarm.
>
> Thanks!
> Phani
>
>
> PhaKuDi wrote:
>>
>> Hi Alexey,
>>
>> Please find the source files in the zip file attached.
>>
>> http://www.nabble.com/file/p18503031/src.zip src.zip
>>
>> Regards,
>> Phani
>>
>>
>> Oleksiy Stashok wrote:
>>>
>>> Hello Phani,
>>>
>>> can you pls. send the complete java sources of Grizzly related
>>> classes, it could be easier to read the code.
>>>
>>> Thanks.
>>>
>>> WBR,
>>> Alexey.
>>>
>>> On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>>>
>>>>
>>>> Hello Alexey,
>>>>
>>>> My apologies for the delay in responding. Regarding ...
>>>>
>>>> 1) Thanks for the confirmation. Getting hold of async writer from
>>>> the
>>>> selector handler is the right way to go for me.
>>>> 2) Just using a SSLWritePreprocessor isn't helping me. Have tested
>>>> with the
>>>> re-handshaking code and it works for me.
>>>>
>>>> Also, now that I am stress testing the server I am running into a
>>>> few
>>>> problems. I would very much appreciate your suggestions/comments on
>>>> these
>>>> issues. The server in question currently is capable of catering to
>>>> about
>>>> 4000 clients during peak times and uses a C library for SSL
>>>> communication
>>>> with the clients. We are in the process of moving it to an NIO
>>>> based
>>>> framework with minimal impact on existing design. My
>>>> implementation of
>>>> Grizzly is as below ...
>>>>
>>>> --> Server initalizes the controller as follows ...
>>>>
>>>> SSLReadFilter sslReadFilter = new
>>>> MySSLReadFilter(gameServer,
>>>> USE_GRIZZLY_ASYNC_READS);
>>>> sslReadFilter.setSSLContext(context);
>>>> final ProtocolChain protocolChain = new
>>>> DefaultProtocolChain();
>>>> protocolChain.addFilter(sslReadFilter);
>>>> ProtocolChainInstanceHandler chainInstanceHandler = new
>>>> DefaultProtocolChainInstanceHandler() {
>>>> public ProtocolChain poll() {
>>>> return protocolChain;
>>>> }
>>>> };
>>>> Controller controller = new Controller();
>>>> controller.setHandleReadWriteConcurrently(true);
>>>> SSLSelectorHandler selectorHandler = new
>>>> MySSLSelectorHandler();
>>>> selectorHandler.setPort(port);
>>>> controller.setSelectorHandler(selectorHandler);
>>>>
>>>> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>>>>
>>>> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
>>>> Currently set to 0
>>>> OutputWriter.setDefaultWriteTimeout(5000);
>>>> controller.start();
>>>>
>>>> --> MySSLReadFilter.execute() method does ...
>>>>
>>>> super.execute(context);
>>>> engine = workerThread.getSSLEngine();
>>>> session = engine.getSession();
>>>> Object obj =
>>>> session.getValue(ServerConHandler.class.getName());
>>>> ServerConHandler conHandler;
>>>> if (obj == null || !(obj instanceof ServerConHandler)) {
>>>> SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
>>>> context.getSelectorHandler();
>>>> conHandler = new ServerConHandler(selectorHandler,
>>>> selectionKey,
>>>> (SocketChannel) selectionKey.channel(), engine, user);
>>>> session.putValue(ServerConHandler.class.getName(),
>>>> conHandler);
>>>> } else {
>>>> conHandler = (ServerConHandler) obj;
>>>> }
>>>> conHandler.read(workerThread.getByteBuffer());
>>>>
>>>> --> MySSLSelectorHandler does ...
>>>>
>>>> public void preSelect(Context ctx) throws IOException {
>>>> if (asyncQueueWriter == null) {
>>>> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
>>>> }
>>>> super.preSelect(ctx);
>>>> }
>>>>
>>>> --> ServerConHandler on receiving a write request does ...
>>>>
>>>> try {
>>>> MySSLAsyncQueueWriter writer =
>>>> (MySSLAsyncQueueWriter)
>>>> selectorHandler.getAsyncQueueWriter();
>>>> writer.write(selectionKey, ByteBuffer.wrap(b1),
>>>> sslEngine);
>>>> } catch (ClosedChannelException e) {
>>>> logger.error(e.getMessage());
>>>> } catch (IOException e) {
>>>> logger.error(e.getMessage());
>>>> }
>>>>
>>>> --> MySSLAsyncQueueWriter does ...
>>>>
>>>> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
>>>> sslEngine) throws IOException {
>>>> ChannelRegistrationRecord record =
>>>> channelRegistrationMap.putIfAbsent(key.channel(), new
>>>> ChannelRegistrationRecord(sslEngine));
>>>> if (record != null) {
>>>> record.writeCount.incrementAndGet();
>>>> }
>>>> super.write(key, buffer);
>>>> }
>>>>
>>>> protected void doWrite(WritableByteChannel channel, SocketAddress
>>>> dstAddress, ByteBuffer byteBuffer) throws IOException {
>>>> SSLEngine sslEngine;
>>>> ChannelRegistrationRecord record =
>>>> channelRegistrationMap.get(channel);
>>>> if (record == null) {
>>>> logger.error("Channel is not registered for an async
>>>> write");
>>>> return;
>>>> }
>>>> sslEngine = record.sslEngine;
>>>> if (sslEngine == null) {
>>>> logger.error("SSLEngine is null.");
>>>> return;
>>>> }
>>>> if (record.writeCount.intValue() > 200) {
>>>> ((SocketChannel)channel).socket().close();
>>>> channel.close();
>>>> channelRegistrationMap.remove(channel);
>>>> return;
>>>> }
>>>> if (record.writeCount.decrementAndGet() == 0) {
>>>> channelRegistrationMap.remove(channel);
>>>> }
>>>> SSLOutputWriter.flushChannel((SelectableChannel) channel,
>>>> byteBuffer,
>>>> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
>>>> sslEngine.getSession().getPacketBufferSize())), sslEngine);
>>>> }
>>>>
>>>>
>>>> The issues we are observing are ...
>>>>
>>>> --> As the number of concurrent connections shoots up to a number
>>>> above 200
>>>> the stress clients start facing disconnections and following
>>>> exceptions are
>>>> seen on the server side.
>>>>
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun
>>>> .nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
>>>> 125)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
>>>> 294)
>>>> at
>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>> 105)
>>>> at
>>>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:
>>>> 73)
>>>> at
>>>> com
>>>> .sun
>>>> .grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
>>>> 100)
>>>> at
>>>> com
>>>> .my
>>>> .server
>>>> .grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:
>>>> 70)
>>>> at
>>>> com
>>>> .sun
>>>> .grizzly
>>>> .async
>>>> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:
>>>> 374)
>>>> at
>>>> com
>>>> .sun
>>>> .grizzly
>>>> .async
>>>> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:
>>>> 291)
>>>> at
>>>> com
>>>> .sun
>>>> .grizzly
>>>> .async
>>>> .AsyncQueueWriterContextTask
>>>> .doCall(AsyncQueueWriterContextTask.java:
>>>> 86)
>>>> at
>>>> com
>>>> .sun
>>>> .grizzly
>>>> .SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
>>>> at
>>>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:
>>>> 169)
>>>>
>>>> --> The CPU load on the 4 CPU machine shoots up to a 100%
>>>> consistently.
>>>> --> The memory graph shows no decline.
>>>>
>>>> I think it must be something that I am doing wrong. Also, I wanted
>>>> to know
>>>> if Grizzly has been tested to scale up to 4000 ssl connections and
>>>> beyond if
>>>> required.
>>>>
>>>> Please feel free to revert in case you need some more information.
>>>>
>>>> Thanks & Regards,
>>>> Phani
>>>>
>>>>
>>>>
>>>> Oleksiy Stashok wrote:
>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>> 1) I think I should have given you a bit more info on this,
>>>>>> this is
>>>>>> what
>>>>>> happens on the server ...
>>>>>>
>>>>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>>>>> ByteBuffer
>>>>>> --> Server handler parses the ByteBuffer and hands it over to a
>>>>>> business
>>>>>> processing tier (using custom server threads).
>>>>>> --> Meanwhile, Grizzly context is recycled
>>>>>> --> A thread from the Business processing tier (BPT) digests the
>>>>>> message
>>>>>> asynchronously and produces a response
>>>>>> --> The BPT turns to Grizzly for handing over the response
>>>>>> bytebuffer in an
>>>>>> async write.
>>>>>> --> The processes of parsing messages on read byte buffers,
>>>>>> processing
>>>>>> parsed messages and writing them back to the network framework
>>>>>> are
>>>>>> independent of each other. Eg. the server could be parsing a
>>>>>> message
>>>>>> M2 in
>>>>>> one BPT while writing M1 in another BPT.
>>>>>>
>>>>>> Rephrasing the question, can I use the solution mentioned in
>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>> while preserving my server's luxury of writing back to grizzly
>>>>>> using a
>>>>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>>>>> without
>>>>>> suspending the reads and other business processing?
>>>>> Absolutely. Then, in your case I'm not sure you need to use
>>>>> SuspendFilter, as basically you don't want to suspend the
>>>>> connection -
>>>>> but keep it process other requests.
>>>>> Once your custom thread will complete the task execution - context
>>>>> will not be accessible anymore, as it was recycled by Grizzly
>>>>> worker
>>>>> thread, so you'll need to access async queue writer different way
>>>>> like:
>>>>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>>>>> byteBuffer);
>>>>>
>>>>>
>>>>>> 2) Yes, I did mean the re-handshaking. It is done for every
>>>>>> chunk of
>>>>>> the
>>>>>> bytearray written synchronously using
>>>>>> SSLOutputWriter.flushChannel()
>>>>>> but,
>>>>>> the same is not done if I try to write asynchronously using an
>>>>>> AsyncQueueWriter.write() with SSL as the final write is handled
>>>>>> by
>>>>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>>>>> WritePreprocessor and tries writing the resultant bytebuffer
>>>>>> contents
>>>>>> directly to the channel without any checks for re-handshaking. I
>>>>>> wanted to
>>>>>> know if it is left to the user to implement it in his/her own
>>>>>> code
>>>>>> for now?
>>>>> you mean this part of SSLOutputWriter.flushChannel()?
>>>>> //--------------- Cut --------------------------
>>>>> case OK:
>>>>> if (result.getHandshakeStatus() ==
>>>>> HandshakeStatus.NEED_TASK) {
>>>>> SSLUtils.executeDelegatedTask(sslEngine);
>>>>> }
>>>>> break;
>>>>> //----------------------------------------------
>>>>> If yes, can I ask you to add the same lines to your SSL
>>>>> preprocessor
>>>>> to check whether it works for you - if yes - we'll fix that in
>>>>> Grizzly
>>>>> examples.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> Phani
>>>>>>
>>>>>>
>>>>>> Oleksiy Stashok wrote:
>>>>>>>
>>>>>>> Hello Phani,
>>>>>>>
>>>>>>>> I am working on using Grizzly for our server application which
>>>>>>>> uses
>>>>>>>> SSLv3
>>>>>>>> and a custom binary application level protocol for client-
>>>>>>>> server
>>>>>>>> talk. My
>>>>>>>> server side implementation of Grizzly and the problems I am
>>>>>>>> facing
>>>>>>>> are
>>>>>>>> similar to the ones in the thread below ...
>>>>>>>>
>>>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>>>
>>>>>>>> However I have some doubts before I implement the solution
>>>>>>>> recommended in
>>>>>>>> the thread ...
>>>>>>>>
>>>>>>>> I am going to implement both Async reads and writes on the
>>>>>>>> server
>>>>>>>> side and
>>>>>>>> the problem with adopting the solution for this is that once I
>>>>>>>> suspend
>>>>>>>> execution of a ProtocolChain using the SuspendableFilter a
>>>>>>>> call to
>>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>>>>>> correct me
>>>>>>>> if I am wrong but, as a result of this reads won't happen on
>>>>>>>> the
>>>>>>>> channel. Is
>>>>>>>> there a way out for this? I do not want reads to be waiting on
>>>>>>>> writes
>>>>>>>> caused by the previous read. I want reads and writes to be
>>>>>>>> independent of
>>>>>>>> each other.
>>>>>>> You're right. In your case you don't need to have
>>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just
>>>>>>> let
>>>>>>> channel to be re-registered for OP_READ.
>>>>>>>
>>>>>>>
>>>>>>>> Also, I see that in case of Async writes for SSL / TLS the
>>>>>>>> final
>>>>>>>> channel
>>>>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>>>>> processor
>>>>>>>> to wrap
>>>>>>>> my bytes before sending them out but, the part to redo the
>>>>>>>> broken
>>>>>>>> handshakes
>>>>>>>> before sending every chunk of bytes out to the channel as in
>>>>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>>>>> TCPAsyncQueueWriter.
>>>>>>>> Please advice on how to work around this problem without
>>>>>>>> drifting
>>>>>>>> away from
>>>>>>>> the framework code and classes. For now, I am thinking of
>>>>>>>> implementing a new
>>>>>>>> SSLAsyncQueueWriter.
>>>>>>> Can you pls. provide details what part, you think, is missed in
>>>>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>>>>> If it's some common problem, we have to implement that in
>>>>>>> default
>>>>>>> Grizzly implementation and your help will be very appreciated!
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>> WBR,
>>>>>>> Alexey.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any help would be very much appreciated.
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> Phani
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>>>>> Sent from the Grizzly - Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>>> For additional commands, e-mail: users-
>>>>>>>> help_at_grizzly.dev.java.net
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18505329.html
> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>