Hi Alexey,
Please find the source files in the zip file attached.
http://www.nabble.com/file/p18503031/src.zip src.zip
Regards,
Phani
Oleksiy Stashok wrote:
>
> Hello Phani,
>
> can you pls. send the complete java sources of Grizzly related
> classes, it could be easier to read the code.
>
> Thanks.
>
> WBR,
> Alexey.
>
> On Jul 16, 2008, at 8:55 , PhaKuDi wrote:
>
>>
>> Hello Alexey,
>>
>> My apologies for the delay in responding. Regarding ...
>>
>> 1) Thanks for the confirmation. Getting hold of async writer from the
>> selector handler is the right way to go for me.
>> 2) Just using a SSLWritePreprocessor isn't helping me. Have tested
>> with the
>> re-handshaking code and it works for me.
>>
>> Also, now that I am stress testing the server I am running into a few
>> problems. I would very much appreciate your suggestions/comments on
>> these
>> issues. The server in question currently is capable of catering to
>> about
>> 4000 clients during peak times and uses a C library for SSL
>> communication
>> with the clients. We are in the process of moving it to an NIO based
>> framework with minimal impact on existing design. My implementation of
>> Grizzly is as below ...
>>
>> --> Server initalizes the controller as follows ...
>>
>> SSLReadFilter sslReadFilter = new
>> MySSLReadFilter(gameServer,
>> USE_GRIZZLY_ASYNC_READS);
>> sslReadFilter.setSSLContext(context);
>> final ProtocolChain protocolChain = new
>> DefaultProtocolChain();
>> protocolChain.addFilter(sslReadFilter);
>> ProtocolChainInstanceHandler chainInstanceHandler = new
>> DefaultProtocolChainInstanceHandler() {
>> public ProtocolChain poll() {
>> return protocolChain;
>> }
>> };
>> Controller controller = new Controller();
>> controller.setHandleReadWriteConcurrently(true);
>> SSLSelectorHandler selectorHandler = new
>> MySSLSelectorHandler();
>> selectorHandler.setPort(port);
>> controller.setSelectorHandler(selectorHandler);
>>
>> controller.setProtocolChainInstanceHandler(chainInstanceHandler);
>>
>> controller.setReadThreadsCount(GRIZZLY_READ_THREADS_COUNT); //
>> Currently set to 0
>> OutputWriter.setDefaultWriteTimeout(5000);
>> controller.start();
>>
>> --> MySSLReadFilter.execute() method does ...
>>
>> super.execute(context);
>> engine = workerThread.getSSLEngine();
>> session = engine.getSession();
>> Object obj =
>> session.getValue(ServerConHandler.class.getName());
>> ServerConHandler conHandler;
>> if (obj == null || !(obj instanceof ServerConHandler)) {
>> SSLSelectorHandler selectorHandler = (SSLSelectorHandler)
>> context.getSelectorHandler();
>> conHandler = new ServerConHandler(selectorHandler,
>> selectionKey,
>> (SocketChannel) selectionKey.channel(), engine, user);
>> session.putValue(ServerConHandler.class.getName(),
>> conHandler);
>> } else {
>> conHandler = (ServerConHandler) obj;
>> }
>> conHandler.read(workerThread.getByteBuffer());
>>
>> --> MySSLSelectorHandler does ...
>>
>> public void preSelect(Context ctx) throws IOException {
>> if (asyncQueueWriter == null) {
>> asyncQueueWriter = new MySSLAsyncQueueWriter(this);
>> }
>> super.preSelect(ctx);
>> }
>>
>> --> ServerConHandler on receiving a write request does ...
>>
>> try {
>> MySSLAsyncQueueWriter writer =
>> (MySSLAsyncQueueWriter)
>> selectorHandler.getAsyncQueueWriter();
>> writer.write(selectionKey, ByteBuffer.wrap(b1),
>> sslEngine);
>> } catch (ClosedChannelException e) {
>> logger.error(e.getMessage());
>> } catch (IOException e) {
>> logger.error(e.getMessage());
>> }
>>
>> --> MySSLAsyncQueueWriter does ...
>>
>> public void write(SelectionKey key, ByteBuffer buffer, SSLEngine
>> sslEngine) throws IOException {
>> ChannelRegistrationRecord record =
>> channelRegistrationMap.putIfAbsent(key.channel(), new
>> ChannelRegistrationRecord(sslEngine));
>> if (record != null) {
>> record.writeCount.incrementAndGet();
>> }
>> super.write(key, buffer);
>> }
>>
>> protected void doWrite(WritableByteChannel channel, SocketAddress
>> dstAddress, ByteBuffer byteBuffer) throws IOException {
>> SSLEngine sslEngine;
>> ChannelRegistrationRecord record =
>> channelRegistrationMap.get(channel);
>> if (record == null) {
>> logger.error("Channel is not registered for an async
>> write");
>> return;
>> }
>> sslEngine = record.sslEngine;
>> if (sslEngine == null) {
>> logger.error("SSLEngine is null.");
>> return;
>> }
>> if (record.writeCount.intValue() > 200) {
>> ((SocketChannel)channel).socket().close();
>> channel.close();
>> channelRegistrationMap.remove(channel);
>> return;
>> }
>> if (record.writeCount.decrementAndGet() == 0) {
>> channelRegistrationMap.remove(channel);
>> }
>> SSLOutputWriter.flushChannel((SelectableChannel) channel,
>> byteBuffer,
>> ByteBuffer.allocate(Math.max(GRIZZLY_DEFAULT_OUTPUT_BB_SIZE,
>> sslEngine.getSession().getPacketBufferSize())), sslEngine);
>> }
>>
>>
>> The issues we are observing are ...
>>
>> --> As the number of concurrent connections shoots up to a number
>> above 200
>> the stress clients start facing disconnections and following
>> exceptions are
>> seen on the server side.
>>
>> java.nio.channels.ClosedChannelException
>> at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:
>> 125)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:
>> 294)
>> at
>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:105)
>> at
>> com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:73)
>> at
>> com
>> .sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:
>> 100)
>> at
>> com
>> .my
>> .server
>> .grizzly.MySSLAsyncQueueWriter.doWrite(MySSLAsyncQueueWriter.java:70)
>> at
>> com
>> .sun
>> .grizzly
>> .async
>> .AbstractAsyncQueueWriter.doWrite(AbstractAsyncQueueWriter.java:374)
>> at
>> com
>> .sun
>> .grizzly
>> .async
>> .AbstractAsyncQueueWriter.onWrite(AbstractAsyncQueueWriter.java:291)
>> at
>> com
>> .sun
>> .grizzly
>> .async
>> .AsyncQueueWriterContextTask.doCall(AsyncQueueWriterContextTask.java:
>> 86)
>> at
>> com
>> .sun
>> .grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
>> at
>> com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:169)
>>
>> --> The CPU load on the 4 CPU machine shoots up to a 100%
>> consistently.
>> --> The memory graph shows no decline.
>>
>> I think it must be something that I am doing wrong. Also, I wanted
>> to know
>> if Grizzly has been tested to scale up to 4000 ssl connections and
>> beyond if
>> required.
>>
>> Please feel free to revert in case you need some more information.
>>
>> Thanks & Regards,
>> Phani
>>
>>
>>
>> Oleksiy Stashok wrote:
>>>
>>>>
>>>
>>>
>>>> 1) I think I should have given you a bit more info on this, this is
>>>> what
>>>> happens on the server ...
>>>>
>>>> --> Grizzly's SSLReadFilter provides my server Handler with a
>>>> ByteBuffer
>>>> --> Server handler parses the ByteBuffer and hands it over to a
>>>> business
>>>> processing tier (using custom server threads).
>>>> --> Meanwhile, Grizzly context is recycled
>>>> --> A thread from the Business processing tier (BPT) digests the
>>>> message
>>>> asynchronously and produces a response
>>>> --> The BPT turns to Grizzly for handing over the response
>>>> bytebuffer in an
>>>> async write.
>>>> --> The processes of parsing messages on read byte buffers,
>>>> processing
>>>> parsed messages and writing them back to the network framework are
>>>> independent of each other. Eg. the server could be parsing a message
>>>> M2 in
>>>> one BPT while writing M1 in another BPT.
>>>>
>>>> Rephrasing the question, can I use the solution mentioned in
>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>> while preserving my server's luxury of writing back to grizzly
>>>> using a
>>>> context.getAsyncQueueWriter.write(byteBuffer) at its own time
>>>> without
>>>> suspending the reads and other business processing?
>>> Absolutely. Then, in your case I'm not sure you need to use
>>> SuspendFilter, as basically you don't want to suspend the
>>> connection -
>>> but keep it process other requests.
>>> Once your custom thread will complete the task execution - context
>>> will not be accessible anymore, as it was recycled by Grizzly worker
>>> thread, so you'll need to access async queue writer different way
>>> like:
>>> tcpSelectorHandler.getAsyncQueueWriter().write(SelectionKey,
>>> byteBuffer);
>>>
>>>
>>>> 2) Yes, I did mean the re-handshaking. It is done for every chunk of
>>>> the
>>>> bytearray written synchronously using SSLOutputWriter.flushChannel()
>>>> but,
>>>> the same is not done if I try to write asynchronously using an
>>>> AsyncQueueWriter.write() with SSL as the final write is handled by
>>>> TCPAsyncQueueWriter which passes the bytebuffer through the
>>>> WritePreprocessor and tries writing the resultant bytebuffer
>>>> contents
>>>> directly to the channel without any checks for re-handshaking. I
>>>> wanted to
>>>> know if it is left to the user to implement it in his/her own code
>>>> for now?
>>> you mean this part of SSLOutputWriter.flushChannel()?
>>> //--------------- Cut --------------------------
>>> case OK:
>>> if (result.getHandshakeStatus() ==
>>> HandshakeStatus.NEED_TASK) {
>>> SSLUtils.executeDelegatedTask(sslEngine);
>>> }
>>> break;
>>> //----------------------------------------------
>>> If yes, can I ask you to add the same lines to your SSL preprocessor
>>> to check whether it works for you - if yes - we'll fix that in
>>> Grizzly
>>> examples.
>>>
>>> Thanks.
>>>
>>> WBR,
>>> Alexey.
>>>
>>>>
>>>>
>>>> Thanks & Regards,
>>>> Phani
>>>>
>>>>
>>>> Oleksiy Stashok wrote:
>>>>>
>>>>> Hello Phani,
>>>>>
>>>>>> I am working on using Grizzly for our server application which
>>>>>> uses
>>>>>> SSLv3
>>>>>> and a custom binary application level protocol for client-server
>>>>>> talk. My
>>>>>> server side implementation of Grizzly and the problems I am facing
>>>>>> are
>>>>>> similar to the ones in the thread below ...
>>>>>>
>>>>>> http://www.nabble.com/Asynchronous-Request-Processing-with-TCPIP-td18234121.html
>>>>>>
>>>>>> However I have some doubts before I implement the solution
>>>>>> recommended in
>>>>>> the thread ...
>>>>>>
>>>>>> I am going to implement both Async reads and writes on the server
>>>>>> side and
>>>>>> the problem with adopting the solution for this is that once I
>>>>>> suspend
>>>>>> execution of a ProtocolChain using the SuspendableFilter a call to
>>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE) is made,
>>>>>> correct me
>>>>>> if I am wrong but, as a result of this reads won't happen on the
>>>>>> channel. Is
>>>>>> there a way out for this? I do not want reads to be waiting on
>>>>>> writes
>>>>>> caused by the previous read. I want reads and writes to be
>>>>>> independent of
>>>>>> each other.
>>>>> You're right. In your case you don't need to have
>>>>> ctx.setKeyRegistrationState(KeyRegistrationState.NONE), but just
>>>>> let
>>>>> channel to be re-registered for OP_READ.
>>>>>
>>>>>
>>>>>> Also, I see that in case of Async writes for SSL / TLS the final
>>>>>> channel
>>>>>> write happens in TCPAsyncQueueWriter.doWrite() using
>>>>>> channel.write(byteBuffer) call. I am supplying a write pre
>>>>>> processor
>>>>>> to wrap
>>>>>> my bytes before sending them out but, the part to redo the broken
>>>>>> handshakes
>>>>>> before sending every chunk of bytes out to the channel as in
>>>>>> SSLOutputWriter.flushChannel() will be missing in
>>>>>> TCPAsyncQueueWriter.
>>>>>> Please advice on how to work around this problem without drifting
>>>>>> away from
>>>>>> the framework code and classes. For now, I am thinking of
>>>>>> implementing a new
>>>>>> SSLAsyncQueueWriter.
>>>>> Can you pls. provide details what part, you think, is missed in
>>>>> AsyncWrite SSL preprocessor? You meant re-handshaking?
>>>>> If it's some common problem, we have to implement that in default
>>>>> Grizzly implementation and your help will be very appreciated!
>>>>>
>>>>> Thank you.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>>
>>>>>>
>>>>>> Any help would be very much appreciated.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> Phani
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/SSL-ARP-tp18381476p18381476.html
>>>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/SSL-ARP-tp18381476p18382989.html
>>>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SSL-ARP-tp18381476p18481339.html
>> Sent from the Grizzly - Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
>> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe_at_grizzly.dev.java.net
> For additional commands, e-mail: users-help_at_grizzly.dev.java.net
>
>
>
--
View this message in context: http://www.nabble.com/SSL-ARP-tp18381476p18503031.html
Sent from the Grizzly - Users mailing list archive at Nabble.com.