On 25/05/07, charlie hunt <charlie.hunt_at_sun.com> wrote:
> An approach that I find has been working very well on the reading side
> general approach where upon receiving a read event notification, read as
> much data as can be read into a ByteBuffer. Then, ask a message parser
> that knows how to parse the data just read into messages. As messages
> are parsed give those messages to a protocol processor. If you are left
> with a partial message as the last message in your ByteBuffer you
> continue to try to read more data. This is a condition I call,
> "expecting more data".
This is essentially what we do in Qpid, in MINA terminology this is
the "CumulativeProtocolDecoder".
In our broker we have certain cases where messages coming in can go
straight out to one or more consumers so we take care in that decoder
not to compact() the bytebyffer in the event you get a number of
unprocessed bytes at the end. Avoiding the compact means we can just
take a slice() of the data when handing it off to the next stage.
> As long as you are "expecting more data", you
> use a temporary Selector to wait for more data. When you are no longer
> "expecting more data", you can then consider being done with the overall
> read event.
What is the motivation for that? In Qpid, the SocketIOProcessor is
constantly constantly reading off the socket(s) it is responsible for.
What is the cost of creating a selector and tearing it down?
> There's some additional variations one can incorporate too
> such as ... distributed network applications tend to be bursty, for that
> reason you might consider adding to the definition "expecting more data"
> the notion of waiting a few more seconds before considering the overall
> read event being done.
Do you build anything in to ensure fairness/avoid starvation?
> The writing side is a little more interesting. One could consider
> putting outbound messages into a queue structure and have an writing
> thread waiting for data on the queue to be written and doing scatter
> writes when more than one entry is on the queue at a given time. This
> approach has its advantages and challenges, as you well know.
Yes, the queue approach is what we do in Qpid.
I actually spent a reasonable amount of time trying to get a
measurable improvement using scattering writes (and gathering reads
too) but I was unable to get any measurable improvement.
We found some interesting characteristics of the ConcurrentLinkedQueue
when looking at this. It didn't seem to perform too well when the
queue was often empty so in the end I believe we actually just used a
standard LinkedList with sychronized blocks around the accessors.
> And,
> there's also the approach that a thread that has formulated a response
> or constructed an outbound message simply just invokes the connection write.
Presumably it would then have to deal with issues like the kernel
buffer being full etc.
> Performance testing message brokers is a little different ;-)
> Throughput and scalability are both equally crucial.
Yes, and for some applications latency can be an issue and getting
that right can be a challenge. Other message brokers we tested came
close to Qpid in some throughput tests but had horrific latency.
One other thing we found with Qpid was that direct buffers were
significantly slower than heap buffers and that pooling buffers
(something that MINA can do) was counterproductive if you use heap
buffers. Do you use heap or direct buffers?
RG