On Thu, 2011-10-06 at 12:04 +1100, Greg Wilkins wrote:
> I agree with Remy about his concerns regarding the unlimited message
> size in websockets. There are two compelling features of
> websockets: 1) bidirectional 2) message based. It is a great pity
> that potentially large size of WS messages might push us towards
> streaming APIs when the vast majority of currently conceived usage
> will be small/medium messages that will easily fit in buffers.
I agree with that.
Websockets messages can be made of any number of frames. So what's the
point with having frames be unlimited in size ?
> Note also that the browser implementations do not support either frame
> or streamed based APIs and they limit their frame size (and I suspect
> message size) to <16MB.
> http://en.wikipedia.org/wiki/Comparison_of_WebSocket_implementations
>
> So webapps are not going to see unlimited messages any time soon and
> it would be a pity to come up with a more complex than needed API
> because of these non existent messages. Having said that, 16MB
> buffers are still significant on a server and we may need to have
> limits anyway.
Thanks for the link. I read there Jetty supports the full frame size
with your buffer based API, is that really accurate ?
Realistically, we're not going to be able to arbitrarily limit this
here, it was ok for early adopters impl, but moving to production it is
not. 16MB is way way too big to buffer and scale already anyway IMO.
> By giving a streaming interface to the application, you are just going
> to make the application responsible for an extra layer of buffering
> and for policing any resource limits. Applications will often have
> no idea (or more importantly, often no care) about resource limits
> and I suspect that the vast majority of implementations would just
> stream the messages into unlimited StringBuilders. With streaming we
> would have many many applications vulnerable to simple DOS attacks.
> Also note that if limits are exceeded, then the WS connection needs to
> be closed with the correct error codes, else clients are likely to
> just try again.
>
> It is the job of the container to protect applications from these
> kinds of concerns - even if it means that we have to impose some
> limits. We already do this with HTTP, where the server imposes a
> maximum size on the URI, the headers and often on POST form content.
> Just because the HTTP protocol does not limit the size of URIs,
> headers or form content does not mean that we went to a stream API for
> getPathInfo(), getHeader(String) and getParameter(String).
>
> So I strongly believe that we should offer a simple message based API
> (byte[] and String), with limits imposed and enforced by the
> container, which will run the correct error conversations and any
> negotiations as needed.
Usually when a frame is that large, it is because the application is
about file transfer or media content distribution. In either case, the
application does not need a full frame to do something. A buffer API is
not protective, it will only waste a lot of memory in many cases.
So I don't see how to avoid using streams ... Unfortunately, that's
where the complexity kicks in to have non blocking.
The protocol is not final yet, we can still hope for some unexpected
reversal, and they suddenly drop the 64bit size for frames (effectively
limiting them to 16bit, which would make a buffer API great). The
application would only need to read the frames until it reaches the end
of the message if it needs to buffer a full message before parsing, that
looks easy to do.
For API design options, assuming the 64bit limit stays, we could use a
pure stream based API, with non blocking flags like Rajiv's first
proposal. This would make it look a bit similar to classic Servlet, with
added controls for frame fragmentation on output. It is also possible to
have a companion buffer (ByteBuffer and CharBuffer please, I'd like to
avoid straight byte[] and Strings to be honest) based API which would be
used when the frames have a 16bit length, it could help a lot folks who
design protocols that only use this type of frame (essentially rewarding
them for being reasonable).
--
Remy Maucherat <rmaucher_at_redhat.com>
Red Hat Inc