4 ms·

Glanced through the spec and the comparison to protocol buffers on their blog. [0] A couple of quick notes: - The messages are prefixed with the message length

by bkirwi 14y ago

Glanced through the spec and the comparison to protocol buffers on their blog. [0] A couple of quick notes: - The messages are prefixed with the message length; this allows has a variety of advantages (ie. splitting up messages with minimal parsing) but forces you to know your message size in advance. In the general case, this requires processing the full structure before the first byte can be written. (On the other hand, protocol buffers can be written incrementally, with a single pass over the data and using constant memory.) - Since the blink format contains no metadata, the recipient must have the schema with which the message was written. If schema migrations must be supported, this requires prefixing each batch of messages with the (encoded) schema. This may only be efficient when the batch size is large. (See Apache Avro [1] for another format that makes this tradeoff; since Avro is intended for processing in Hadoop, where the typical dataset has some millions of records, prefixing with the schema is not significant overhead.) [0] http://blog.blinkprotocol.org/2013/01/blink-compared-to-google-protocol.html http://blog.blinkprotocol.org/2013/01/blink-compared-to-goog... [1] http://avro.apache.org/docs/current/spec.html http://avro.apache.org/docs/current/spec.html

badgar 14y ago
> Since the blink format contains no metadata, the recipient must have the schema with which the message was written. If schema migrations must be supported, this requires prefixing each batch of messages with the (encoded) schema. This may only be efficient when the batch size is large. This is essentially a non-starter for me after a couple years of building distributed systems using protocol buffers, for which protocol evolution is the first feature.
- jamesaguilar 14y ago
  They should just assign each schema a unique id, and have the server error out when receiving an ID it doesn't know. Then the client sends over the schema and the server caches it until death. This means the first request bears one additional round-trip, but the average request just carries an extra few bytes of identification.
  - afuruhed 14y ago
    Please have a look at the schema exchange specification. We welcome feedback. It is too costly to include meta data in every message. http://blinkprotocol.org/s/BlinkSchemaExchangeSpec-beta1.pdf http://blinkprotocol.org/s/BlinkSchemaExchangeSpec-beta1.pdf Anders
    - deleted 14y ago
      [deleted]
seliopou 14y ago
Avro as a data serialization format makes sense. As an RPC protocol, I think that dream died a long time ago. If you don't transmit the schema of the message, you're in the same spot as Thrift or Protocol Buffers. Which, then, leaves you with that eternal question: pourquoi, ets-ce que?
teraflop 14y ago
Also from the protocol buffer comparison: "Finally, Blink defines a method of transferring a serialized schema that can be used to notify a receiver of a Blink message stream of the structure of the transferred messages. A similar method could be added on top of GPB, but would again be non-standard." I think this is incorrect; the protobuf library includes exactly the same thing, unless I'm misunderstanding what the Blink authors are talking about. protoc can turn a textual schema into a protobuf-serialized message that describes it, which a receiver can use to decode messages of that type with no prior information: https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.proto https://code.google.com/p/protobuf/source/browse/trunk/src/g... https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/DynamicMessage https://developers.google.com/protocol-buffers/docs/referenc...
- randersson 14y ago
  I was maybe a little terse in my blog post. I agree that you can certainly do this, but it seems G decided that it was not worth adding as a core feature. At least that is my understanding from reading their docs. Blink on the other hand relies heavily on schema exchange. Rolf
eckzow 14y ago
In re your first bullet: If such things are important to you, it looks like you can produce fixed size packets through a variety of methods. In particular, they support overlength encodings.
- bkirwi 14y ago
  True; but it's hard to support a fixed-length encoding when a message can contain nested collections of arbitrary length.
  - drosenborg 14y ago
    The just released Blink Native binary format might be an alternative if you want a fixed-length encoding: http://blog.blinkprotocol.org/2013/01/blink-native-format-released.html http://blog.blinkprotocol.org/2013/01/blink-native-format-re...
tripzilch 14y ago
There was this recent security presentation about formal language-theoretic reasons why message-length prefixes are a bad idea. http://www.cs.dartmouth.edu/~sergey/langsec/ http://www.cs.dartmouth.edu/~sergey/langsec/ The problem was that it causes unnecessary complexity in validation and parsing, as <length-of-message><message>... describes a context-sensitive grammar, while a more XML-, JSON-like syntax with delimiters <delim><message></delim>... is context-free. The complexity of proper CSG validation can be exploited for resource-depletion/DOS attacks, while not validating the syntax of a message leaves part of the program that uses the data in these messages with potentially invalid data, putting it into an inconsistent state, possibly leading to DOS or worse exploits. At least that's what I got from their presentation, which was very interesting BTW, especially the part about "weird machines"--that's basically the inconsistent state, the emergent machine that is made out of all the things an attacker can make a system do by exploiting it. It was also extremely geeky. It just left me wondering: if I understood this right, every protocol that uses message-length prefixes is at least vulnerable to resource-depletion attacks, and there's no way of solving this short of proving P=NP. But that's quite a lot of protocols! Why isn't the Internet burning down as we speak?
- m_eiman 14y ago
  if ( msg.total_length > MAX_ALLOWED_SIZE ) { reset_parser(); }? Should take care of any resource depletion issues, and do it way faster than eating data waiting for an end of field marker would.
  - mcherm 14y ago
    But it produces a protocol which is incapable of handling messages greater than a certain length. The marketplace is littered with such protocols, but they're all useless since they can no longer even transmit a single wordprocessing document now that a page of text takes up megabytes of overhead.
    - m_eiman 14y ago
      It's not every protocol that must handle arbitrary messages, that doesn't make them useless. Having a 9600 baud half duplex connection for messaging that requires low latency is quite different from 1Gbps Ethernet, and that makes every byte count.

by bkirwi 14y ago

badgar 14y ago

> Since the blink format contains no metadata, the recipient must have the schema with which the message was written. If schema migrations must be supported, this requires prefixing each batch of messages with the (encoded) schema. This may only be efficient when the batch size is large. This is essentially a non-starter for me after a couple years of building distributed systems using protocol buffers, for which protocol evolution is the first feature.

jamesaguilar 14y ago
They should just assign each schema a unique id, and have the server error out when receiving an ID it doesn't know. Then the client sends over the schema and the server caches it until death. This means the first request bears one additional round-trip, but the average request just carries an extra few bytes of identification.
- afuruhed 14y ago
  Please have a look at the schema exchange specification. We welcome feedback. It is too costly to include meta data in every message. http://blinkprotocol.org/s/BlinkSchemaExchangeSpec-beta1.pdf http://blinkprotocol.org/s/BlinkSchemaExchangeSpec-beta1.pdf Anders
  - deleted 14y ago
    [deleted]

seliopou 14y ago

Avro as a data serialization format makes sense. As an RPC protocol, I think that dream died a long time ago. If you don't transmit the schema of the message, you're in the same spot as Thrift or Protocol Buffers. Which, then, leaves you with that eternal question: pourquoi, ets-ce que?

teraflop 14y ago

Also from the protocol buffer comparison: "Finally, Blink defines a method of transferring a serialized schema that can be used to notify a receiver of a Blink message stream of the structure of the transferred messages. A similar method could be added on top of GPB, but would again be non-standard." I think this is incorrect; the protobuf library includes exactly the same thing, unless I'm misunderstanding what the Blink authors are talking about. protoc can turn a textual schema into a protobuf-serialized message that describes it, which a receiver can use to decode messages of that type with no prior information: https://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.proto https://code.google.com/p/protobuf/source/browse/trunk/src/g... https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/DynamicMessage https://developers.google.com/protocol-buffers/docs/referenc...

randersson 14y ago
I was maybe a little terse in my blog post. I agree that you can certainly do this, but it seems G decided that it was not worth adding as a core feature. At least that is my understanding from reading their docs. Blink on the other hand relies heavily on schema exchange. Rolf

eckzow 14y ago

In re your first bullet: If such things are important to you, it looks like you can produce fixed size packets through a variety of methods. In particular, they support overlength encodings.

bkirwi 14y ago
True; but it's hard to support a fixed-length encoding when a message can contain nested collections of arbitrary length.
- drosenborg 14y ago
  The just released Blink Native binary format might be an alternative if you want a fixed-length encoding: http://blog.blinkprotocol.org/2013/01/blink-native-format-released.html http://blog.blinkprotocol.org/2013/01/blink-native-format-re...

tripzilch 14y ago

There was this recent security presentation about formal language-theoretic reasons why message-length prefixes are a bad idea. http://www.cs.dartmouth.edu/~sergey/langsec/ http://www.cs.dartmouth.edu/~sergey/langsec/ The problem was that it causes unnecessary complexity in validation and parsing, as <length-of-message><message>... describes a context-sensitive grammar, while a more XML-, JSON-like syntax with delimiters <delim><message></delim>... is context-free. The complexity of proper CSG validation can be exploited for resource-depletion/DOS attacks, while not validating the syntax of a message leaves part of the program that uses the data in these messages with potentially invalid data, putting it into an inconsistent state, possibly leading to DOS or worse exploits. At least that's what I got from their presentation, which was very interesting BTW, especially the part about "weird machines"--that's basically the inconsistent state, the emergent machine that is made out of all the things an attacker can make a system do by exploiting it. It was also extremely geeky. It just left me wondering: if I understood this right, every protocol that uses message-length prefixes is at least vulnerable to resource-depletion attacks, and there's no way of solving this short of proving P=NP. But that's quite a lot of protocols! Why isn't the Internet burning down as we speak?

m_eiman 14y ago
if ( msg.total_length > MAX_ALLOWED_SIZE ) { reset_parser(); }? Should take care of any resource depletion issues, and do it way faster than eating data waiting for an end of field marker would.
- mcherm 14y ago
  But it produces a protocol which is incapable of handling messages greater than a certain length. The marketplace is littered with such protocols, but they're all useless since they can no longer even transmit a single wordprocessing document now that a page of text takes up megabytes of overhead.
  - m_eiman 14y ago
    It's not every protocol that must handle arbitrary messages, that doesn't make them useless. Having a 9600 baud half duplex connection for messaging that requires low latency is quite different from 1Gbps Ethernet, and that makes every byte count.