7 ms·
The Blink Protocol
- shurcooL 14y agoWhat can be said about this?
- nanodeath 14y agoAs someone who knows little about Protocol Buffers, my first question was how does this compare to Protocol Buffers...which they say blogging about is "on their todo list". The protocol comes from Pantor Engineering, which apparently provides an "advanced trading system" out of Stockholm. Neat.
- afuruhed 14y agoHi, I'm going to update the FAQ. We have started comparing to GPB, please see here. http://blog.blinkprotocol.org/2013/01/blink-compared-to-google-protocol.html http://blog.blinkprotocol.org/2013/01/blink-compared-to-goog... As you can see in a comment by Rolf, part of the perf limitation of GPB is the implementation and not the wire protocol. Anders
- Inufu 14y agoIs there a reason why all the fields for protobuf are declared as "optional", when it looks like they are in fact not optional at all?
- bchallenor 14y agoTo support schema evolution. Of course it's up to you to write code that can actually handle missing fields, but once you have a required field in a message, you can never remove it. From the docs: "You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field – old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some engineers at Google have come to the conclusion that using required does more harm than good; they prefer to use only optional and repeated. However, this view is not universal." https://developers.google.com/protocol-buffers/docs/proto https://developers.google.com/protocol-buffers/docs/proto
- randersson 14y agogood question; I usually declare fields optional for the reason described in the reply above. I just re-ran the test with all fields declared as "required". It ran ~2% faster.
- yannis 14y agoGo straight to the tutorial http://blinkprotocol.org/s/tutorial.html http://blinkprotocol.org/s/tutorial.html, easier to understand what it is all about.
- mingpan 14y agoFeels kinda like a cross between ProtoBuf and MessagePack. We already have so many serialization formats, though. What is the compelling use-case for this one?
- cjensen 14y agoAnother binary message format? Another language for describing the format instead of XML or JSON? "Why?" is the big question which sits there unanswered.
- cbsmith 14y agoI'm with you on this. When Thrift came out, it kind of made sense, because there wasn't really anything quite like it (closest I'd say was Hessian) in terms of the mix of simplicity, performance, etc. When Protocol Buffers came out, it kind of made sense, because Google had worked on it prior to Thrift even existing (indeed, it inspired Thrift). When Avro came out, I was like, WTF!? This is making that mistake yet again!! EDIT: Damn, I forgot to mention MessagePack. ;-)
- DHowett 14y agoXML and JSON violate the following key property (which can be interpreted as a requirement): - Suitability for FPGA implementation
- cjensen 14y agoI meant the specification language, not the formatting,
- randersson 14y ago
- swalsh 14y agoWhy would I use this instead of ASN.1?
- codex 14y agoFrom the FAQ: "We did not know how to make use of ASN.1 without making Blink a lot more complex." This is the same reason Protocol Buffers exist: it was more fun to write something new than to understand how to use a better but more complex technology.
- cbsmith 14y agoI think you failed to grasp the point of that decision. It's not about understanding the more complex technology. At least in the case of protocol buffers, they understood ASN.1. Implementing all that complexity comes with a price. If nothing else, it takes time away from other priorities. If you look at protocol buffers, there are still ways to make the code faster, cleaner, etc., even without adding new features. Having worked with a variety of ASN.1 libraries in a variety of ways, I can tell you that in almost every case, I found either performance bugs, functional bugs, or just missing features that literally got in my way.
- andrewcooke 14y agoalso, the ASN.1 spec isn't free, which reduces adoption.
- bkirwi 14y agoGlanced through the spec and the comparison to protocol buffers on their blog. [0] A couple of quick notes: - The messages are prefixed with the message length; this allows has a variety of advantages (ie. splitting up messages with minimal parsing) but forces you to know your message size in advance. In the general case, this requires processing the full structure before the first byte can be written. (On the other hand, protocol buffers can be written incrementally, with a single pass over the data and using constant memory.) - Since the blink format contains no metadata, the recipient must have the schema with which the message was written. If schema migrations must be supported, this requires prefixing each batch of messages with the (encoded) schema. This may only be efficient when the batch size is large. (See Apache Avro [1] for another format that makes this tradeoff; since Avro is intended for processing in Hadoop, where the typical dataset has some millions of records, prefixing with the schema is not significant overhead.) [0] http://blog.blinkprotocol.org/2013/01/blink-compared-to-google-protocol.html http://blog.blinkprotocol.org/2013/01/blink-compared-to-goog... [1] http://avro.apache.org/docs/current/spec.html http://avro.apache.org/docs/current/spec.html
- badgar 14y ago> Since the blink format contains no metadata, the recipient must have the schema with which the message was written. If schema migrations must be supported, this requires prefixing each batch of messages with the (encoded) schema. This may only be efficient when the batch size is large. This is essentially a non-starter for me after a couple years of building distributed systems using protocol buffers, for which protocol evolution is the first feature.
- jamesaguilar 14y agoThey should just assign each schema a unique id, and have the server error out when receiving an ID it doesn't know. Then the client sends over the schema and the server caches it until death. This means the first request bears one additional round-trip, but the average request just carries an extra few bytes of identification.
- 14y ago
- deleted 14y ago[deleted]
- deleted 14y ago[deleted]
- vinkelhake 14y agoThis is by the guys who came up with FAST. That protocol is somewhat common in the dissemination of market data from exchanges. Pantor targets the financial sector and real-time trading systems in particular. It's a small company with smart people. http://www.fixprotocol.org/fast http://www.fixprotocol.org/fast
- tlrobinson 14y agoSuitability for FPGA implementation That's an interesting requirement you don't see very often.
- IvyMike 14y agoHigh Frequency Trading applications, maybe?
- georgeorwell 14y agoI'm curious, why do you have a 3-clause BSD-style license on some PDF files that contain specifications? I guess they are a kind of binary built from the page layout source code, but I haven't really seen this before. I'm not sure if you can license the specification itself, i.e. put conditions on people that want to implement it. http://blinkprotocol.org/s/specs.html http://blinkprotocol.org/s/specs.html
- jws 14y agoThere is a lawsuit related to the FAST protocol origination. They are certainly aware of this. Best to be explicit.
- alanctgardner2 14y agoI'm not an expert, but you can definitely restrict the documents which describe the standard. Now, if someone sits there and reverse-engineers the spec without ever seeing that doc, that might be tough to prevent. But generally you can apply a license to any creative work you want; this includes technical documents. A good (or bad, depending on how you view it) example is service manuals for laptops.
- randersson 14y agothe license is essentially a statement that the Blink specification is and will be entirely free to use. There has been a lot of discussion re lock-in and we wanted to be clear about our intentions. Implementors are obviously free to choose their own license terms and we may find that some will prefer to use GPL or LGPL, rather than BSD for contributed code.
- wavesounds 14y agoWhere does this fit on the network protocol stack? Does it replace TCP/UDP? Or is it on top of HTTP? Or is it not even a network protocol? I've skimmed the website but I'm surprised this isn't jumping right out at me tbh.
- wavesounds 14y ago"Blink can be used as an efficient presentation layer on top of middleware solutions like message queues and messaging platforms. However, Blink is also well-suited to be layered on TCP and UDP without any need for encapsulating headers, thanks to its built in message type system." How is it TCP without the headers? And how can you put a layer on top of TCP and call that saving space? Shouldn't you just be using TCP? It's late I must be missing something ...
- ekimekim 14y agoIt's a standard for encoding data over streams like TCP. If you recieve "00 4a 33 2b" over TCP, you have no idea what it means. But if you have a codebook or language defined to understand that data, you'll understand it means "Meet me on the first floor in half an hour". Blink is: - A meta-language for defining such languages - A specification of how messages in these languages are turned into binary data and back. When both client and server know the definitions (written in the meta-language), they can determine which kind of message the other side is using because of the metadata sent with the message (their "message type system") without the need for any extra information, like a HTTP content-type header.
- berlinbrown 14y agoInteresting. And it applies to anything?
- gdp 14y agoCould be useful for some things. I feel like it's a missed opportunity to employ somethings like session types, given the existence of a quasi-typed human-readable "schema" definition. It seems like one could (and should!) go further and make it an unambiguous protocol specification against which implementing programs could eventually be checked.
- afuruhed 14y agoGood point. Is it possible to create an unambiguous protocol specification that scales to more complex interactions? If so, and if you know how, I guess you would need a different specification language to get expressiveness? Blink has an extensible schema but that won't cut it. Anyhow, we have so far not even tried to do what you want but we hope that there is nothing in Blink that puts limitations on doing that separately. There is a lot more that we don't try to do in the protocol core.
- wrl 14y agoThis actually looks pretty cool. Disappointed to see that there's no reference implementation available as a library, however. Is there a plan to release one?
- afuruhed 14y agoThe (maybe only) good thing about there being no reference implementation is that we get feedback on the protocol itself and not the characteristics of a particular implementation. In this first phase, we are providing tools to ease independent implementation. We think it would be a sign of health that the specs are (become) good enough to create interoperable implementations from. We will make our plans known as they develop. We are doing more with the core protocol next.
- vinkelhake 14y agoOne thing I sorely missed when working with FAST was an official and comprehensive test suite. A wide set of templates and encoded files to exercise all the nooks and crannies of the spec would have been very useful. It looks like blink has far fewer special cases so the need for an official test suite might not be as great. Might be something worth considering anyway.
- randersson 14y agoagreed, a test suite will be essential for implementors. I'd be very interested to hear you thoughts on this. Please mail me if you have interest. Rolf
- dpakrk 14y agoDoes any binary message format (proto buffer , message pack , blink etc)support data streaming ? I as know none.