5 ms·

There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://dev

There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers

rewq4321 5y ago
And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...
blacklion 5y ago
"old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.
- blibble 5y ago
  there's still ascii FIX floating around on the market data side for a few esoteric venues FAST is not particularly common in Europe the large European venues use fixed-width binary encoding (LSE group, Euronext, CBOE Europe)
  - blacklion 5y ago
    Eurex uses FAST for sure, but I can be wrong about "common".
    - gpderetta 5y ago
      Eurex/XETRA EOBI no longer uses FAST. It is about 8 year old t this point, but IIRC some products are still on the older protocol.
  - nly 5y ago
    Euronext uses SBE specifically.

There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers

rewq4321 5y ago

And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...

blacklion 5y ago

"old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.

blibble 5y ago
there's still ascii FIX floating around on the market data side for a few esoteric venues FAST is not particularly common in Europe the large European venues use fixed-width binary encoding (LSE group, Euronext, CBOE Europe)
- blacklion 5y ago
  Eurex uses FAST for sure, but I can be wrong about "common".
  - gpderetta 5y ago
    Eurex/XETRA EOBI no longer uses FAST. It is about 8 year old t this point, but IIRC some products are still on the older protocol.
- nly 5y ago
  Euronext uses SBE specifically.