6 ms·

Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto

Crypto markets are very small :-) I'm working and company which process "real" exchanges, like NASDAQ, LSE, and, especially, OPRA feed. We've added 20+ crypto exchanges in our portfolio this year, and all of them are processed on one old server which is unable to process NASDAQ Total View in real-time anymore. On the other hand, whole OPRA feed (more than 5Gbit/s or 65B/day, yes, it is billions, messages of very optimized binary protocol, not this crappy JSON) is processed by our code on one modern server. Nothing special, two sockets of Intel Xeons (not even Platinums).

joering2 5y ago
Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.
- paraph1n 5y ago
  There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers
  - rewq4321 5y ago
    And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...
  - blacklion 5y ago
    "old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.
    - blibble 5y ago
      there's still ascii FIX floating around on the market data side for a few esoteric venues FAST is not particularly common in Europe the large European venues use fixed-width binary encoding (LSE group, Euronext, CBOE Europe)
      - blacklion 5y ago
        Eurex uses FAST for sure, but I can be wrong about "common".
        gpderetta 5y ago
        Eurex/XETRA EOBI no longer uses FAST. It is about 8 year old t this point, but IIRC some products are still on the older protocol.
      - nly 5y ago
        Euronext uses SBE specifically.
- wffurr 5y ago
  https://www.opraplan.com/datafeeds https://www.opraplan.com/datafeeds
- nostrademons 5y ago
  Some Googling turned up this protocol descriptor: https://uploads-ssl.webflow.com/5ba40927ac854d8c97bc92d7/5bf4197268f8b277dbb7c12d_opra_output_binary_dr_spec.pdf https://uploads-ssl.webflow.com/5ba40927ac854d8c97bc92d7/5bf... If you're optimizing for latency JSON is pretty terrible, but most people who use it are optimizing for interoperability and ease of development. It works just fine for that, and you can recover decent bandwidth just by compressing it.
- blacklion 5y ago
  It is not "our" protocol, it is protocol designed by exchange and we need to support it, as we can not change it :). Simple binary messages, with binary encoded numbers, etc. No string parsing, no syntax, nothing like this, only bytes and offsets. Think about TCP header, for example. JSON is very inefficient both in bytes (32 bit price is 4 bytes in binary and could be 7+ bytes as string, think "1299.99" for example) and CPU: to parse "1299.99" you need burn a lot of cycles, and if it is number of cents stored as native 4-byte number you need 3 shifts and 4 binary ors at most, if you need to change endianness, and in most cases it is simple memory copy of 4 bytes, 1-2 CPU cycle. When you have binary protocol, you could skip fields which you are not interested in as simple as "offset = offset + <filed-size>" (where <filed-size> is compile-time constant!) and in JSON you need to parse whole thing anyway. Difference between converting binary packet to internal data structure and parsing JSON with same data to same structure could be ten-fold easily, and you need to be very creative to parse JSON without additional memory allocations (it is possible, but code becomes very dirty and fragile), and memory allocation and/or deallocation costs a lot, both in GC languages and languages with manual memory management.
  - joering2 5y ago
    Thanks for the writeup!
  - localhost 5y ago
    Curious if the binary protocol uses floating point or a fixed point representation? Or is floating point with its rounding issues sufficient for the protocol's needs?
    - andylynch 5y ago
      No GP but familiar with these protocols. They use fixed point extensively; I can't even thing of an exchange protocol which would use floating point since the rounding issues would cause unnecessary and expensive problems. This is typical (from NASDAQ http://www.nasdaqtrader.com/content/technicalsupport/specifications/TradingProducts/OUCH4.2.pdf http://www.nasdaqtrader.com/content/technicalsupport/specifi... ): Prices are integer fields. When converted to a decimal format, prices are in fixed point format with 6 whole number places followed by 4 decimal digits. The maximum price in OUCH 4.2 is $199,999.9900 (decimal, 7735939C hex). When entering market orders for a cross, use the special price of $214,748.3647 (decimal, 7FFFFFFF hex).
- andylynch 5y ago
  Have a look at NASDAQ ITCH, OUCH, and RASH (these are the real names, the story I heard is the original author of didn't like the usual corporate style brand names and wanted certain people to squirm when talking about them).
deleted 5y ago
[deleted]
Foomf 5y ago
I've read your few posts a few times and I'm still not sure why you made your post. You're telling the person that you handle more data than them and thus need more resources than them. Was your goal to smugly belittle them? It's not like they said any problem can be solved on their specific resources.
- Aeolun 5y ago
  I feel like he’s saying two things. One, that he’s surprised by how small crypto markets are. Two, that this one server (or very few server processing) thing scales quite well to billions of messages a day. I didn’t find any element of smugness here, but maybe I misread the tone.
- lost-found 5y ago
  They are just pointing out the size of crypto market data vs normal market data. I found it interesting, not their fault you didn’t.
- blacklion 5y ago
  Nope, I want to say, that even much more data could be processed on very limited hardware, and that it is additional confirmation that current hardware is immensely powerful and very under-estimated.
  - Cthulhu_ 5y ago
    And that go-to solutions / golden hammers like REST / JSON are very much suboptimal. I mean I'm still going to use it for client/server communication and the like because I don't have serious performance constraints enough to warrant something that will be more difficult to develop for etc, but still.
- nostrademons 5y ago
  I'd initially responded, but I found that my response had an element of a pissing match over whose data is bigger, so I deleted it. The thing is - when two engineers get smug, oftentimes lots of fairly interesting technical details get exchanged, so such discussions aren't really useless to bystanders.
secondcoming 5y ago
It depends on what your server does with each request though; '65B a day' means little. If all it does is write it to a log then I'm surprised you're not using a rPI.

joering2 5y ago

Could you share some more about that very optimized binary protocol? I know there are ways to be more efficient than JSON but since you call it crappy, your solution must be much much better. Honestly interested to readup more.

paraph1n 5y ago
There are many binary encoding protocols. A popular one is protobufs[1], which is used by gRPC. [1]: https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers
- rewq4321 5y ago
  And msgpack if you want an order of magnitude faster serialization/deserialisation and can put up with worse compression (I think mainly due to schema overhead since protobuf files don't store the schema?) https://msgpack.org/index.html https://msgpack.org/index.html Good protobuf vs msgpack comparison: https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4 https://medium.com/@hugovs/the-need-for-speed-experimenting-...
- blacklion 5y ago
  "old skool" exchanges uses either FIX (old and really vernose), FAST (binary encoding for FIX) or custom fixed-layout protocols. Most big USA exchanges uses custom fixed-layout protocols, where each message is described in documentation, but not in machine-readable way. European ones still use FAST. I didn't seen FIX in the wild for data feeds, but it is used for brokers, to submit orders to exchange (our company didn't do this part, we only consume feeds). I don't know why, but all Crypto Exchanges use JSON, not protobufs or something like this, and didn't publish any formal schemes. Fun fact: one crypto exchange put GZIP'ed and base64'ed JSON data into JSON which pushed to websocket, to save bandwidth. IMHO, it is peak of bad design.
  - blibble 5y ago
    there's still ascii FIX floating around on the market data side for a few esoteric venues FAST is not particularly common in Europe the large European venues use fixed-width binary encoding (LSE group, Euronext, CBOE Europe)
    - blacklion 5y ago
      Eurex uses FAST for sure, but I can be wrong about "common".
      - gpderetta 5y ago
        Eurex/XETRA EOBI no longer uses FAST. It is about 8 year old t this point, but IIRC some products are still on the older protocol.
    - nly 5y ago
      Euronext uses SBE specifically.
wffurr 5y ago
https://www.opraplan.com/datafeeds https://www.opraplan.com/datafeeds
nostrademons 5y ago
Some Googling turned up this protocol descriptor: https://uploads-ssl.webflow.com/5ba40927ac854d8c97bc92d7/5bf4197268f8b277dbb7c12d_opra_output_binary_dr_spec.pdf https://uploads-ssl.webflow.com/5ba40927ac854d8c97bc92d7/5bf... If you're optimizing for latency JSON is pretty terrible, but most people who use it are optimizing for interoperability and ease of development. It works just fine for that, and you can recover decent bandwidth just by compressing it.
blacklion 5y ago
It is not "our" protocol, it is protocol designed by exchange and we need to support it, as we can not change it :). Simple binary messages, with binary encoded numbers, etc. No string parsing, no syntax, nothing like this, only bytes and offsets. Think about TCP header, for example. JSON is very inefficient both in bytes (32 bit price is 4 bytes in binary and could be 7+ bytes as string, think "1299.99" for example) and CPU: to parse "1299.99" you need burn a lot of cycles, and if it is number of cents stored as native 4-byte number you need 3 shifts and 4 binary ors at most, if you need to change endianness, and in most cases it is simple memory copy of 4 bytes, 1-2 CPU cycle. When you have binary protocol, you could skip fields which you are not interested in as simple as "offset = offset + <filed-size>" (where <filed-size> is compile-time constant!) and in JSON you need to parse whole thing anyway. Difference between converting binary packet to internal data structure and parsing JSON with same data to same structure could be ten-fold easily, and you need to be very creative to parse JSON without additional memory allocations (it is possible, but code becomes very dirty and fragile), and memory allocation and/or deallocation costs a lot, both in GC languages and languages with manual memory management.
- joering2 5y ago
  Thanks for the writeup!
- localhost 5y ago
  Curious if the binary protocol uses floating point or a fixed point representation? Or is floating point with its rounding issues sufficient for the protocol's needs?
  - andylynch 5y ago
    No GP but familiar with these protocols. They use fixed point extensively; I can't even thing of an exchange protocol which would use floating point since the rounding issues would cause unnecessary and expensive problems. This is typical (from NASDAQ http://www.nasdaqtrader.com/content/technicalsupport/specifications/TradingProducts/OUCH4.2.pdf http://www.nasdaqtrader.com/content/technicalsupport/specifi... ): Prices are integer fields. When converted to a decimal format, prices are in fixed point format with 6 whole number places followed by 4 decimal digits. The maximum price in OUCH 4.2 is $199,999.9900 (decimal, 7735939C hex). When entering market orders for a cross, use the special price of $214,748.3647 (decimal, 7FFFFFFF hex).
andylynch 5y ago
Have a look at NASDAQ ITCH, OUCH, and RASH (these are the real names, the story I heard is the original author of didn't like the usual corporate style brand names and wanted certain people to squirm when talking about them).

deleted 5y ago

[deleted]

Foomf 5y ago

I've read your few posts a few times and I'm still not sure why you made your post. You're telling the person that you handle more data than them and thus need more resources than them. Was your goal to smugly belittle them? It's not like they said any problem can be solved on their specific resources.

Aeolun 5y ago
I feel like he’s saying two things. One, that he’s surprised by how small crypto markets are. Two, that this one server (or very few server processing) thing scales quite well to billions of messages a day. I didn’t find any element of smugness here, but maybe I misread the tone.
lost-found 5y ago
They are just pointing out the size of crypto market data vs normal market data. I found it interesting, not their fault you didn’t.
blacklion 5y ago
Nope, I want to say, that even much more data could be processed on very limited hardware, and that it is additional confirmation that current hardware is immensely powerful and very under-estimated.
- Cthulhu_ 5y ago
  And that go-to solutions / golden hammers like REST / JSON are very much suboptimal. I mean I'm still going to use it for client/server communication and the like because I don't have serious performance constraints enough to warrant something that will be more difficult to develop for etc, but still.
nostrademons 5y ago
I'd initially responded, but I found that my response had an element of a pissing match over whose data is bigger, so I deleted it. The thing is - when two engineers get smug, oftentimes lots of fairly interesting technical details get exchanged, so such discussions aren't really useless to bystanders.

secondcoming 5y ago

It depends on what your server does with each request though; '65B a day' means little. If all it does is write it to a log then I'm surprised you're not using a rPI.