5 ms·

Where can I read more about those formats? I work in crypto, same as cantor trading that posted this, and always keen on re-using solved problems if they work :

by exdsq 5y ago

Where can I read more about those formats? I work in crypto, same as cantor trading that posted this, and always keen on re-using solved problems if they work :)

andylynch 5y ago
A great starting point is FIX. They are the open standards group in this area, at https://www.fixtrading.org/ https://www.fixtrading.org/. ITCH and OUCH came from Island, now part of NASDAQ and the specs are on their website.
- exdsq 5y ago
  Thanks! I saw they have a working group on crypto so applied to join that, keen to see what they're up to. Standardizing data structures across chains makes a lot of sense but sadly there's a lot of competition rather than collaboration between them "Mine is better than yours hahahaha moon" etc... I'll have to give this some real thought! I think one issue may be that the specification is so complex implementing it within a new DEX without buy-in from others means it's a lot of work with little reward. Might be the case that if larger established front-offices/exchanges start expecting it that crypto could follow. Coinbase released an API standard that blockchains had to conform to for testing the chain prior to listing so it's not unprecedented for that to happen.
  - ajoseps 5y ago
    a couple of exchanges already have FIX APIs. I think Coinbase Pro and Gemini both do.
    - Bootvis 5y ago
      Bitstamp as well
    - exdsq 5y ago
      Interesting! I’ve worked on the other side - blockchains themselves - and only recently started working for a DEX. Lots to learn!
- amluto 5y ago
  > A great starting point is FIX. I beg to differ. FIX could almost be a case study on how not to design a protocol. Session management and recovery is outside the login protocol? Check! Can’t reliably recover from a failed login or wrong password attempt? Check! [0] Message boundaries are nontrivial to detect? Check! Field values are binary but cannot represent the byte 0x01? Check! But FIX has a brilliant extensible hierarchical message structure that is more type safe than JSON, you say! Sure, kind of, but the encoding of lists of nested objects is utter nonsense. Not only is it somewhat redundant (the number of repeats is sent first, but the number of repeats is also determinable after parsing the whole thing, resulting in some awkward questions about what to do if the numbers disagree), but it has worse problems. Specifically, if the sender and receiver don’t agree on the precise set of fields and their order in each nested object, then the list cannot be parsed. This makes the whole concept of extensibility someone dubious. (This is because there is no delimiter between repeated objects; instead you notice that the field you’re parsing can’t legally belong to the object you’re parsing, meaning it’s time to use a crystal ball to determine whether a new repeat has started, the whole list is done, or an error has occurred.) On top of all this, the standard implementation, QuickFIX, is a work of art, and not in a good way. FIX SBE is IMO not at all better except that it can be serialized and deserialized faster. [0]. No, really. If you try to log in to someone else’s session and the server rejects the password, you have just fried their session. The protocol cannot recover. Sometimes I wonder if this was deliberate, because screwing this up at the protocol level takes some creativity. Imagine if trying to log in to someone else’s email account caused their next login attempt to fail! Oh wait, some password attempt policies do this, but FIX does it by corrupting the whole protocol state machine rather than merely having a misguided policy.
  - exdsq 5y ago
    That login stuff sounds… painful. So you’re saying as UserA if I try to login as UserB, UserBs login is corrupted and they have to relogin?
    - amluto 5y ago
      Worse. For a “session” (connection from a client to a server with corresponding IDs — the IDs serve a purpose somewhere between a username and a TCP port), there is some state that is used to recover potentially lost messages if the session drops and reconnects. If a session is down, someone tries to connect and restore that session, and the login fails, then the replay state will get out of sync on both ends and a subsequent valid reconnection will either fail or lose messages.
twic 5y ago
Eurex's 'Enhanced Order Book Interface' is another one, with documentation that i think is pretty good - there's a PDF manual here: https://www.eurex.com/ex-en/support/initiatives/t7-release-10-0 https://www.eurex.com/ex-en/support/initiatives/t7-release-1...
- exdsq 5y ago
  Thanks Twic!
- blibble 5y ago
  T7 is a mess LSE's platform is much more modern: GTP being pretty much the perfect example of how to build a modern super-fast protocol https://www.lseg.com/sites/default/files/content/documents/GTP%20002%20-%20Technical%20Guide%20-%20Issue%2020.8_4.pdf https://www.lseg.com/sites/default/files/content/documents/G...

by exdsq 5y ago

Where can I read more about those formats? I work in crypto, same as cantor trading that posted this, and always keen on re-using solved problems if they work :)

andylynch 5y ago

A great starting point is FIX. They are the open standards group in this area, at https://www.fixtrading.org/ https://www.fixtrading.org/. ITCH and OUCH came from Island, now part of NASDAQ and the specs are on their website.

exdsq 5y ago
Thanks! I saw they have a working group on crypto so applied to join that, keen to see what they're up to. Standardizing data structures across chains makes a lot of sense but sadly there's a lot of competition rather than collaboration between them "Mine is better than yours hahahaha moon" etc... I'll have to give this some real thought! I think one issue may be that the specification is so complex implementing it within a new DEX without buy-in from others means it's a lot of work with little reward. Might be the case that if larger established front-offices/exchanges start expecting it that crypto could follow. Coinbase released an API standard that blockchains had to conform to for testing the chain prior to listing so it's not unprecedented for that to happen.
- ajoseps 5y ago
  a couple of exchanges already have FIX APIs. I think Coinbase Pro and Gemini both do.
  - Bootvis 5y ago
    Bitstamp as well
  - exdsq 5y ago
    Interesting! I’ve worked on the other side - blockchains themselves - and only recently started working for a DEX. Lots to learn!
amluto 5y ago
> A great starting point is FIX. I beg to differ. FIX could almost be a case study on how not to design a protocol. Session management and recovery is outside the login protocol? Check! Can’t reliably recover from a failed login or wrong password attempt? Check! [0] Message boundaries are nontrivial to detect? Check! Field values are binary but cannot represent the byte 0x01? Check! But FIX has a brilliant extensible hierarchical message structure that is more type safe than JSON, you say! Sure, kind of, but the encoding of lists of nested objects is utter nonsense. Not only is it somewhat redundant (the number of repeats is sent first, but the number of repeats is also determinable after parsing the whole thing, resulting in some awkward questions about what to do if the numbers disagree), but it has worse problems. Specifically, if the sender and receiver don’t agree on the precise set of fields and their order in each nested object, then the list cannot be parsed. This makes the whole concept of extensibility someone dubious. (This is because there is no delimiter between repeated objects; instead you notice that the field you’re parsing can’t legally belong to the object you’re parsing, meaning it’s time to use a crystal ball to determine whether a new repeat has started, the whole list is done, or an error has occurred.) On top of all this, the standard implementation, QuickFIX, is a work of art, and not in a good way. FIX SBE is IMO not at all better except that it can be serialized and deserialized faster. [0]. No, really. If you try to log in to someone else’s session and the server rejects the password, you have just fried their session. The protocol cannot recover. Sometimes I wonder if this was deliberate, because screwing this up at the protocol level takes some creativity. Imagine if trying to log in to someone else’s email account caused their next login attempt to fail! Oh wait, some password attempt policies do this, but FIX does it by corrupting the whole protocol state machine rather than merely having a misguided policy.
- exdsq 5y ago
  That login stuff sounds… painful. So you’re saying as UserA if I try to login as UserB, UserBs login is corrupted and they have to relogin?
  - amluto 5y ago
    Worse. For a “session” (connection from a client to a server with corresponding IDs — the IDs serve a purpose somewhere between a username and a TCP port), there is some state that is used to recover potentially lost messages if the session drops and reconnects. If a session is down, someone tries to connect and restore that session, and the login fails, then the replay state will get out of sync on both ends and a subsequent valid reconnection will either fail or lose messages.

twic 5y ago

Eurex's 'Enhanced Order Book Interface' is another one, with documentation that i think is pretty good - there's a PDF manual here: https://www.eurex.com/ex-en/support/initiatives/t7-release-10-0 https://www.eurex.com/ex-en/support/initiatives/t7-release-1...

exdsq 5y ago
Thanks Twic!
blibble 5y ago
T7 is a mess LSE's platform is much more modern: GTP being pretty much the perfect example of how to build a modern super-fast protocol https://www.lseg.com/sites/default/files/content/documents/GTP%20002%20-%20Technical%20Guide%20-%20Issue%2020.8_4.pdf https://www.lseg.com/sites/default/files/content/documents/G...