6 ms·

A great starting point is FIX. They are the open standards group in this area, at https://www.fixtrading.org/ https://www.fixtrading.org/. ITCH and OUCH came

A great starting point is FIX. They are the open standards group in this area, at https://www.fixtrading.org/ https://www.fixtrading.org/. ITCH and OUCH came from Island, now part of NASDAQ and the specs are on their website.

exdsq 5y ago
Thanks! I saw they have a working group on crypto so applied to join that, keen to see what they're up to. Standardizing data structures across chains makes a lot of sense but sadly there's a lot of competition rather than collaboration between them "Mine is better than yours hahahaha moon" etc... I'll have to give this some real thought! I think one issue may be that the specification is so complex implementing it within a new DEX without buy-in from others means it's a lot of work with little reward. Might be the case that if larger established front-offices/exchanges start expecting it that crypto could follow. Coinbase released an API standard that blockchains had to conform to for testing the chain prior to listing so it's not unprecedented for that to happen.
- ajoseps 5y ago
  a couple of exchanges already have FIX APIs. I think Coinbase Pro and Gemini both do.
  - Bootvis 5y ago
    Bitstamp as well
  - exdsq 5y ago
    Interesting! I’ve worked on the other side - blockchains themselves - and only recently started working for a DEX. Lots to learn!
amluto 5y ago
> A great starting point is FIX. I beg to differ. FIX could almost be a case study on how not to design a protocol. Session management and recovery is outside the login protocol? Check! Can’t reliably recover from a failed login or wrong password attempt? Check! [0] Message boundaries are nontrivial to detect? Check! Field values are binary but cannot represent the byte 0x01? Check! But FIX has a brilliant extensible hierarchical message structure that is more type safe than JSON, you say! Sure, kind of, but the encoding of lists of nested objects is utter nonsense. Not only is it somewhat redundant (the number of repeats is sent first, but the number of repeats is also determinable after parsing the whole thing, resulting in some awkward questions about what to do if the numbers disagree), but it has worse problems. Specifically, if the sender and receiver don’t agree on the precise set of fields and their order in each nested object, then the list cannot be parsed. This makes the whole concept of extensibility someone dubious. (This is because there is no delimiter between repeated objects; instead you notice that the field you’re parsing can’t legally belong to the object you’re parsing, meaning it’s time to use a crystal ball to determine whether a new repeat has started, the whole list is done, or an error has occurred.) On top of all this, the standard implementation, QuickFIX, is a work of art, and not in a good way. FIX SBE is IMO not at all better except that it can be serialized and deserialized faster. [0]. No, really. If you try to log in to someone else’s session and the server rejects the password, you have just fried their session. The protocol cannot recover. Sometimes I wonder if this was deliberate, because screwing this up at the protocol level takes some creativity. Imagine if trying to log in to someone else’s email account caused their next login attempt to fail! Oh wait, some password attempt policies do this, but FIX does it by corrupting the whole protocol state machine rather than merely having a misguided policy.
- exdsq 5y ago
  That login stuff sounds… painful. So you’re saying as UserA if I try to login as UserB, UserBs login is corrupted and they have to relogin?
  - amluto 5y ago
    Worse. For a “session” (connection from a client to a server with corresponding IDs — the IDs serve a purpose somewhere between a username and a TCP port), there is some state that is used to recover potentially lost messages if the session drops and reconnects. If a session is down, someone tries to connect and restore that session, and the login fails, then the replay state will get out of sync on both ends and a subsequent valid reconnection will either fail or lose messages.
- dahfizz 5y ago
  FIX is less ergonomic than JSON, that is not in question. FIX is much faster to generate and parse than JSON, and much easier to deal with than bespoke binary feeds. I think FIX makes the right tradeoffs. I don't understand what you mean regarding killing someone else's session. You can reset sequence numbers in the login message.
  - amluto 5y ago
    There is absolutely no reason that FIX should be faster to generate than JSON. I agree that one could plausibly parse it faster. JSON wasn’t meant as a real comparison though — FIX is structurally much more like protocol buffers, but protocol buffers are much better. As for sequence numbers: suppose your session drops. You’ve received the message with sequence number 5, but you’ve missed 6 and 7. You’ve sent 10 and 11 but the other end only got 10. Now someone (your own misconfigured software or someone else) connects and sends Logon to the server with your comp id, with sequence number 11 but the wrong authentication data. What is even supposed to happen? The session now contains two different message 11s to the server. It’s broken. You can (sometimes) recover by resetting sequence numbers, but that defeats the entire purpose of having sequence numbers in the first place: the server was supposed to recover the original message 11 on a reconnection attempt and correctly handle it as a possible duplicate. (As a practical matter merely experiencing network trouble during a reconnection also seems to break everything, mostly because common FIX implementations are terrible.) I suppose it must be possible to be so deeply entrenched in the FIX ecosystem that this makes any sense at all, but the whole protocol is backwards. You should connect, authenticate that connection, and then run the session protocol inside the authenticated connection. The Logon should not be sequenced — that gets the layering backwards.

exdsq 5y ago

Thanks! I saw they have a working group on crypto so applied to join that, keen to see what they're up to. Standardizing data structures across chains makes a lot of sense but sadly there's a lot of competition rather than collaboration between them "Mine is better than yours hahahaha moon" etc... I'll have to give this some real thought! I think one issue may be that the specification is so complex implementing it within a new DEX without buy-in from others means it's a lot of work with little reward. Might be the case that if larger established front-offices/exchanges start expecting it that crypto could follow. Coinbase released an API standard that blockchains had to conform to for testing the chain prior to listing so it's not unprecedented for that to happen.

ajoseps 5y ago
a couple of exchanges already have FIX APIs. I think Coinbase Pro and Gemini both do.
- Bootvis 5y ago
  Bitstamp as well
- exdsq 5y ago
  Interesting! I’ve worked on the other side - blockchains themselves - and only recently started working for a DEX. Lots to learn!

amluto 5y ago

> A great starting point is FIX. I beg to differ. FIX could almost be a case study on how not to design a protocol. Session management and recovery is outside the login protocol? Check! Can’t reliably recover from a failed login or wrong password attempt? Check! [0] Message boundaries are nontrivial to detect? Check! Field values are binary but cannot represent the byte 0x01? Check! But FIX has a brilliant extensible hierarchical message structure that is more type safe than JSON, you say! Sure, kind of, but the encoding of lists of nested objects is utter nonsense. Not only is it somewhat redundant (the number of repeats is sent first, but the number of repeats is also determinable after parsing the whole thing, resulting in some awkward questions about what to do if the numbers disagree), but it has worse problems. Specifically, if the sender and receiver don’t agree on the precise set of fields and their order in each nested object, then the list cannot be parsed. This makes the whole concept of extensibility someone dubious. (This is because there is no delimiter between repeated objects; instead you notice that the field you’re parsing can’t legally belong to the object you’re parsing, meaning it’s time to use a crystal ball to determine whether a new repeat has started, the whole list is done, or an error has occurred.) On top of all this, the standard implementation, QuickFIX, is a work of art, and not in a good way. FIX SBE is IMO not at all better except that it can be serialized and deserialized faster. [0]. No, really. If you try to log in to someone else’s session and the server rejects the password, you have just fried their session. The protocol cannot recover. Sometimes I wonder if this was deliberate, because screwing this up at the protocol level takes some creativity. Imagine if trying to log in to someone else’s email account caused their next login attempt to fail! Oh wait, some password attempt policies do this, but FIX does it by corrupting the whole protocol state machine rather than merely having a misguided policy.

exdsq 5y ago
That login stuff sounds… painful. So you’re saying as UserA if I try to login as UserB, UserBs login is corrupted and they have to relogin?
- amluto 5y ago
  Worse. For a “session” (connection from a client to a server with corresponding IDs — the IDs serve a purpose somewhere between a username and a TCP port), there is some state that is used to recover potentially lost messages if the session drops and reconnects. If a session is down, someone tries to connect and restore that session, and the login fails, then the replay state will get out of sync on both ends and a subsequent valid reconnection will either fail or lose messages.
dahfizz 5y ago
FIX is less ergonomic than JSON, that is not in question. FIX is much faster to generate and parse than JSON, and much easier to deal with than bespoke binary feeds. I think FIX makes the right tradeoffs. I don't understand what you mean regarding killing someone else's session. You can reset sequence numbers in the login message.
- amluto 5y ago
  There is absolutely no reason that FIX should be faster to generate than JSON. I agree that one could plausibly parse it faster. JSON wasn’t meant as a real comparison though — FIX is structurally much more like protocol buffers, but protocol buffers are much better. As for sequence numbers: suppose your session drops. You’ve received the message with sequence number 5, but you’ve missed 6 and 7. You’ve sent 10 and 11 but the other end only got 10. Now someone (your own misconfigured software or someone else) connects and sends Logon to the server with your comp id, with sequence number 11 but the wrong authentication data. What is even supposed to happen? The session now contains two different message 11s to the server. It’s broken. You can (sometimes) recover by resetting sequence numbers, but that defeats the entire purpose of having sequence numbers in the first place: the server was supposed to recover the original message 11 on a reconnection attempt and correctly handle it as a possible duplicate. (As a practical matter merely experiencing network trouble during a reconnection also seems to break everything, mostly because common FIX implementations are terrible.) I suppose it must be possible to be so deeply entrenched in the FIX ecosystem that this makes any sense at all, but the whole protocol is backwards. You should connect, authenticate that connection, and then run the session protocol inside the authenticated connection. The Logon should not be sequenced — that gets the layering backwards.