3 ms·

Trading is for multiple reasons ideal for this, one is that total global ordering is a key feature (and requirement) of the domain so this "1 fast big server" t

Trading is for multiple reasons ideal for this, one is that total global ordering is a key feature (and requirement) of the domain so this "1 fast big server" thing is good. It is also quite widely known that several of the big exchange operate this model, a single sequencer application and then using multicast to transmit the outcomes of what it sees. The other thing that is helping a lot here compared to Discord: Trading is very neatly organized in trading days and shuts down for hours between each trading day. So you don't have the issue that Discord had where some channels have low message volumes and others have high, leading to having scattered data all over the place. You can naturally partition data by day and you know at query time which data you want to have.

enlyth 5y ago
What about cryptocurrency trading which goes on continuously for 24 hours a day?
- jfoutz 5y ago
  you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?
  - lmilcin 5y ago
    > but servers crash all the time This is both wise and stupid at the same time. It is wise if you mean "be ready for servers to crash at any time by thinking they are going to crash at the worst possible moment". But it is stupid, because people think they need massive parallel deployments just because servers will be constantly crashing and it is just not true. The cost they pay is in having couple of times more nodes than they really need to have if they got their focus right (making the application efficient first, scalable later) The reality is, servers do not crash. At least not the kind of hardware I am working on. I have been responsible for keeping communication with a stock exchange for like 3 years in one of my past jobs and during that time we haven't lost a single packet. And aside from some massive parallel loads which used tens of thousands of nodes and aside from one time my server room boiled over due to failed AC (and no environmental monitoring) I never had a server crash on me for the past 20 years. So you can reasonably assume that your servers will be functioning properly (if you bought quality) and it kinda helps a lot at design stage.
    - jfoutz 5y ago
      > 3 years in one of my past jobs and during that time we haven't lost a single packet. I. uh. I. That's amazing. Fuck, that's incredible. I'm trying to envision how I could track the success of each packet ever sent over the last 3 years. Just the metrics around that seems overwhelming. Jesus, that makes me question a lot about my personal skills. Fucking 'A man! Not losing a single packet in 3 years is. Shit I don't have the words. Congratulations! That is SICK (old American slang for good, very good). I can't even figure out how you would track that. Yeah, I'll delete my comments if you'd like. Wow. not a packet in 3 years. That's just. I don't know what that is. That's making me question my choice of profession. Damn. That's sooooo good. Holy shit that's crazy good.
      - lmilcin 5y ago
        We know because that communication happens on UDP and each packet on app layer has sequence number. It is used on receiving side to rebuild sequence of packets (events from the exchange can only be understood correctly when processed in same order as generated and only if you have complete stream -- you can't process a packet until you processed the one preceding it). It is trivial to detect that we haven't missed a packet. We had a special alert for a missing packet. To my knowledge that has never activated in production except for exchange-mandated tests (the exchange runs tests regularly to ensure every brokerage house can handle faults in communications, maximum loads, etc.) If a packed was missed, the same data is replicated on another, independent link through another carrier. And if this wasn't enough, if your system is down (which it shouldn't happen during trading) you can contact exchange's TCP service and request for the missing sequences. But that never happened, either. As we really liked this pattern, we built a small framework and used it for internal communication as well including data flowing to traders' workstations. Mind, that neither the carrier link, the networking devices or people who maintain it are cheap.
    - bob1029 5y ago
      > The reality is, servers do not crash This is the regime we operate in as well. For our business, a failure, while really bad, is not catastrophic (we still maintain non-repudiation). We look at it like any other risk model in the market. For many in our industry, the cost of not engineering this way and eating the occasional super rare bag of shit is orders of magnitude higher than otherwise tolerable. One well-managed server forged of the highest binned silicon is usually the cheapest and most effective engineering solution over the long term.
  - t0mas88 5y ago
    In regular trading (not crypto, see the other comment about the volume differences) it is common to tune Java for example to run GC outside of trading hours. That works if you don't allocate new heap memory in every transaction/message but instead only use the stack + pre-allocated pools.
  - deleted 5y ago
    [deleted]
- anonymoushn 5y ago
  Cryptocurrency venues handle extremely low throughput and, with maybe one exception, regularly go down for hours.
  - MacsHeadroom 5y ago
    The decentralized Solana network is currently capable of over 50k TPS 24/7 in the live Beta with a target transaction finality speed of 30ms round trip from your browser/device. Their unofficial moto is "DeFi at Nasdaq Speed." Solana is nascent and will likely reach 10M+ TPS and 10ms finality within a couple years time. Decentralized Acyclic Graph based networks (e.g. Hashigraph, which are not technically blockchains) can reach effectively infinite TPS but suffer in time to finality. Solana is a blockchain with zero downtown (and a Turing complete smartchain), mind you-- nota centralized exchange.
    - anonymoushn 5y ago
      Solana has 400ms block times, so I don't think it can achieve finality in 30ms, and the whole comment seems a bit off-topic.

enlyth 5y ago

What about cryptocurrency trading which goes on continuously for 24 hours a day?

jfoutz 5y ago
you can handle the load or not, right? A built in maintenance window is super nice, but servers crash all the time. So, that's a problem, or you've got a system in place. if you can handle failover, you've got free maintenance windows anyway, so it seems not any more difficult?
- lmilcin 5y ago
  > but servers crash all the time This is both wise and stupid at the same time. It is wise if you mean "be ready for servers to crash at any time by thinking they are going to crash at the worst possible moment". But it is stupid, because people think they need massive parallel deployments just because servers will be constantly crashing and it is just not true. The cost they pay is in having couple of times more nodes than they really need to have if they got their focus right (making the application efficient first, scalable later) The reality is, servers do not crash. At least not the kind of hardware I am working on. I have been responsible for keeping communication with a stock exchange for like 3 years in one of my past jobs and during that time we haven't lost a single packet. And aside from some massive parallel loads which used tens of thousands of nodes and aside from one time my server room boiled over due to failed AC (and no environmental monitoring) I never had a server crash on me for the past 20 years. So you can reasonably assume that your servers will be functioning properly (if you bought quality) and it kinda helps a lot at design stage.
  - jfoutz 5y ago
    > 3 years in one of my past jobs and during that time we haven't lost a single packet. I. uh. I. That's amazing. Fuck, that's incredible. I'm trying to envision how I could track the success of each packet ever sent over the last 3 years. Just the metrics around that seems overwhelming. Jesus, that makes me question a lot about my personal skills. Fucking 'A man! Not losing a single packet in 3 years is. Shit I don't have the words. Congratulations! That is SICK (old American slang for good, very good). I can't even figure out how you would track that. Yeah, I'll delete my comments if you'd like. Wow. not a packet in 3 years. That's just. I don't know what that is. That's making me question my choice of profession. Damn. That's sooooo good. Holy shit that's crazy good.
    - lmilcin 5y ago
      We know because that communication happens on UDP and each packet on app layer has sequence number. It is used on receiving side to rebuild sequence of packets (events from the exchange can only be understood correctly when processed in same order as generated and only if you have complete stream -- you can't process a packet until you processed the one preceding it). It is trivial to detect that we haven't missed a packet. We had a special alert for a missing packet. To my knowledge that has never activated in production except for exchange-mandated tests (the exchange runs tests regularly to ensure every brokerage house can handle faults in communications, maximum loads, etc.) If a packed was missed, the same data is replicated on another, independent link through another carrier. And if this wasn't enough, if your system is down (which it shouldn't happen during trading) you can contact exchange's TCP service and request for the missing sequences. But that never happened, either. As we really liked this pattern, we built a small framework and used it for internal communication as well including data flowing to traders' workstations. Mind, that neither the carrier link, the networking devices or people who maintain it are cheap.
  - bob1029 5y ago
    > The reality is, servers do not crash This is the regime we operate in as well. For our business, a failure, while really bad, is not catastrophic (we still maintain non-repudiation). We look at it like any other risk model in the market. For many in our industry, the cost of not engineering this way and eating the occasional super rare bag of shit is orders of magnitude higher than otherwise tolerable. One well-managed server forged of the highest binned silicon is usually the cheapest and most effective engineering solution over the long term.
- t0mas88 5y ago
  In regular trading (not crypto, see the other comment about the volume differences) it is common to tune Java for example to run GC outside of trading hours. That works if you don't allocate new heap memory in every transaction/message but instead only use the stack + pre-allocated pools.
- deleted 5y ago
  [deleted]
anonymoushn 5y ago
Cryptocurrency venues handle extremely low throughput and, with maybe one exception, regularly go down for hours.
- MacsHeadroom 5y ago
  The decentralized Solana network is currently capable of over 50k TPS 24/7 in the live Beta with a target transaction finality speed of 30ms round trip from your browser/device. Their unofficial moto is "DeFi at Nasdaq Speed." Solana is nascent and will likely reach 10M+ TPS and 10ms finality within a couple years time. Decentralized Acyclic Graph based networks (e.g. Hashigraph, which are not technically blockchains) can reach effectively infinite TPS but suffer in time to finality. Solana is a blockchain with zero downtown (and a Turing complete smartchain), mind you-- nota centralized exchange.
  - anonymoushn 5y ago
    Solana has 400ms block times, so I don't think it can achieve finality in 30ms, and the whole comment seems a bit off-topic.