5 ms·

The outages are caused by the execution brokers not being able to execute orders fast enough resulting in the order queue getting longer and longer. Also other

• motohagiography 6y ago
  That surprises me, as horizontal scaling to demand is a solved problem. One provider, sure, but cascading failures through many, that's a systemic critical infrastructure vulnerability that is going to bring down the regulatory hammer.
  • bpicolo 6y ago
    I'm under the impression that order books tend to be hard to scale horizontally, because they have tremendous atomicity requirement. Can anybody chime in whether these systems, today, tend to be on distributed infra versus e.g. mainframes or other big vertical compute pieces?
    • hangonhn 6y ago
      I don't know the specific answer to your question but I do have a data point to add. When I was just starting out in my career in the early 2000s, I worked for a hedge fund. One curious thing I noticed is that all the traders had time stamping machines on their desks that are inspected and sealed regularly. I was told that during an outage, these timestamp machines allow trades to continue to happen because the trader can write down the order and stamp it and their counter party would presumably do the same. Later the orders can then be reconciled when things are back up and running. This leads me to think that it is at the very least possible for transactions to be recorded in a distributed manner. Example of machine: http://www.carpenterstimesystems.com/Amano-TS3000i-OATS-Time-Validation-Clock-System.html http://www.carpenterstimesystems.com/Amano-TS3000i-OATS-Time... This might all be historical now.
    • LittlePeter 6y ago
      The matching of orders for a single asset happens on one machine, call it matching engine. Matching is done based on price-time priority. Price is set by the trader, the time is set to the time the order was received by one of the order gateways sitting in front of the matching engine. The order gateways' clocks are in sync, up to precision epsilon. So if you send two orders serially from same data center (to ignore effects of public Internet latencies and routing), and if they happen to be processed by different gateways, you still will see same ordering of your orders in matching engine, bar some cosmic ray events. It is all commodity hardware running some flavor of Linux. However this was couple years ago, they may be using ASICs and FPGAs now. The matching of orders for a single asset must still be done in single process otherwise you cannot guarantee price-time priority.
    • motohagiography 6y ago
      Following up, there was a random theory on WSB that since hedge funds were short over %130 of Gamestop's stock, there is a real risk that this will cause a liquidity crunch where there is literally no available stock to cover their positions. As in there does not exist the stock to cover those positions, which means institutions are going to be in a race to avoid bankruptcy and have to outbid each other to acquire the remaining stock held by all these retail investors who know what they have. This is why WSB posters are saying $5000 is a conceivable price. This is what would cause regulatory intervention, and certainly sets up the means/motive/opportunity for interrupting trading. Like the Fed steps in and buys it out or something. This looks like an LTCM level event.
  • manigandham 6y ago
    Every order affects the entire market (at least for that security and its derivatives) so they must be processed serially in a single process.
  • kristjansson 6y ago
    No. https://www.youtube.com/watch?v=qDhTjE0XmkE https://www.youtube.com/watch?v=qDhTjE0XmkE