3 ms·
I once made a similar mistake. It ended up being reported in the Economist. We had developed the central trading system for a major country, which conducted i
by mianos 3y ago
I once made a similar mistake. It ended up being reported in the Economist.
We had developed the central trading system for a major country, which conducted its primary bill and bond issuance through our platform. Approximately once a week, in the mornings, banks would place orders for government bills, specifying a price and the volume they wished to purchase at that price. They submitted multiple orders, indicating a willingness to buy more at higher yields.
Once all orders were submitted, the instrument would be suspended, and a scaled order report generated. The central bank would then retreat to a room to analyze the data, determining how much they could borrow and at what rate. For example, they might be able to borrow a billion at 10% interest, but only 500 million at 8%.
After settling on a price, they would issue a large sell order to match the existing orders, leaving any unmatched orders in the book. We would then restart the instrument in a continuous trading mode, allowing banks to continue buying and selling the new series.
I was tasked with enhancing the instrument suspension feature. The enhancement involved adding a flag and a button to the GUI, allowing for the optional withdrawal of orders when suspending an instrument. This seemingly simple protocol required adding a byte as a flag, indicating either 0 or 1, at the end of the message. After incorporating this feature into both the system and the UI, and following extensive testing, we deployed it.
However, during the next auction, an issue arose. When the operators suspended the market, all orders were mistakenly withdrawn. To avoid the embarrassment of asking participants to resubmit their orders, they announced the auction's cancellation due to poor participation and unfavorable pricing.
The root of the problem was discovered later. Operations had been using a command-line tool to suspend the market, a tool I wasn't even aware of. As the auction price report was generated from the command line just before the report, the command-line tool had a random value in the withdrawal flag's place, leading to the withdrawal of all orders.
- KMag 3y agoSo, the system had been accepting malformed messages that were too long? Did the cli tool get fixed and the system get better sanity checks on incoming messages?
- mianos 3y agoThe command line tool generated a message using the old format, it was one byte short (missing the 'withdraw on suspend' flag). By the time the message got to the matching engine, the engine looked at the byte on the end of the withdraw message and it randomly happened to be true. The fix, at the time, was to add a 'N' to the end of the message the command line tool generated. Later I re-wrote everything to use a compiler generated message encoder and decoder, SNACC ASN.1 (Something like protobuffers, as invented again by Google, which was also invented the first time by Sun as XDR/RPC, that was also invented for the first time as CORBA. I think MS invented it as well for COM).