3 ms·
I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and
by pclmulqdq 5y ago
I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation.
There is a much more serious focus on having a defense in depth, and making sure that problems like this are noticed before they become an issue. Rollbacks are no longer the first action when something goes wrong: the kill switch comes first.
Dead code, tech debt, repurposed flags, and spotty test coverage are everywhere still.
- commandlinefan 5y ago> poor test coverage Yet you don't have to hang around here long to be told that "Unit Testing is Overrated": https://tyrrrz.me/blog/unit-testing-is-overrated https://tyrrrz.me/blog/unit-testing-is-overrated
- aaronharnly 5y agoI’m curious about the “repurposed flags” part. I wouldn’t think of flags as expensive / effortful to make more of, but clearly they must be if people are tempted to reuse them. Can you help me understand what is meant by a flag in this context, and why it would be repurposed?
- isogon 5y agoRepurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint. Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of physical memory. The memory use is a near-constant percentage of your total memory, and you wouldn't want it to be any larger than necessary. Even when there are not a lot of these objects, in low-latency software it's important to hit the cache. Your program should always just be as compact in memory as possible. Semantically flags are booleans (is proposition P true of this object). They are stored compactly as bitsets, often implicitly, say: #define FLAG_1 0x01 #define FLAG_2 0x02 /* ... */ #define FLAG_8 0x80 struct order { u32 qty; u16 id; u8 type; u8 flags; }; This struct will fit into 8 bytes. This is great, as you probably won't waste space to alignment in many cases -- 8 is a good multiple. But if you wanted to add FLAG_9 here, your flags would become a u16, and your struct would, frustratingly, stop fitting into 8 bytes. To avoid this, one might repurpose flags. Another example of this is intrustive flagging, using, for example, the high or low bits of a pointer aligned to 2^n bytes. If you run out of bits there, not much you can do. [0] https://github.com/torvalds/linux/blob/master/include/linux/mm_types.h#L72 https://github.com/torvalds/linux/blob/master/include/linux/...
- pclmulqdq 5y agoThis is pretty much why flags get repurposed. It's also important to mention that things like JSON and protobufs are too expensive for HFT, so you are likely going to be sending structs over the wire. Repurposing flags lets you change a wire format with a lot less friction than adding a byte to a struct. Essentially, it lets you change the minor version number on a protocol and only recompile the endpoints without changing the major version number and recompiling everything.
- aaronharnly 5y agoThank you both!
- lordnacho 5y agoThis is what I thought as well, and I'm in the field. It's legit to try to fit everything in cache. However, you can still do something about making this safe. For instance the program could do some sort of version check on startup and panic if things weren't correct. There's a bunch of stuff that needs to be done before the program reaches its low latency steady state, where speed doesn't matter. Might as well add checks there to make sure things are correct.