6 ms·

Out of interest, do you have any literature that you'd recommend instead?

by mbo 2y ago

nickelpro 2y ago
On the software side I don't think HFT is as special a space as this paper makes it out to be.[1] Each year at cppcon there's another half-dozen talks going in depth on different elements of performance that cover more ground collectively than any single paper will. Similarly, there's an immense amount of formal literature and textbooks out of the game development space that can be very useful to newcomers looking for structural approaches to high performance compute and IO loops. Games care a lot about local and network latency, the problem spaces aren't that far apart (and writing games is a very fun way to learn). I don't have specific recommendations for holistic introductions to the field. I learn new techniques primarily through building things, watching conference talks, reading source code of other low latency projects, and discussion with coworkers. [1]: HFT is quite special on the hardware side, which is discussed in the paper. The NICs, network stacks, and extensive integration of FPGAs do heavily differentiate the industry and I don't want to insinuate otherwise. You will not find a lot of SystemVerilog programmers at a typical video game studio.
- Maxatar 2y ago
  As someone who does quant trading professionally and game development as a hobby, they both are performance sensitive, but they emphasize different kinds of performance. Trading is about minimizing latency while video games are about maximizing bandwidth. Video games try to cram as much work as possible within about 16 milliseconds whereas for most trading algorithms 16 milliseconds is too slow to do anything, you want to process and produce a response to input within the span of microseconds, which is 3 orders of magnitude faster than a single frame in a video game. The problem spaces really are quite distinct in this respect and a lot of techniques from one space don't really carry over to the other.
  - nickelpro 2y ago
    Of course, time spans are different. But read the paper, it's dealing with incredibly basic techniques. When we're speaking on the level of "data should be in cache" and "cold code should be far away from the hot path" the fields are practically identical. You're entirely correct that quant work is dealing with pure latency in situations where game engines might amortize throughput over multiple frames. But a question about "how do I get started in performance/latency work?" isn't usefully answered by "get a Hudson River Trading internship".
    - Maxatar 2y ago
      On that point we certainly agree.
  - ssfrr 2y ago
    I'm curious how HFT relates to pro audio programming. The timescale is close to gaming (usually <10ms for desktop PC work), but you really care about the tail of the distribution. All the audio processing happens in a high-priority callback that's called for every audio frame and needs complete within the time budget, so you typically never allocate, do IO, or use any blocking synchronization primitives [1]. It's not hard real-time like you're going to crash your car, but if you miss your deadline it causes an unacceptable and audible glitch. I've always been a bit surprised that Jane Street uses OCaml. I know they've put a lot of attention into the GC, but it still seems fundamentally indeterminate in a way that would make most audio devs nervous. [1]: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...
    - steve1977 2y ago
      In audio, you can usually tolerate a few milliseconds of latency (more when mixing, less when recording). This means you can buffer your audio stream, for example in blocks of 64 or 128 samples. As far as I know, these millisecond latencies are orders of magnitude higher than what would be tolerable in HFT. There the units are microseconds and nanoseconds.
    - deleted 2y ago
      [deleted]
    - dintech 2y ago
      Audio has a lot of buffering behaviour that you wouldn't generally see in event-reactive HFT. Think of all the plugins that you know of that have non-zero latency, compressors with 'lookahead' etc. There are maybe some similarities where the logic is more complex (loop unrolling, SIMD and so on) but I feel like plugins are generally optimizing for throughput (CPU usage) and quality (oversampling etc) rather than purely latency in most cases.
      - ssfrr 2y ago
        I guess the difference I'm interested in is whether HFT tends to be "realtime", in the sense that there's a hard deadline that you need to hit every time. Put another way, audio is operating on a much longer timescale, but cares a lot about worst-case latency (as well as throughput and quality). Is that true of HFT also, or are they more concerned with average latency? If computing a trade takes too long, can they just discard it, or is it catastrophic to miss that deadline?
  - atomicnumber3 2y ago
    "which is 3 orders of magnitude faster than a single frame in a video game." You're right on the money. I worked in HFT for half a decade. Back in the late 2000s you were on the cutting edge if you were writing really good C++ and had overclocked some CPUs to hell and back and then shoved them in a rack in a datacenter in new jersey. "To hell and back" means "they only crash every hour or two" (because while they're up, they're faster than your competition). Around the mid 2010's it had moved entirely to FPGAs. Do something smart in software, then give the FPGA something dumb to wait for (e.g. use software to produce a model and wait for some kind of alpha to develop, then tell the FPGA "ok wait for X > Y and fire the order" was basically the name of the game. And it was often better to be faster than it was to be smarter, so really you just kept the FPGA as simple and fast as possible. At that point, your hard realtime constraints for the language doing the smart stuff really dissolve. Compared to an FPGA getting an order out the door in some fraction of a mic, a CPU doing anything is slow. So you might as well use python, yknow? People also play all kinds of different speed games. There's latency races around NJ and chicago where microseconds matter around price movements and C++ isn't really part of the picture in a competitive way anymore, but that's not to say someone isn't ekeing out a niche where they're doing something vaguely smarter faster than opponents. But these things, at scale, tend to be questions of - either your alpha is very short-lived (microseconds, and if it isn't organically microseconds, it will be competed until it is), or it is fundamentally longer-lived (seconds to minutes?) and you might as well use python and develop 100x faster. The silly thing is, the really good trades that focus on short-term alphas (ms's and less) are generally obviously good trades and so you don't have to be super smart to realize it's a really fucking good trade, you had better be fast and go get it. So there's also this kind of built-in bias for being fast because if a trade is so marginally good that you needed a crazy smart and slow model to tell it apart from a bad trade, it's probably still a relatively crummy trade and all your smarts let you do is pick up a few extra pennies for your trouble. I'll close by saying don't take my words as representative of the industry - the trades my firm specialized in was literally a dying breed and my understanding is that most of the big crazy-crazy-fast-microwave-network people have moved to doing order execution anyway, because most of the money in the crazy-crazy-fast game dried up as more firms switched to either DIYing the order execution or paying a former-HFT to do it for them.
    - 2y ago
- hawk_ 2y ago
  What other low latency projects in public domain are worth learning from?
  - nickelpro 2y ago
    Whatever is relevant to the domain you're tackling. How you structurally approach latency in a web server is different than a 3D renderer, which is different than an audio stack, etc. All of those domains have great open source or source available code to learn from, but it wouldn't be useful for me to say "go read HAProxy" if you're never going to touch an HTTP packet. When I'm tackling a problem the first thing I do is research everything everyone else has done on the problem and read their code, benchmark it if possible, and steal all the good ideas. The basic principles never change. Avoid copies, never allocate, keep the hot path local, and really, good codebases should be doing these things anyway. I don't code any differently when I'm writing a command line argument parser vs low-latency RPC servers. It's just a matter of how long I spend tweaking and trying to improve a specific section of code, how willing I am to throw out an entire abstraction I've been working on for perf. In the domain of web stuff, effectively all the major load balancers are good to study. HAProxy, nginx, Envoy. Also anything antirez has ever touched. Application servers are also interesting to study because there's many tricks to learn from a piece of software that needs to interface with something very slow but otherwise wants to be completely transparent in the call graph. FastWSGI is a good example.
- mgaunard 2y ago
  Latency isn't even as important in HFT as people claim. What's most important is deterministically staying into a reasonable enveloppe (even at the 99.9 percentile) to satisfy real-time requirements and not fall behind. When it's really important, it's implemented in FPGA or with an ASIC.
  - ssfrr 2y ago
    This is close to the question I asked above, comparing to audio. If you're processing 32 samples at a time at 48kHz, you need to process a frame every 667us. If you only hit the deadline 99.9% of the time, that's a glitch more than once per second. You'd need a bunch more nines for an acceptable commercial product, so most audio devs just treat it as a hard realtime constraint. I think the main split is whether you care more about average latency or worst-case latency (or some # of nines percentile).
    - mgaunard 2y ago
      In HFT you typically only measure latency for events that you did react on, not all events (at least, in the systems that I've built). That's arguably a bit problematic as events you don't react to can cause you to fall behind, so it's useful to also track how far behind you are from the point where you start processing a packet (which requires hardware timestamping, unfortunately not present on many general-purpose NICs like what's in the cloud). For a given market, you may have less than 1M samples a day (I've even seen some so-called HFT strategies or sub-strategies that only had 300 samples per day). So overall, you'd have fewer events than 44.1kHz, and there are typically expected outliers at the beginning of the data before the system gets warmed up or recovers from bad initial parameters (which I suppose you could just ignore from your distribution, or you could try not to require a warmup). But you're right, you probably want to look at quantiles closer to 1 as well. You're also looking at the max regardless.
_aavaa_ 2y ago
I'd recommend: https://www.computerenhance.com https://www.computerenhance.com The author has a strong game development (engine and tooling) background and I have found it incredibly useful. It also satisfies the requirement for "A genuine fear, hate, and anger, towards unnecessary allocations, copies, and other performance killers."
- hi_dang_ 2y ago
  Very anecdotal but of the people I know in game studios who are tasked with engine work, and people who make a killing doing FPGA work for HFT firms, both camps shook their head at Casey’s HMH thing. Uniformly I do not know of a single professional developer of this sort of caliber who looked at HMH and thought it looks great. Quite the opposite. I think they found his approach and justifications unsound as it would instil awful practices of premature unfounded optimization and a disdain for normal library code in favour of hand-rolling your own half-baked implementations based on outdated trivia. I agree with them on the basis that HMH exposed an unprepared and inexperienced audience to something that has to be regarded with utmost care. For this, I refer to Jonathan Blow’s presentation of “a list is probably good enough” as an antidote. I think JB’s recommendations are more in line with actual practices, whereas Casey just raised red flags uniformly from here-and-now engine devs shipping multi platform games.
  - nickelpro 2y ago
    Casey's a fundamentalist, in the religious extremist sense. Just as a theologian might have something to learn from the most extremely devout, so it is that there is valuable information in Casey's proselytizing. But you should no more follow Casey in form and function than you would any other fundamentalist.
    - _aavaa_ 2y ago
      While gesturing at religion and saying he's a fundamentalist is an easy rhetorical technique, it doesn't add much. Yes good resources do exist, but they are few and far between. The number of new undergraduates from cs or software engineering who learned to think in the ways needed to make low latency applications are a rounding error based on my experience with them. Most forgot it, weren't taught it well, or never saw it at all. They are more worried about learning the syntax of the current hot framework rather than on anything even resembling "Avoid copies, never allocate, keep the hot path local". The religious comparison is also a telling one given the state of the industry; for we aren't the theologian, we're the common folk looking for someone or something to follow in order to write better code. Who are Casey's alternatives? Gesturing the cppcon as a learning resource has "read research papers to learn about a field" vibes. They can be highly informative and worth the effort, but not for beginners. Who are Casey's contemporaries? If he's a fundamentalist then the atheists are nowhere to be seen by the beginners. Instead we have agile ceremony shamans, clean code bible missionaries, tdd doomsday cults, oop/rust/haskell/etc. zealots, a thriving megachurch industry of Lambda schools, and the mother of all cargo cults masquerading as web dev.

Out of interest, do you have any literature that you'd recommend instead?

by mbo 2y ago

Out of interest, do you have any literature that you'd recommend instead?

On the software side I don't think HFT is as special a space as this paper makes it out to be.[1] Each year at cppcon there's another half-dozen talks going in depth on different elements of performance that cover more ground collectively than any single paper will. Similarly, there's an immense amount of formal literature and textbooks out of the game development space that can be very useful to newcomers looking for structural approaches to high performance compute and IO loops. Games care a lot about local and network latency, the problem spaces aren't that far apart (and writing games is a very fun way to learn). I don't have specific recommendations for holistic introductions to the field. I learn new techniques primarily through building things, watching conference talks, reading source code of other low latency projects, and discussion with coworkers. [1]: HFT is quite special on the hardware side, which is discussed in the paper. The NICs, network stacks, and extensive integration of FPGAs do heavily differentiate the industry and I don't want to insinuate otherwise. You will not find a lot of SystemVerilog programmers at a typical video game studio.

Maxatar 2y ago
As someone who does quant trading professionally and game development as a hobby, they both are performance sensitive, but they emphasize different kinds of performance. Trading is about minimizing latency while video games are about maximizing bandwidth. Video games try to cram as much work as possible within about 16 milliseconds whereas for most trading algorithms 16 milliseconds is too slow to do anything, you want to process and produce a response to input within the span of microseconds, which is 3 orders of magnitude faster than a single frame in a video game. The problem spaces really are quite distinct in this respect and a lot of techniques from one space don't really carry over to the other.
- nickelpro 2y ago
  Of course, time spans are different. But read the paper, it's dealing with incredibly basic techniques. When we're speaking on the level of "data should be in cache" and "cold code should be far away from the hot path" the fields are practically identical. You're entirely correct that quant work is dealing with pure latency in situations where game engines might amortize throughput over multiple frames. But a question about "how do I get started in performance/latency work?" isn't usefully answered by "get a Hudson River Trading internship".
  - Maxatar 2y ago
    On that point we certainly agree.
- ssfrr 2y ago
  I'm curious how HFT relates to pro audio programming. The timescale is close to gaming (usually <10ms for desktop PC work), but you really care about the tail of the distribution. All the audio processing happens in a high-priority callback that's called for every audio frame and needs complete within the time budget, so you typically never allocate, do IO, or use any blocking synchronization primitives [1]. It's not hard real-time like you're going to crash your car, but if you miss your deadline it causes an unacceptable and audible glitch. I've always been a bit surprised that Jane Street uses OCaml. I know they've put a lot of attention into the GC, but it still seems fundamentally indeterminate in a way that would make most audio devs nervous. [1]: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...
  - steve1977 2y ago
    In audio, you can usually tolerate a few milliseconds of latency (more when mixing, less when recording). This means you can buffer your audio stream, for example in blocks of 64 or 128 samples. As far as I know, these millisecond latencies are orders of magnitude higher than what would be tolerable in HFT. There the units are microseconds and nanoseconds.
  - deleted 2y ago
    [deleted]
  - dintech 2y ago
    Audio has a lot of buffering behaviour that you wouldn't generally see in event-reactive HFT. Think of all the plugins that you know of that have non-zero latency, compressors with 'lookahead' etc. There are maybe some similarities where the logic is more complex (loop unrolling, SIMD and so on) but I feel like plugins are generally optimizing for throughput (CPU usage) and quality (oversampling etc) rather than purely latency in most cases.
    - ssfrr 2y ago
      I guess the difference I'm interested in is whether HFT tends to be "realtime", in the sense that there's a hard deadline that you need to hit every time. Put another way, audio is operating on a much longer timescale, but cares a lot about worst-case latency (as well as throughput and quality). Is that true of HFT also, or are they more concerned with average latency? If computing a trade takes too long, can they just discard it, or is it catastrophic to miss that deadline?
- atomicnumber3 2y ago
  "which is 3 orders of magnitude faster than a single frame in a video game." You're right on the money. I worked in HFT for half a decade. Back in the late 2000s you were on the cutting edge if you were writing really good C++ and had overclocked some CPUs to hell and back and then shoved them in a rack in a datacenter in new jersey. "To hell and back" means "they only crash every hour or two" (because while they're up, they're faster than your competition). Around the mid 2010's it had moved entirely to FPGAs. Do something smart in software, then give the FPGA something dumb to wait for (e.g. use software to produce a model and wait for some kind of alpha to develop, then tell the FPGA "ok wait for X > Y and fire the order" was basically the name of the game. And it was often better to be faster than it was to be smarter, so really you just kept the FPGA as simple and fast as possible. At that point, your hard realtime constraints for the language doing the smart stuff really dissolve. Compared to an FPGA getting an order out the door in some fraction of a mic, a CPU doing anything is slow. So you might as well use python, yknow? People also play all kinds of different speed games. There's latency races around NJ and chicago where microseconds matter around price movements and C++ isn't really part of the picture in a competitive way anymore, but that's not to say someone isn't ekeing out a niche where they're doing something vaguely smarter faster than opponents. But these things, at scale, tend to be questions of - either your alpha is very short-lived (microseconds, and if it isn't organically microseconds, it will be competed until it is), or it is fundamentally longer-lived (seconds to minutes?) and you might as well use python and develop 100x faster. The silly thing is, the really good trades that focus on short-term alphas (ms's and less) are generally obviously good trades and so you don't have to be super smart to realize it's a really fucking good trade, you had better be fast and go get it. So there's also this kind of built-in bias for being fast because if a trade is so marginally good that you needed a crazy smart and slow model to tell it apart from a bad trade, it's probably still a relatively crummy trade and all your smarts let you do is pick up a few extra pennies for your trouble. I'll close by saying don't take my words as representative of the industry - the trades my firm specialized in was literally a dying breed and my understanding is that most of the big crazy-crazy-fast-microwave-network people have moved to doing order execution anyway, because most of the money in the crazy-crazy-fast game dried up as more firms switched to either DIYing the order execution or paying a former-HFT to do it for them.
  - 2y ago
hawk_ 2y ago
What other low latency projects in public domain are worth learning from?
- nickelpro 2y ago
  Whatever is relevant to the domain you're tackling. How you structurally approach latency in a web server is different than a 3D renderer, which is different than an audio stack, etc. All of those domains have great open source or source available code to learn from, but it wouldn't be useful for me to say "go read HAProxy" if you're never going to touch an HTTP packet. When I'm tackling a problem the first thing I do is research everything everyone else has done on the problem and read their code, benchmark it if possible, and steal all the good ideas. The basic principles never change. Avoid copies, never allocate, keep the hot path local, and really, good codebases should be doing these things anyway. I don't code any differently when I'm writing a command line argument parser vs low-latency RPC servers. It's just a matter of how long I spend tweaking and trying to improve a specific section of code, how willing I am to throw out an entire abstraction I've been working on for perf. In the domain of web stuff, effectively all the major load balancers are good to study. HAProxy, nginx, Envoy. Also anything antirez has ever touched. Application servers are also interesting to study because there's many tricks to learn from a piece of software that needs to interface with something very slow but otherwise wants to be completely transparent in the call graph. FastWSGI is a good example.
mgaunard 2y ago
Latency isn't even as important in HFT as people claim. What's most important is deterministically staying into a reasonable enveloppe (even at the 99.9 percentile) to satisfy real-time requirements and not fall behind. When it's really important, it's implemented in FPGA or with an ASIC.
- ssfrr 2y ago
  This is close to the question I asked above, comparing to audio. If you're processing 32 samples at a time at 48kHz, you need to process a frame every 667us. If you only hit the deadline 99.9% of the time, that's a glitch more than once per second. You'd need a bunch more nines for an acceptable commercial product, so most audio devs just treat it as a hard realtime constraint. I think the main split is whether you care more about average latency or worst-case latency (or some # of nines percentile).
  - mgaunard 2y ago
    In HFT you typically only measure latency for events that you did react on, not all events (at least, in the systems that I've built). That's arguably a bit problematic as events you don't react to can cause you to fall behind, so it's useful to also track how far behind you are from the point where you start processing a packet (which requires hardware timestamping, unfortunately not present on many general-purpose NICs like what's in the cloud). For a given market, you may have less than 1M samples a day (I've even seen some so-called HFT strategies or sub-strategies that only had 300 samples per day). So overall, you'd have fewer events than 44.1kHz, and there are typically expected outliers at the beginning of the data before the system gets warmed up or recovers from bad initial parameters (which I suppose you could just ignore from your distribution, or you could try not to require a warmup). But you're right, you probably want to look at quantiles closer to 1 as well. You're also looking at the max regardless.

_aavaa_ 2y ago

I'd recommend: https://www.computerenhance.com https://www.computerenhance.com The author has a strong game development (engine and tooling) background and I have found it incredibly useful. It also satisfies the requirement for "A genuine fear, hate, and anger, towards unnecessary allocations, copies, and other performance killers."

hi_dang_ 2y ago
Very anecdotal but of the people I know in game studios who are tasked with engine work, and people who make a killing doing FPGA work for HFT firms, both camps shook their head at Casey’s HMH thing. Uniformly I do not know of a single professional developer of this sort of caliber who looked at HMH and thought it looks great. Quite the opposite. I think they found his approach and justifications unsound as it would instil awful practices of premature unfounded optimization and a disdain for normal library code in favour of hand-rolling your own half-baked implementations based on outdated trivia. I agree with them on the basis that HMH exposed an unprepared and inexperienced audience to something that has to be regarded with utmost care. For this, I refer to Jonathan Blow’s presentation of “a list is probably good enough” as an antidote. I think JB’s recommendations are more in line with actual practices, whereas Casey just raised red flags uniformly from here-and-now engine devs shipping multi platform games.
- nickelpro 2y ago
  Casey's a fundamentalist, in the religious extremist sense. Just as a theologian might have something to learn from the most extremely devout, so it is that there is valuable information in Casey's proselytizing. But you should no more follow Casey in form and function than you would any other fundamentalist.
  - _aavaa_ 2y ago
    While gesturing at religion and saying he's a fundamentalist is an easy rhetorical technique, it doesn't add much. Yes good resources do exist, but they are few and far between. The number of new undergraduates from cs or software engineering who learned to think in the ways needed to make low latency applications are a rounding error based on my experience with them. Most forgot it, weren't taught it well, or never saw it at all. They are more worried about learning the syntax of the current hot framework rather than on anything even resembling "Avoid copies, never allocate, keep the hot path local". The religious comparison is also a telling one given the state of the industry; for we aren't the theologian, we're the common folk looking for someone or something to follow in order to write better code. Who are Casey's alternatives? Gesturing the cppcon as a learning resource has "read research papers to learn about a field" vibes. They can be highly informative and worth the effort, but not for beginners. Who are Casey's contemporaries? If he's a fundamentalist then the atheists are nowhere to be seen by the beginners. Instead we have agile ceremony shamans, clean code bible missionaries, tdd doomsday cults, oop/rust/haskell/etc. zealots, a thriving megachurch industry of Lambda schools, and the mother of all cargo cults masquerading as web dev.