5 ms·

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It

I built a PoC of a 5us trading system (guaranteed 5us response in every situation) for a brokerage house a long time ago, around the time of LMAX Disruptor. It was one man job and I had to start with nothing (they had no knowledge at all). Fun project and I learned a lot. * full kernel bypass (I even implemented driver for the networking hardware) * everything that could disrupt the application disabled (like SME interrupts, etc.) Memory mapped as huge buffers to prevent tlb lookup failures, etc. * application consists of threads pinned to specified cores * each thread on the path of market data to sending the order is never calling the operating system for anything. When not processing anything it is busy spinning. * all memory preallocated carefully to have it pinned to the local core * data flows from the networking hardware into one core and then passess through cores using disruptor, each core doing further processing and publishing signals to the next core * the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange.

vivzkestrel 3y ago
not gonna lie, i have to literally look up the meanings of half the words in your post lol, but you must have been in my position sometime. What did you have to learn in order to build this
vivzkestrel 3y ago
what language do you think would be the backbone for such a system? C/C++/Golang or something high level like node.js/Java
- defrost 3y ago
  Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it. The major features for performance are to allocate and manage all memory from the start .. determine your thresholds of performance and put everything required together on cold start so as to avoid any thrashing at runtime.
  - onetimeuse92304 3y ago
    You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, but some of that code would be written in C and some of it would be compiled at runtime with Lisp. Parsing the incoming multicast feed from the exchange was implemented with Lisp emitting super efficient machine code (almost no function calls) based on a bunch of XML files describing the messages. I shamelessly stole the idea from the book Practical Common Lisp (really good if you want to get into Lisp). Things like business rules would be compiled into decision trees using current market situation and decision trees would be reorganised, optimised and compiled into machine code and inserted into the processing path. Most of the time a very large decision tree with hundreds of complex decisions (for example, taking into account market volatility) could be distilled to just couple branching instructions. This decision tree compilation would happen after every market signal, up to 10 thousand times a second (the exchange had a basic tick of 1/10000 of a second giving me guaranteed 100us before the next market signal). Same with actual algorithms -- I wrote a small DSL for the traders and this DSL would be compiled with Lisp to machine an inserted into the processing path. Some parts of the framework would be built with Lisp, too. It was easier for me to write a DSL and then compile it to machine than to write it in C. If you want to understand one principle from all this is to look at all instructions and especially branches that you have between receiving the signal and emitting the order and try to figure out if you can eliminate it and if you can't, try to find ways to do it ahead of time even if doing it ahead of time requires a lot more effort.
    - defrost 3y ago
      Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable cancelation, etc. Very similar architectures in many ways - with rolling DSP filters in place of trading response algorithms .. etc. Lisp -> ASM or Lisp -> C either way code generation from a higher language was a good way to get the heavy lifting done.
- marginalia_nu 3y ago
  I wouldn't bundle C with Golang and Java with Node. Golang and Java are roughly a tier on their own. Java is definitely a thing in this space[1], although it's a fairly unidiomatic style of Java that reduces allocations and puts a lot of emphasis on consistent low latency. [1] e.g. https://marketswiki.com/wiki/TRADExpress https://marketswiki.com/wiki/TRADExpress ; though they've been absorbed into Nasdaq now and my visibility into CFT ended with that, no idea how much of their software is still running
  - pjmlp 3y ago
    Hopefully Panama will help to make it better, and there is still hope for Valhala.
jiggawatts 3y ago
> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange. I saw a talk about this dialed up to eleven: the entire processing occurred in a "smart NIC" instead of the CPU. The response would start getting sent even as the inbound packet was still being received. The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. Before that talk I couldn't figure out why there was a market for NICs with embedded FPGAs, CPUs and memory. Day traders basically subsidised these things, and now they do efficient packet switching for large cloud providers. Reminds me of how crypto-mining subsidised a lot of GPU development, and now we have 4K ray tracing and AIs thanks to that.
- onetimeuse92304 3y ago
  Cool! I actually wasn't aware about NICs with FPGAs on them. You learn something new every day on HN. My solution wasn't as fast and it could never do what you describe (start sending bytes before the packet was fully received). The market signal messages were actually batched together (usually one to 5), compressed with zlib and sent as a single multicast packet.
- imtringued 3y ago
  You could in principle accelerate the most CPU intensive parts of Webservers with smart NICs. gzip, TLS, JSON serdes, html templates. There are also accelerators for databases, leaving just the business logic to be executed on the CPU.
- reisse 3y ago
  > The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. This trick will get you banned on some of exchanges now :) Another one, which is public knowledge for years, and also often penalized, is to send TCP fragment with header of the message well in advance, "booking" place in the queue. Then send the finishing fragment with the real order after doing all the calculations.
bnastic 3y ago
> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange Ah those nasty market opens in the morning and trying to get a good spot in the queue
bitcharmer 3y ago
Appreciate you describing this as a PoC because in reality it's impossible to do <5us tick to trade in software including risk, deal booking, drop copy and a tonne of other components that go into a real-life trading system. In fact, you can't guarantee 5us for anything, at least not on common operating systems. You would have to run your code with the interrupt flag cleared to prevent any IPIs or hrticks getting in the way. But that would be opening a scary can of worms.
- onetimeuse92304 3y ago
  > Appreciate you describing this as a PoC because in reality it's impossible to do <5us tick to trade in software including risk, deal booking, (...) Absolutely not true. I explained, risk is calculated concurrently and compiled to decision tree and then inserted in the path in the form of "if X happens do Y". The fact it was PoC has nothing to do with it, in fact it showed that yes, it is possible to do so. That's entire point of the PoC.
- reisse 3y ago
  It is absolutely possible, I've seen multiple such systems. > In fact, you can't guarantee 5us for anything, at least not on common operating systems. Oh yes you can, with a screen-sized kernel cmdline, some proper configuration, both hardware and software, and a bit of luck.
- FridgeSeal 3y ago
  > you can't guarantee 5us for anything, at least not on common operating systems I don’t doubt it, but from my reading of OP’s post, it sounds they were skipping most of the kernel and OS.
  - onetimeuse92304 3y ago
    Yes. You really don't want your real time threads to have anything to do with the OS after startup sequence. Significant part of the project was learning about various ways a piece of code can be preempted. Once you get it done, it really is all really nice and predictable. And also a fire hazard because you have disabled things like SMI interrupts that are used by the CPU to figure out if it is overheating...
vbezhenar 3y ago
Why do you need OS for that kind of project? If you're implementing network drivers and avoiding scheduler, you could just run your code on bare metal.
- gpderetta 3y ago
  Because it is convenient being able to SSH to your trading machine and run standard Unix applications for deployments, normal start of day/end of day processing, diagnostics, profiling, etc. The overhead, during normal operations can be significantly reduced.

vivzkestrel 3y ago

not gonna lie, i have to literally look up the meanings of half the words in your post lol, but you must have been in my position sometime. What did you have to learn in order to build this

vivzkestrel 3y ago

what language do you think would be the backbone for such a system? C/C++/Golang or something high level like node.js/Java

defrost 3y ago
Each to their own but if you read and understand the comment above they're describing a dedicated OS for the task .. so think about what you'd choose to write a small task dedicated OS with. Simple C is most likely, ASM is possible, a language such as OCaml generating C to hook into the low level buffers would be intriguing ... the list is long and largely determined by the experience preference of whoever tackles it. The major features for performance are to allocate and manage all memory from the start .. determine your thresholds of performance and put everything required together on cold start so as to avoid any thrashing at runtime.
- onetimeuse92304 3y ago
  You are surprisingly accurate. I used combination of Common Lisp (SBCL), ANSI C and assembly (not much assembly, though, only very small pieces that I had trouble emitting other ways). The main application was in Lisp, it would start up and set up the environment (using some low level code written in C/assembly). But everything on the path of the market signal to order would be highly optimised native machine code, but some of that code would be written in C and some of it would be compiled at runtime with Lisp. Parsing the incoming multicast feed from the exchange was implemented with Lisp emitting super efficient machine code (almost no function calls) based on a bunch of XML files describing the messages. I shamelessly stole the idea from the book Practical Common Lisp (really good if you want to get into Lisp). Things like business rules would be compiled into decision trees using current market situation and decision trees would be reorganised, optimised and compiled into machine code and inserted into the processing path. Most of the time a very large decision tree with hundreds of complex decisions (for example, taking into account market volatility) could be distilled to just couple branching instructions. This decision tree compilation would happen after every market signal, up to 10 thousand times a second (the exchange had a basic tick of 1/10000 of a second giving me guaranteed 100us before the next market signal). Same with actual algorithms -- I wrote a small DSL for the traders and this DSL would be compiled with Lisp to machine an inserted into the processing path. Some parts of the framework would be built with Lisp, too. It was easier for me to write a DSL and then compile it to machine than to write it in C. If you want to understand one principle from all this is to look at all instructions and especially branches that you have between receiving the signal and emitting the order and try to figure out if you can eliminate it and if you can't, try to find ways to do it ahead of time even if doing it ahead of time requires a lot more effort.
  - defrost 3y ago
    Not my first rodeo :-) I too made a real time hardware level trading system back in the day, on the back of building a multi channel seismic aquisition system with a custom RealtimeOS talking to a bunch of DSP cards that each sampled a trailing flotation cable that each had multiple microphones with the entire grid going toward building up a profile of the seafloor and layers underneath along with bobbing boat|cable cancelation, etc. Very similar architectures in many ways - with rolling DSP filters in place of trading response algorithms .. etc. Lisp -> ASM or Lisp -> C either way code generation from a higher language was a good way to get the heavy lifting done.
marginalia_nu 3y ago
I wouldn't bundle C with Golang and Java with Node. Golang and Java are roughly a tier on their own. Java is definitely a thing in this space[1], although it's a fairly unidiomatic style of Java that reduces allocations and puts a lot of emphasis on consistent low latency. [1] e.g. https://marketswiki.com/wiki/TRADExpress https://marketswiki.com/wiki/TRADExpress ; though they've been absorbed into Nasdaq now and my visibility into CFT ended with that, no idea how much of their software is still running
- pjmlp 3y ago
  Hopefully Panama will help to make it better, and there is still hope for Valhala.

jiggawatts 3y ago

> the main insight was that rather than wait for market signals to then decide what to do, you can precalculate your responses up to and including the actual message to be sent to the exchange. I saw a talk about this dialed up to eleven: the entire processing occurred in a "smart NIC" instead of the CPU. The response would start getting sent even as the inbound packet was still being received. The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. Before that talk I couldn't figure out why there was a market for NICs with embedded FPGAs, CPUs and memory. Day traders basically subsidised these things, and now they do efficient packet switching for large cloud providers. Reminds me of how crypto-mining subsidised a lot of GPU development, and now we have 4K ray tracing and AIs thanks to that.

onetimeuse92304 3y ago
Cool! I actually wasn't aware about NICs with FPGAs on them. You learn something new every day on HN. My solution wasn't as fast and it could never do what you describe (start sending bytes before the packet was fully received). The market signal messages were actually batched together (usually one to 5), compressed with zlib and sent as a single multicast packet.
imtringued 3y ago
You could in principle accelerate the most CPU intensive parts of Webservers with smart NICs. gzip, TLS, JSON serdes, html templates. There are also accelerators for databases, leaving just the business logic to be executed on the CPU.
reisse 3y ago
> The go/no-go decision was effectively just sending the final CRC bytes correctly or deliberately incorrectly, thus invalidating the outbound packet that was already 99% sent. This trick will get you banned on some of exchanges now :) Another one, which is public knowledge for years, and also often penalized, is to send TCP fragment with header of the message well in advance, "booking" place in the queue. Then send the finishing fragment with the real order after doing all the calculations.

bnastic 3y ago

bitcharmer 3y ago

Appreciate you describing this as a PoC because in reality it's impossible to do <5us tick to trade in software including risk, deal booking, drop copy and a tonne of other components that go into a real-life trading system. In fact, you can't guarantee 5us for anything, at least not on common operating systems. You would have to run your code with the interrupt flag cleared to prevent any IPIs or hrticks getting in the way. But that would be opening a scary can of worms.

onetimeuse92304 3y ago
> Appreciate you describing this as a PoC because in reality it's impossible to do <5us tick to trade in software including risk, deal booking, (...) Absolutely not true. I explained, risk is calculated concurrently and compiled to decision tree and then inserted in the path in the form of "if X happens do Y". The fact it was PoC has nothing to do with it, in fact it showed that yes, it is possible to do so. That's entire point of the PoC.
reisse 3y ago
It is absolutely possible, I've seen multiple such systems. > In fact, you can't guarantee 5us for anything, at least not on common operating systems. Oh yes you can, with a screen-sized kernel cmdline, some proper configuration, both hardware and software, and a bit of luck.
FridgeSeal 3y ago
> you can't guarantee 5us for anything, at least not on common operating systems I don’t doubt it, but from my reading of OP’s post, it sounds they were skipping most of the kernel and OS.
- onetimeuse92304 3y ago
  Yes. You really don't want your real time threads to have anything to do with the OS after startup sequence. Significant part of the project was learning about various ways a piece of code can be preempted. Once you get it done, it really is all really nice and predictable. And also a fire hazard because you have disabled things like SMI interrupts that are used by the CPU to figure out if it is overheating...

vbezhenar 3y ago

Why do you need OS for that kind of project? If you're implementing network drivers and avoiding scheduler, you could just run your code on bare metal.

gpderetta 3y ago
Because it is convenient being able to SSH to your trading machine and run standard Unix applications for deployments, normal start of day/end of day processing, diagnostics, profiling, etc. The overhead, during normal operations can be significantly reduced.