5 ms·

Am I understanding correctly that those firms really really care about latency & have unlimited money & chose C++/JAVA with a standard off-the-shelve compiler +

Am I understanding correctly that those firms really really care about latency & have unlimited money & chose C++/JAVA with a standard off-the-shelve compiler + an OS and/or JVM?? If you'd really really care about latency and have the resources, wouldn't it be more effective to just go bare metal? Bare metal as in: no OS (talk to the hardware registers directly from code), possibly a custom compiler, use all the relevant instructions that your CPU offers you. Or at the very least mess with the compiler to optimize it more to the specific make/model of the CPU/hardware that it will run on? What are the considerations in those companies for or against this?

layoutIfNeeded 6y ago
I used to work at a market maker where they used FPGAs to beat HFTs. Of course it was a constant arms race, so I’m not sure what they would be using nowadays...
- bboygravity 6y ago
  I would imagine they switched to ASICs which are more optimized than FPGAs (much smaller feature size, less general reconfigurable bloat, higher clocks, etc).
  - vgatherps 6y ago
    Only a few have, and even then only for a few fixed things. They have extreme versions of the inflexibility problems that fpgas have, and few exchanges have low enough jitter to justify ASICS over fpgas. It might surprise people that relatively few hfts do latency arb (and the ones that do often have some other structural advantage), and at that the ones are more and more mixing in serious short-term alpha. In that context speed is simply becoming table stakes for playing the game, and not the one true race to win it all.
scatters 6y ago
I don't know about java, but when using c++ you can use the OS to start the program, set up the hardware mappings and then move it to an isolated core where you talk to the hardware directly without ever issuing a system call. Compilers already offer flags to optimize for specific processor generations (and use the newly available instructions); you aren't going to be able to do better than that with a custom compiler.
- bboygravity 6y ago
  Oh, that's cool. However, you'd still be sharing at least 1 core (and all peripheral hardware and memory access in the system) with the OS. A core, other hardware and memory that you could otherwise access directly from your application without having to wait for the OS to finish its business? I'm not sure whether it's worth it to go through this trouble (writing for bare metal vs just using an off-the-shelve OS) in a real life trading-app scenario. Hence my wonder :)
  - scatters 6y ago
    Accessing other hardware isn't latency-sensitive, though; that's usually disk and the internal network, for logging, telemetry and control. Typically a trading application will keep a thread running on the shared core and send messages between that and its dedicated core thread using lock-free queues, or simple atomics. As you say, writing for bare metal is a lot of hassle and it's unnecessary for the parts of a trading application that don't need to be super fast; using a commodity OS and isolating resources gives the best of both worlds.
- funcDropShadow 6y ago
  The jit compiler of typical jvms automatically uses the instruction sets (with limits afaik) that are available on the hardware that it is running on.
- sakex 6y ago
  I've never seen that, how do you do it? Also, how do you deal with VM and scheduling?
  - ericbarrett 6y ago
    Brief intro, should bootstrap your knowledge: https://codywu2010.wordpress.com/2015/09/27/isolcpus-numactl-and-taskset/ https://codywu2010.wordpress.com/2015/09/27/isolcpus-numactl...
  - scatters 6y ago
    Isolating a core: isolcpus and taskset, or possibly cgroups, plus nohz and there's a patch set that offers even stronger isolation guarantees (keeping kernel threads off the core). Hardware mappings are done as a ring buffer in memory mapped to the device; a kernel module helps set up that mapping but once it's in place reads and writes to those virtual addresses go directly (well - via MMU and memory bus) to the hardware. The general term for this is user-space networking (although obviously you can do it with other hardware as well). Some addresses may be mapped to device registers that trigger hardware actions on read/write. VM - the key is to get everything mapped in at the start and avoid page faults subsequently. madvise can help with this and obviously you need to avoid memory leaks that could result in sbrk - but in any case allocating at all in the trading thread is generally unnecessary (after startup) and frowned on. This does mean that the (lock-free) queue back to the shared core needs to recycle memory and both sides need to service it regularly or you need a strategy to deal with exhaustion. Scheduling isn't an issue - you have a single thread per core and (with isolcpus) the kernel scheduler knows to avoid placing other threads on it.
chrisseaton 6y ago
I don’t know what you think the practical benefit of running with no OS is?
- bboygravity 6y ago
  More processing capacity for your application (no need to share with OS), no more interruptions by the OS that are unpredictable in duration, no more interruptions by the OS that are unpredictable in terms of timing and priority and above all, since this seems to be the main goal of the software we're talking about: lower overall latency. Standard C++/JAVA (as far as I know) talks to hardware through a software layer (the OS's system calls). This takes time. Time you can save by cutting out the middle man. There are apparently ways to do this with an OS running in parallel as well though when you have multiple cores available apparently (see another comment to this thread).
  - chrisseaton 6y ago
    You can get all this by pinning to a core and DMA to hardware. Think of the OS like a library - if you don't call it and don't ask it to call you it stays out of the way. There's no need to throw away the OS.
    - vgatherps 6y ago
      This is exactly what is done. OS’s are really nice to have around and will stay out of the way. In the day and age of fpgas no reason to ruin the software side to maybe kind of sort of get a bit faster in your software?
- toolslive 6y ago
  - raw, unshared access to devices. - no more interrupts. You can get really close to this via virtualization though. Run a guest machine without operating system on the hypervisor. It doesn't even have to be low level code to get most of the benefits (for example MirageOS)
  - 6y ago
bluGill 6y ago
There is only a limit amount of money to make. If your algorithm generates 100k in gross profit a year that doesn't even pay for the programmers to maintain it, unless they can do other things as well. (I'd guess common, there are a lot of algorithms) the sum total doesn't pay. Where it does pay you are better off spending money on fpga, or even asic.
fma 6y ago
Well...the article says: "If you have an unlimited amount of time and resources, the best solution for speed will be coded in FPGA," says Lawrey, referring to the language that directly codes field-programmable gate arrays. "If you still have a lot of time and resources and you want to create multiple exchange adaptors, you'll choose C++. - But if you want to engage with 20+ exchanges, to go to market quickly, and implement continuous performance tuning, you'll choose Java."

layoutIfNeeded 6y ago

I used to work at a market maker where they used FPGAs to beat HFTs. Of course it was a constant arms race, so I’m not sure what they would be using nowadays...

bboygravity 6y ago
I would imagine they switched to ASICs which are more optimized than FPGAs (much smaller feature size, less general reconfigurable bloat, higher clocks, etc).
- vgatherps 6y ago
  Only a few have, and even then only for a few fixed things. They have extreme versions of the inflexibility problems that fpgas have, and few exchanges have low enough jitter to justify ASICS over fpgas. It might surprise people that relatively few hfts do latency arb (and the ones that do often have some other structural advantage), and at that the ones are more and more mixing in serious short-term alpha. In that context speed is simply becoming table stakes for playing the game, and not the one true race to win it all.

scatters 6y ago

I don't know about java, but when using c++ you can use the OS to start the program, set up the hardware mappings and then move it to an isolated core where you talk to the hardware directly without ever issuing a system call. Compilers already offer flags to optimize for specific processor generations (and use the newly available instructions); you aren't going to be able to do better than that with a custom compiler.

bboygravity 6y ago
Oh, that's cool. However, you'd still be sharing at least 1 core (and all peripheral hardware and memory access in the system) with the OS. A core, other hardware and memory that you could otherwise access directly from your application without having to wait for the OS to finish its business? I'm not sure whether it's worth it to go through this trouble (writing for bare metal vs just using an off-the-shelve OS) in a real life trading-app scenario. Hence my wonder :)
- scatters 6y ago
  Accessing other hardware isn't latency-sensitive, though; that's usually disk and the internal network, for logging, telemetry and control. Typically a trading application will keep a thread running on the shared core and send messages between that and its dedicated core thread using lock-free queues, or simple atomics. As you say, writing for bare metal is a lot of hassle and it's unnecessary for the parts of a trading application that don't need to be super fast; using a commodity OS and isolating resources gives the best of both worlds.
funcDropShadow 6y ago
The jit compiler of typical jvms automatically uses the instruction sets (with limits afaik) that are available on the hardware that it is running on.
sakex 6y ago
I've never seen that, how do you do it? Also, how do you deal with VM and scheduling?
- ericbarrett 6y ago
  Brief intro, should bootstrap your knowledge: https://codywu2010.wordpress.com/2015/09/27/isolcpus-numactl-and-taskset/ https://codywu2010.wordpress.com/2015/09/27/isolcpus-numactl...
- scatters 6y ago
  Isolating a core: isolcpus and taskset, or possibly cgroups, plus nohz and there's a patch set that offers even stronger isolation guarantees (keeping kernel threads off the core). Hardware mappings are done as a ring buffer in memory mapped to the device; a kernel module helps set up that mapping but once it's in place reads and writes to those virtual addresses go directly (well - via MMU and memory bus) to the hardware. The general term for this is user-space networking (although obviously you can do it with other hardware as well). Some addresses may be mapped to device registers that trigger hardware actions on read/write. VM - the key is to get everything mapped in at the start and avoid page faults subsequently. madvise can help with this and obviously you need to avoid memory leaks that could result in sbrk - but in any case allocating at all in the trading thread is generally unnecessary (after startup) and frowned on. This does mean that the (lock-free) queue back to the shared core needs to recycle memory and both sides need to service it regularly or you need a strategy to deal with exhaustion. Scheduling isn't an issue - you have a single thread per core and (with isolcpus) the kernel scheduler knows to avoid placing other threads on it.

chrisseaton 6y ago

I don’t know what you think the practical benefit of running with no OS is?

bboygravity 6y ago
More processing capacity for your application (no need to share with OS), no more interruptions by the OS that are unpredictable in duration, no more interruptions by the OS that are unpredictable in terms of timing and priority and above all, since this seems to be the main goal of the software we're talking about: lower overall latency. Standard C++/JAVA (as far as I know) talks to hardware through a software layer (the OS's system calls). This takes time. Time you can save by cutting out the middle man. There are apparently ways to do this with an OS running in parallel as well though when you have multiple cores available apparently (see another comment to this thread).
- chrisseaton 6y ago
  You can get all this by pinning to a core and DMA to hardware. Think of the OS like a library - if you don't call it and don't ask it to call you it stays out of the way. There's no need to throw away the OS.
  - vgatherps 6y ago
    This is exactly what is done. OS’s are really nice to have around and will stay out of the way. In the day and age of fpgas no reason to ruin the software side to maybe kind of sort of get a bit faster in your software?
toolslive 6y ago
- raw, unshared access to devices. - no more interrupts. You can get really close to this via virtualization though. Run a guest machine without operating system on the hypervisor. It doesn't even have to be low level code to get most of the benefits (for example MirageOS)
- 6y ago

bluGill 6y ago

There is only a limit amount of money to make. If your algorithm generates 100k in gross profit a year that doesn't even pay for the programmers to maintain it, unless they can do other things as well. (I'd guess common, there are a lot of algorithms) the sum total doesn't pay. Where it does pay you are better off spending money on fpga, or even asic.

fma 6y ago

Well...the article says: "If you have an unlimited amount of time and resources, the best solution for speed will be coded in FPGA," says Lawrey, referring to the language that directly codes field-programmable gate arrays. "If you still have a lot of time and resources and you want to create multiple exchange adaptors, you'll choose C++. - But if you want to engage with 20+ exchanges, to go to market quickly, and implement continuous performance tuning, you'll choose Java."