4 ms·
A thousand nanoseconds per packet? Luxury! At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount
by ncmncm 6y ago
A thousand nanoseconds per packet? Luxury!
At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount that can be done in that time is limited. Typically, you just dump them in a big-ass ring buffer and let processes on a bunch of other cores pick out interesting ones to spend more time on.
The way to get the packets is usually with some proprietary kernel-bypass library (some have success with DPDK), and a very carefully isolated core that the kernel is kept from interrupting, and just spins moving packets. NIC hardware-level ring buffers are usually pretty small, so pauses make for drops.
Many NICs can be persuaded to filter packets out to multiple ring buffers, and then you can get multiple cores picking them off; that is necessary when you get to 40 and 100 Gbps. Multiple rings even with one core multiplies headroom, making occasional housekeeping without drops possible.
- londons_explore 6y agoSounds to me like you should tell the hardware manufacturers to make bigger ring buffers, or even to provide an indexed ring buffer which indexes by a user-defined part of the packet...
- ncmncm 6y agoI have found limited success dictating to hardware manufacturers. I would be satisfied if the kernel could just be persuaded to leave my cores the hell alone without an endless list of boot flags.
- LunaSea 6y agoDo you have links about the flags that you mention or were you referring to process pining? Have you tried out the new eBPF libraries and kTLS?
- ncmncm 6y agoSearch isolcpus, nohz_full, and rcu_nocbs. Process pinning & isolation from other processes is the first step; keeping kernel interruptions out is the hard part. Kernel people generally think anything they do is implicitly more important than every single thing you do. eBPF and kTLS tend to involve the kernel in your business. But, eBPF can put code directly on certain NICs (e.g. Netronome), writing directly to user process memory. Then there is no need to pin or isolate, because the action is on the NIC, out of reach.
- lifty 6y agoAnd then some server vendors insist on on doing some out of bound interrupts from their management engines, which introduce odd latency spikes that are hard to debug.
- jkulubya 6y agoExcuse my naïveté, but I’d love to know what kind of feeds you connect to to pull in 40gbps of financial data. Are these basically feeds of all available securities trading everywhere in the world? Background is that I did some work connecting to a few smaller exchanges, but we never ever thought we’d ever need to go past “just” 20mbps.
- ncmncm 6y ago> Are these basically feeds of all available securities trading everywhere in the world? Well, yes. In March, the per-day volumes went to more than double the previous peak. That said, traffic is bursty, particularly at market open and close, and latency matters, so you have to statically provision for peak traffic.
- 7532yahoogmail 6y agoDon't get too impressed here by finance. Almost everything we're talking about here Linux Kernel Developments, SolarFlare (or Mellonx), core isolation (cpu set) was invented elsewhere. Finance merely buys it; it's a great client of all this tech. The only vaguely created-in-finance thing he/she references but then not even by name is https://martinfowler.com/articles/lmax.html https://martinfowler.com/articles/lmax.html which comes from ring buffers coming from CS EE networking itself.
- ncmncm 6y agoNever heard of LMAX; looks like a Java thing. Nobody said finance invented anything. But high performance demands produce similar architectures everywhere.
- 7532yahoogmail 6y agoLMAX is a java thing. It's high speed, low latency messaging system used a betting facility in the UK somewhere where in certain betting situations, the number of messages drives up explosively high, and need to be well handled. Thus high, high speed and low latency with reliably low latency were key.
- matdehaast 6y agoDo you have any links/resources I can read more about this? Sounds crazy interesting!
- ncmncm 6y agohttps://news.ycombinator.com/item?id=21011976 https://news.ycombinator.com/item?id=21011976
- matdehaast 6y agoThanks thats amazing! One question I don't see addressed in that thread.How do you deal with failures of the node? Do you use active-active type replication or some other strategy?
- ncmncm 6y agoNormally there are two or more entirely separate machines getting and recording the same multicast traffic streams. For active trading, likewise, but maybe when one dies, the other is then allowed bigger trades? It is usually best to keep systems as decoupled as possible, and to make possible responses to failure as few and simple as possible.