3 ms·
I generally fix performance problems nowadays by eliminating threads and queues, particularly "lock-free" queues, and splitting out work to whole separate singl
by ncmncm 6y ago
I generally fix performance problems nowadays by eliminating threads and queues, particularly "lock-free" queues, and splitting out work to whole separate single-thread processes that communicate only via ring buffers in mmapped hugepages.
The freedom to kill and restart the programs as needed makes a huge difference in ease of managing a system. I use kernel-bypass networking and "isolcpus" so the time-critical code never does any system calls. The programs do all their slow, mallocy OS-coupled work at startup, and then never again except to log errors or when shutting down. They mmap structs onto files under /dev/shm/ for status reporting, mostly just incrementing counters that another program, running every few seconds, formats into snapshot reports logged and displayed in grafana.
It is not the right architecture for everything, but it captures to disk all the activity of all the stock and options market exchanges in New York and Chicago using just a single 1U server attached to a 4U disk array.
"Lock-free" operations are not really lock-free at all; they involve atomic bus operations, which just do locking in hardware similarly to what you would otherwise have done. So, to get performance, you minimize interactions enough that the time wasted per interaction is negligible, and batch work so that the branch predictors and caches can get well primed and then used for a stretch. Atomic operations when you are the only writer, and so already own the cache lines, are cheap.