3 ms·
> However, people want more throughput. This is where the dragons enter the room. 9/10 times the thing that the business wants to make go faster is almost excl
by bob1029 5y ago
> However, people want more throughput.
This is where the dragons enter the room. 9/10 times the thing that the business wants to make go faster is almost exclusively a serialized narrative of events that has to occur in a certain order.
For problems that do not demand a global serialized narrative of events, we can certainly throw Parallel.ForEach at it and call it a day.
For everything else, you want a ring buffer with spinwait & (micro)batch processing. 100k events per second is paltry compared to what is possible if you take advantage of pipelining and cache in modern CPUs. I have personally tested practical code piles that can handle ~14 million events per second with persistence to disk using only 1 thread the entire time. Impractical academic test cases can get to upwards of half a billion events per second on a single thread:
https://medium.com/@ocoanet/improving-net-disruptor-performance-part-3-introducing-the-valuedisruptor-5b467730bbe https://medium.com/@ocoanet/improving-net-disruptor-performa...
You would think low latency and "batch" processing would not go hand-in-hand, but they certainly do when you are dealing with things at scale and need to take advantage of aggregate effects across all users. This technique also has the effect of substantially reducing jitter. It's amazing what is possible if you can keep everything warmed up.
- dragontamer 5y ago> For everything else, you want a ring buffer with spinwait & (micro)batch processing. Note: orders are only globally defined with single-consumer / single-producer. Which... probably should just be a function-call in most situations. (Notable exception: the consumer wants to stay "hot" in consumer code, and the producer wants to stay "hot" in producer code. 2-threads, one for the producer, one for the consumer). If you even go to single-consumer / multi-producer, then order is no longer defined. Ex: Producer A creates A1, A2, A3, A4. Producer B creates B1, B2, B3. Single-consumer/multi-producer can consume in many orders: A1, B1, A2, B2, A3, A4 for example. Or maybe even A1, A2, A3, A4, B1, B2, B3 is also valid. Or maybe B1, B2, B3, A1, A2, A3, A4. Multi-consumer/multi-producer is even "better". Multi-consumer means that you can "execute" in A4, A3, A2, A1, B3, B2, B1. (Lets say you have 4x consumers, all working on A1, A2, A3, and A4. Well, Consumer1 takes A1, but Consumer1 is slower because L1 cache wasn't warm, and Consumer4 had L1 cache warmed up just right. That means Consumer4 executing A4 will finish first). As such: sequentially-consistent spinwait is good, but suboptimal. The answer is to allow __race-conditions__ to pick the order, because no one cares anymore about ordering if they're asking for multi-consumer/multi-producers. --------- > 100k events per second is paltry compared to what is possible if you take advantage of pipelining and cache in modern CPUs. I agree. But 100k events per second is more than enough for a great number of tasks. There's a degree of simplicity to fork/join or pthread_create / pthread_joins. Especially if you avoid async-style code. Keeping all your logic together (instead of spread out across a variety of async functions) can be beneficial. Sure its suboptimal, but you don't want to make things harder on yourself unless you actually need that performance.
- bob1029 5y ago> I agree. But 100k events per second is more than enough for a great number of tasks. There's a degree of simplicity to fork/join or pthread_create / pthread_joins. Especially if you avoid async-style code. 100%. There is certainly an additional complexity cost to be paid if you want (need) to go beyond 7 figure-per-second numbers. Redefining a problem to fit a ring buffer is a lot harder than throwing basic threading primitives at it.