3 ms·

The trick isn't "use a bunch of threads" but to have each cpu do a separate task (N1 threads for accept, N2 threads for read, N3 threads for write, N4 threads f

by geocar 11y ago

The trick isn't "use a bunch of threads" but to have each cpu do a separate task (N1 threads for accept, N2 threads for read, N3 threads for write, N4 threads for close, etc) and using a separate keventfd/epollfd for each thread. The tasks can still use blocking calls (or nonblocking, as they like), which makes it different than strategy 1 or 2. It shards by task which is different than 3, or 4. Completion-notification is not explicit. It is clearly different than 5 or 6. I think you should read it carefully. Popular webservers like nginx and varnish do not implement this method, so I do not believe it is common, and there are a few neat features of this method: 1. Tasks will usually fit into cache. If the kernel hotpaths to IP they usually won't fall out of cache either. This can produce 100x or even 1000x speed improvements for complex tasks. 2. It may be easy to convert an existing program to this design than to replace everything with nonblocking IO, or to move into kernel space, or to attach to a userland TCP stack. These things are important if you want to write faster servers.

cbsmith 11y ago
What if you have to map accept, read, write & close (so four types of tasks) to a six cores? ;-) Yes, nginx & varnish have different approaches. Their approaches tend to work better for their type of application. One thread or process per core, with its own event loop, is indeed a very standard pattern. The process-per-core style approaches tend to shard per fd (e.g., nginx), while the thread-per-core style tend to either shard by IO request or don't shard at all (each thread has an independent poll of all outstanding IOs). Particularly if you use a continuations style paradigm, it makes it really easy to convert over existing code to the design. In practice, you will typically have better cache locality with the fd than your position in the state machine (and the model you are describing isn't quite where you are in the state machine, but you can adjust it to fit). If your state machine is simple enough, your approach might get better instruction cache hit rate, but you are basically guaranteeing that data has to move between the processor caches in very rapid succession. In practice, not only are you forcing the fd's to be passed between processor caches, but there tends to be data locality with that fd. While it is easy to have the instruction cache replicated across all processors, the data mutates quite rapidly. Again, this stuff is covered in the discussions on the c10k site.

by geocar 11y ago

cbsmith 11y ago

What if you have to map accept, read, write & close (so four types of tasks) to a six cores? ;-) Yes, nginx & varnish have different approaches. Their approaches tend to work better for their type of application. One thread or process per core, with its own event loop, is indeed a very standard pattern. The process-per-core style approaches tend to shard per fd (e.g., nginx), while the thread-per-core style tend to either shard by IO request or don't shard at all (each thread has an independent poll of all outstanding IOs). Particularly if you use a continuations style paradigm, it makes it really easy to convert over existing code to the design. In practice, you will typically have better cache locality with the fd than your position in the state machine (and the model you are describing isn't quite where you are in the state machine, but you can adjust it to fit). If your state machine is simple enough, your approach might get better instruction cache hit rate, but you are basically guaranteeing that data has to move between the processor caches in very rapid succession. In practice, not only are you forcing the fd's to be passed between processor caches, but there tends to be data locality with that fd. While it is easy to have the instruction cache replicated across all processors, the data mutates quite rapidly. Again, this stuff is covered in the discussions on the c10k site.