4 ms·

Fast-Servers

122 points by tosh 8mo ago

kogus 8mo ago
Slightly tangential, but why is the first diagram duplicated at .1 opacity?
- tecleandor 8mo ago
  That plus the ellipsis makes me thing that it means the additional threads that would open for next connections...
  - kogus 8mo ago
    Ah, that makes sense.
ratrocket 8mo ago
discussed in 2016: https://news.ycombinator.com/item?id=10872209 https://news.ycombinator.com/item?id=10872209 (53 comments)
lmz 8mo ago
Seems similar to the SEDA architecture https://en.wikipedia.org/wiki/Staged_event-driven_architecture https://en.wikipedia.org/wiki/Staged_event-driven_architectu...
bee_rider 8mo ago
> One thread per core, pinned (affinity) to separate CPUs, each with their own epoll/kqueue fd > Each major state transition (accept, reader) is handled by a separate thread, and transitioning one client from one state to another involves passing the file descriptor to the epoll/kqueue fd of the other thread. So this seems like a little pipeline that all of the requests go through, right? For somebody who doesn’t do server stuff, is there a general idea of how many stages a typical server might be able to implement? And does it create a load-balancing problem? I’d expect some stages to be quite cheap…
- marcosdumay 8mo ago
  > For somebody who doesn’t do server stuff, is there a general idea of how many stages a typical server might be able to implement? On the HTTP server from the article, what I understood is that those 2 you are seeing are the ones you have. Or maybe 3, if disposing of things is slow. I'm not sure what I prefer. On one hand, there's some expensive coordination for passing those file descriptors around. On the other hand, having some separate code bother with creating and closing the connections make it easier to focus on the actual performance issues where they appear, and create opportunity to dispatch work smartly. Of course, you can go all the way in and make a green threads server where every bit of IO puts the work back on the queue. But you would use a single queue then, and dispatch the code that works on it. So you get more branching, but less coordination.
luizfelberti 8mo ago
A bit dated in the sense that for Linux you'd probably use io_uring nowadays, but otherwise it's a timeless design Still, I'm conflicted on whether separating stages per thread (accept on one thread and the client loop in another) is a good idea. It sounds like the gains would be minimal or non-existent even in ideal circumstances, and on some workloads where there's not a lot of clients or connection churn it would waste an entire core for handling a low-volume event. I'm open to contrarian opinions on this though, maybe I'm not seeing soemthing...
- eklavya 8mo ago
  It is not a good idea, especially with the new chiplet/CCX processors.
- raggi 8mo ago
  It’s not a good idea and that’s where I’d really start with the dated commentary here rather than focusing on the polling mechanism. It depends on the application but if the buffers are large (>=64kb) such as a common TCP workload then uring won’t necessarily help that much. You’ll gain a lot of scalability regardless of polling mechanism by making sure you can utilize rss and xss optimizations.
  - wild_egg 8mo ago
    It's been a while but why is uring not helpful for larger buffers? I'd think the zero-copy I/O capabilities would make it more helpful for larger payloads, not less
    - Veserv 8mo ago
      uring supports zero-copy, but is not a copy-reduction mechanism; it is a syscall-reduction mechanism. Large buffers mean less syscalls to start with, so less benefit.
      - raggi 8mo ago
        Exactly this. The kernel alloc’d buffers can help but if that was a primary concern you’re in driver territory. Anything still userspace kind of optimization domain the portion of syscalls for large buffers in a buffered flow is heavily amortized and not overly relevant.
fao_ 8mo ago
this is more or less, in some way, what Erlang does and how Erlang is so easy to scale.
epicprogrammer 8mo ago
It’s an interesting throwback to SEDA, but physically passing file descriptors between different cores as a connection changes state is usually a performance killer on modern hardware. While it sounds elegant on a whiteboard to have a dedicated 'accept' core and a 'read' core, you end up trading a slightly simpler state machine for massive L1/L2 cache thrashing. Every time you hand off that connection, you immediately invalidate the buffers and TCP state you just built up. There’s a reason the industry largely settled on shared-nothing architectures like NGINX having a single pinned thread handle the entire lifecycle of a request keeps all that data strictly local to the CPU cache. When you're trying to scale, respecting data locality almost always beats pipeline cleanliness.
- toast0 8mo ago
  You could presumably have an acceptor thread per core, which passes the fds to core alligned next thread, etc. That would get you the code simplicity benefits the article suggests, while keeping the socket bound to a single core, which is definitely needed. Depending on if you actually need to share anything, you could do process per core, thread per loop, and you have no core to core communication from the usual workings of the process (i/o may cross though)
  - scottlamb 8mo ago
    I don't think the author intended "code simplicity" as an end unto itself but a way to reduce cache pressure. He popped into the 2016 discussion [1] to say: > Another benefit of this design overlooked is that individual cores may not ever need to read memory -- the entire task can run in L1 or L2. If a single worker becomes too complicated this benefit is lost, and memory is much much slower than cache. I think this is wrong or at least overstated: if you're passing off fds and their associated (kernel- and/or user-side) buffers between cores, you can't run entirely in L1 or L2. And in general, I'd expect data to be responsible for much more cache pressure than code, so I'm skeptical of localizing the code at the expense of the data. But anyway, if the goal is to organize which cores are doing the work, splitting a single core's work from a single thread (pinnned to it) to several threads (still pinned to it) doesn't help. It just introduces more context switching. [1] https://news.ycombinator.com/item?id=10874616 https://news.ycombinator.com/item?id=10874616
rot13maxi 8mo ago
i havent seen an sdf1.org url in a looooong time. lovely to see its still around
password4321 8mo ago
Always interesting to review the latest techempower web framework benchmarks, though it's been a year: https://www.techempower.com/benchmarks/#section=data-r23&test=plaintext https://www.techempower.com/benchmarks/#section=data-r23&tes...

Fast-Servers

122 points by tosh 8mo ago

kogus 8mo ago

Slightly tangential, but why is the first diagram duplicated at .1 opacity?

tecleandor 8mo ago
That plus the ellipsis makes me thing that it means the additional threads that would open for next connections...
- kogus 8mo ago
  Ah, that makes sense.

ratrocket 8mo ago

discussed in 2016: https://news.ycombinator.com/item?id=10872209 https://news.ycombinator.com/item?id=10872209 (53 comments)

lmz 8mo ago

Seems similar to the SEDA architecture https://en.wikipedia.org/wiki/Staged_event-driven_architecture https://en.wikipedia.org/wiki/Staged_event-driven_architectu...

bee_rider 8mo ago

> One thread per core, pinned (affinity) to separate CPUs, each with their own epoll/kqueue fd > Each major state transition (accept, reader) is handled by a separate thread, and transitioning one client from one state to another involves passing the file descriptor to the epoll/kqueue fd of the other thread. So this seems like a little pipeline that all of the requests go through, right? For somebody who doesn’t do server stuff, is there a general idea of how many stages a typical server might be able to implement? And does it create a load-balancing problem? I’d expect some stages to be quite cheap…

marcosdumay 8mo ago
> For somebody who doesn’t do server stuff, is there a general idea of how many stages a typical server might be able to implement? On the HTTP server from the article, what I understood is that those 2 you are seeing are the ones you have. Or maybe 3, if disposing of things is slow. I'm not sure what I prefer. On one hand, there's some expensive coordination for passing those file descriptors around. On the other hand, having some separate code bother with creating and closing the connections make it easier to focus on the actual performance issues where they appear, and create opportunity to dispatch work smartly. Of course, you can go all the way in and make a green threads server where every bit of IO puts the work back on the queue. But you would use a single queue then, and dispatch the code that works on it. So you get more branching, but less coordination.

luizfelberti 8mo ago

A bit dated in the sense that for Linux you'd probably use io_uring nowadays, but otherwise it's a timeless design Still, I'm conflicted on whether separating stages per thread (accept on one thread and the client loop in another) is a good idea. It sounds like the gains would be minimal or non-existent even in ideal circumstances, and on some workloads where there's not a lot of clients or connection churn it would waste an entire core for handling a low-volume event. I'm open to contrarian opinions on this though, maybe I'm not seeing soemthing...

eklavya 8mo ago
It is not a good idea, especially with the new chiplet/CCX processors.
raggi 8mo ago
It’s not a good idea and that’s where I’d really start with the dated commentary here rather than focusing on the polling mechanism. It depends on the application but if the buffers are large (>=64kb) such as a common TCP workload then uring won’t necessarily help that much. You’ll gain a lot of scalability regardless of polling mechanism by making sure you can utilize rss and xss optimizations.
- wild_egg 8mo ago
  It's been a while but why is uring not helpful for larger buffers? I'd think the zero-copy I/O capabilities would make it more helpful for larger payloads, not less
  - Veserv 8mo ago
    uring supports zero-copy, but is not a copy-reduction mechanism; it is a syscall-reduction mechanism. Large buffers mean less syscalls to start with, so less benefit.
    - raggi 8mo ago
      Exactly this. The kernel alloc’d buffers can help but if that was a primary concern you’re in driver territory. Anything still userspace kind of optimization domain the portion of syscalls for large buffers in a buffered flow is heavily amortized and not overly relevant.

fao_ 8mo ago

this is more or less, in some way, what Erlang does and how Erlang is so easy to scale.

epicprogrammer 8mo ago

It’s an interesting throwback to SEDA, but physically passing file descriptors between different cores as a connection changes state is usually a performance killer on modern hardware. While it sounds elegant on a whiteboard to have a dedicated 'accept' core and a 'read' core, you end up trading a slightly simpler state machine for massive L1/L2 cache thrashing. Every time you hand off that connection, you immediately invalidate the buffers and TCP state you just built up. There’s a reason the industry largely settled on shared-nothing architectures like NGINX having a single pinned thread handle the entire lifecycle of a request keeps all that data strictly local to the CPU cache. When you're trying to scale, respecting data locality almost always beats pipeline cleanliness.

toast0 8mo ago
You could presumably have an acceptor thread per core, which passes the fds to core alligned next thread, etc. That would get you the code simplicity benefits the article suggests, while keeping the socket bound to a single core, which is definitely needed. Depending on if you actually need to share anything, you could do process per core, thread per loop, and you have no core to core communication from the usual workings of the process (i/o may cross though)
- scottlamb 8mo ago
  I don't think the author intended "code simplicity" as an end unto itself but a way to reduce cache pressure. He popped into the 2016 discussion [1] to say: > Another benefit of this design overlooked is that individual cores may not ever need to read memory -- the entire task can run in L1 or L2. If a single worker becomes too complicated this benefit is lost, and memory is much much slower than cache. I think this is wrong or at least overstated: if you're passing off fds and their associated (kernel- and/or user-side) buffers between cores, you can't run entirely in L1 or L2. And in general, I'd expect data to be responsible for much more cache pressure than code, so I'm skeptical of localizing the code at the expense of the data. But anyway, if the goal is to organize which cores are doing the work, splitting a single core's work from a single thread (pinnned to it) to several threads (still pinned to it) doesn't help. It just introduces more context switching. [1] https://news.ycombinator.com/item?id=10874616 https://news.ycombinator.com/item?id=10874616

rot13maxi 8mo ago

i havent seen an sdf1.org url in a looooong time. lovely to see its still around

password4321 8mo ago

Always interesting to review the latest techempower web framework benchmarks, though it's been a year: https://www.techempower.com/benchmarks/#section=data-r23&test=plaintext https://www.techempower.com/benchmarks/#section=data-r23&tes...