7 ms·

How to escape the Linux networking stack

148 points by meysamazad 11mo ago

seabrookmx 11mo ago
I had to read their article on "soft-unicast" before I could really grok this one: https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-anymore/ https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-...
lazyeye 11mo ago
SLATFATF - "So long and thanks for all the fish" is a Douglas Adams quote https://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_the_Fish https://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_th...
- cestith 11mo ago
  A few things in the article are Douglas Adams quotes, and more specifically from the Hitchhiker’s Guide series. Creating the universe being regarded as a mistake and making many unhappy is from those books. Whenever someone figures out the universe it gets replaced with something stranger and having evidence that’s happened repeatedly is too. The Restaurant at the End of the Universe is reference in the article. I’m a bit surprised nothing in the article was mentioned as being “mostly harmless”.
  - gishh 11mo ago
    One of these days I’ll figure out how to throw myself at the ground and miss.
notepad0x90 11mo ago
I'm slightly surprised cloudflare isn't using a userspace tcp/ip stack already (faster - less context switches and copies). It's the type of company I'd expect to actually need one.
- Droobfest 11mo ago
  From 2016: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
  - notepad0x90 11mo ago
    Nice, they know better. But it also makes me wonder, because they're saying "but what if you need to run another app", I'd expect for things like loadbalancers for example, you'd only run one app per server on the data plane, the user space stack handles that, and the OS/services use a different control plane NIC with the kernel stack so that boxes are reachable even if there is link saturation, ddos,etc.. It also makes me wonder, why is tcp/ip special? The kernel should expose a raw network device. I get physical or layer 2 configuration happening in the kernel, but if it is supposed to do IP, then why stop there, why not TLS as well? Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? It sounds like "that's just the way it's always been done" type of a scenario.
    - wmf 11mo ago
      AFAIK Cloudflare runs their whole stack on every machine. I guess that gives them flexibility and maybe better load balancing. They also seem to use only one NIC. why is tcp/ip special? The kernel should expose a raw network device. ... Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? Check out the MIT Exokernel project and Solarflare OpenOnload that used this approach. It never really caught on because the old school way is good enough for almost everyone. why stop there, why not TLS as well? kTLS is a thing now (mostly used by Netflix). Back in the day we also had kernel-mode Web servers to save every cycle.
      - b112 11mo ago
        Was it Tux? I've only used it, a looong time ago, on load balancers. https://en.wikipedia.org/wiki/TUX_web_server https://en.wikipedia.org/wiki/TUX_web_server
alecco 11mo ago
Being a networking company I always wondered why did they pick Linux over FreeBSD.
- HumanOstrich 11mo ago
  Why does being a networking company suggest FreeBSD is the "right" pick?
  - password4321 11mo ago
    Serving Netflix Video at 400Gb/s on FreeBSD [pdf] (2021) https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738 (I don't share this as "the answer" as much as one example from years past.)
    - HumanOstrich 11mo ago
      I think they used FreeBSD because they were already using FreeBSD. The article doesn't mention Linux.
    - victorbjorklund 11mo ago
      To be honest, I think when I heard them speak, they're kind of saying yes, FreeBSD is awesome but that the main reason is that the early people there liked FreeBSD so they just stuck with it. And that it's a good choice, but they don't claim these are things that would be impossible to do with optimizations in Linux.
  - alecco 11mo ago
    Because FreeBSD is known for having the best network stack. The code is elegant and clean. And, at least until a few years ago, it was the preferred choice to build routers or firewalls. AFAIK, they were the first to implement BPF for production ready code almost 3 decades ago. https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter But all this is opinion and anecdotal. Just pick a random network feature and compare by yourself the Linux and the FreeBSD code.
marginalia_nu 11mo ago
This is extremely tangential, but I was working on setting up some manual network namespaces recently, basically manually reproducing what docker does to fix some of its faulty assumptions regarding containers having multiple IPs and a single name causing all sort of jank, and had to freshen up on a lot of Linux virtual networking concepts (namespaces, veths, bridge networks, macvlans and various other interfaces), made a ton of fairly informal notes to make myself sufficiently familiar with the thing to set it up. Would anyone be interested if I polished it up and maybe added a refresher on the relevant layer 2 networking needed to reason about it? It's a fair bit of work and it's a niche topic, so I'm trying to poll a bit to see if the juice is worth the squeeze.
- manuelangel99 11mo ago
  I would def. be interestred!
- msbhvn 11mo ago
  Please do it, I'm very biased but I think there would be lots of interest in seeing all that explained in one place in a coherant fashion (you will likely sharpen your own understanding in the process and have the perfect resource for when you next need to revisit these topics).
- MrResearcher 11mo ago
  Don't forget to post the link here!
- HumanOstrich 11mo ago
  I was actually going down rabbitholes today trying to figure out how to do a sane Docker setup where all the containers couldn't connect to each other. Your notes would be valuable at most any level of polish.
  - esseph 11mo ago
    If you create each container in its own network namespace, they won't be able to.
snvzz 11mo ago
Tangentially related, seL4's LionsOS can now act as a router/firewall[0]. 0. https://news.ycombinator.com/item?id=45959952 https://news.ycombinator.com/item?id=45959952
pjmlp 11mo ago
I would expect they would do the same as other big scalers, and handle most of the networking in dedicated card firmware, https://learn.microsoft.com/en-us/azure/azure-boost/overview#networking https://learn.microsoft.com/en-us/azure/azure-boost/overview... https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-mana-overview https://learn.microsoft.com/en-us/azure/virtual-network/acce...
sora2video 11mo ago
[dead]

148 points by meysamazad 11mo ago

I had to read their article on "soft-unicast" before I could really grok this one: https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-anymore/ https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-...

lazyeye 11mo ago

SLATFATF - "So long and thanks for all the fish" is a Douglas Adams quote https://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_the_Fish https://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_th...

cestith 11mo ago
A few things in the article are Douglas Adams quotes, and more specifically from the Hitchhiker’s Guide series. Creating the universe being regarded as a mistake and making many unhappy is from those books. Whenever someone figures out the universe it gets replaced with something stranger and having evidence that’s happened repeatedly is too. The Restaurant at the End of the Universe is reference in the article. I’m a bit surprised nothing in the article was mentioned as being “mostly harmless”.
- gishh 11mo ago
  One of these days I’ll figure out how to throw myself at the ground and miss.

notepad0x90 11mo ago

I'm slightly surprised cloudflare isn't using a userspace tcp/ip stack already (faster - less context switches and copies). It's the type of company I'd expect to actually need one.

Droobfest 11mo ago
From 2016: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
- notepad0x90 11mo ago
  Nice, they know better. But it also makes me wonder, because they're saying "but what if you need to run another app", I'd expect for things like loadbalancers for example, you'd only run one app per server on the data plane, the user space stack handles that, and the OS/services use a different control plane NIC with the kernel stack so that boxes are reachable even if there is link saturation, ddos,etc.. It also makes me wonder, why is tcp/ip special? The kernel should expose a raw network device. I get physical or layer 2 configuration happening in the kernel, but if it is supposed to do IP, then why stop there, why not TLS as well? Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? It sounds like "that's just the way it's always been done" type of a scenario.
  - wmf 11mo ago
    AFAIK Cloudflare runs their whole stack on every machine. I guess that gives them flexibility and maybe better load balancing. They also seem to use only one NIC. why is tcp/ip special? The kernel should expose a raw network device. ... Why run a complex network protocol stack in the kernel when you can just expose a configured layer 2 device to a user space process? Check out the MIT Exokernel project and Solarflare OpenOnload that used this approach. It never really caught on because the old school way is good enough for almost everyone. why stop there, why not TLS as well? kTLS is a thing now (mostly used by Netflix). Back in the day we also had kernel-mode Web servers to save every cycle.
    - b112 11mo ago
      Was it Tux? I've only used it, a looong time ago, on load balancers. https://en.wikipedia.org/wiki/TUX_web_server https://en.wikipedia.org/wiki/TUX_web_server

alecco 11mo ago

Being a networking company I always wondered why did they pick Linux over FreeBSD.

HumanOstrich 11mo ago
Why does being a networking company suggest FreeBSD is the "right" pick?
- password4321 11mo ago
  Serving Netflix Video at 400Gb/s on FreeBSD [pdf] (2021) https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738 (I don't share this as "the answer" as much as one example from years past.)
  - HumanOstrich 11mo ago
    I think they used FreeBSD because they were already using FreeBSD. The article doesn't mention Linux.
  - victorbjorklund 11mo ago
    To be honest, I think when I heard them speak, they're kind of saying yes, FreeBSD is awesome but that the main reason is that the early people there liked FreeBSD so they just stuck with it. And that it's a good choice, but they don't claim these are things that would be impossible to do with optimizations in Linux.
- alecco 11mo ago
  Because FreeBSD is known for having the best network stack. The code is elegant and clean. And, at least until a few years ago, it was the preferred choice to build routers or firewalls. AFAIK, they were the first to implement BPF for production ready code almost 3 decades ago. https://en.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter But all this is opinion and anecdotal. Just pick a random network feature and compare by yourself the Linux and the FreeBSD code.

marginalia_nu 11mo ago

This is extremely tangential, but I was working on setting up some manual network namespaces recently, basically manually reproducing what docker does to fix some of its faulty assumptions regarding containers having multiple IPs and a single name causing all sort of jank, and had to freshen up on a lot of Linux virtual networking concepts (namespaces, veths, bridge networks, macvlans and various other interfaces), made a ton of fairly informal notes to make myself sufficiently familiar with the thing to set it up. Would anyone be interested if I polished it up and maybe added a refresher on the relevant layer 2 networking needed to reason about it? It's a fair bit of work and it's a niche topic, so I'm trying to poll a bit to see if the juice is worth the squeeze.

manuelangel99 11mo ago
I would def. be interestred!
msbhvn 11mo ago
Please do it, I'm very biased but I think there would be lots of interest in seeing all that explained in one place in a coherant fashion (you will likely sharpen your own understanding in the process and have the perfect resource for when you next need to revisit these topics).
MrResearcher 11mo ago
Don't forget to post the link here!
HumanOstrich 11mo ago
I was actually going down rabbitholes today trying to figure out how to do a sane Docker setup where all the containers couldn't connect to each other. Your notes would be valuable at most any level of polish.
- esseph 11mo ago
  If you create each container in its own network namespace, they won't be able to.

snvzz 11mo ago

Tangentially related, seL4's LionsOS can now act as a router/firewall[0]. 0. https://news.ycombinator.com/item?id=45959952 https://news.ycombinator.com/item?id=45959952

pjmlp 11mo ago

I would expect they would do the same as other big scalers, and handle most of the networking in dedicated card firmware, https://learn.microsoft.com/en-us/azure/azure-boost/overview#networking https://learn.microsoft.com/en-us/azure/azure-boost/overview... https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-mana-overview https://learn.microsoft.com/en-us/azure/virtual-network/acce...

sora2video 11mo ago

[dead]