5 ms·

A few things that weren’t mentioned in detail or which I skimmed over without noticing: - The problem with bind before connect is that the OS thinks you might

A few things that weren’t mentioned in detail or which I skimmed over without noticing: - The problem with bind before connect is that the OS thinks you might call listen after bind instead of connect and listen requires the src port/ip to be unique. - Running out of ports may affect short-lived connections too: by default when you (the client) close a connection, it goes into a ‘TIME_WAIT’ state that locks up the {src,dst}_{ip,port} quadruple for (by default) 2 minutes to protect new connections with that quadruple getting packets that were meant for the previous connection. I think one thing that weakly surprised me about the article was that cloud flare seem to be mostly using standard BSD sockets apis and, presumably, the kernel interface to the network cards. I would have expected their main products (ie being a CDN) to use more specialised network cards and userspace networking (eg the final function they give for UDP does a ton of syscalls, and for tcp it still does several) with a non-BSD api. The main advantage would be having more control over their system and the ability to avoid a lot of unnecessary overhead from syscalls or apis, but there could be other advantages like NIC TLS (though this can also be accessed through the kernel as kernel TLS). I’m sure cloudflare have reasons for it though. Maybe the hardware is more expensive than underutilisation or hard to source or buggy or user space networking just doesn’t improve things much. Many of those things can make servers more efficient but that might not be necessary if many servers are needed at lots of edge locations with each server not needing efficiency. Or maybe those things are mostly just saving mics when the latency over the wire is millis.

MBCook 5y ago
> Running out of ports may affect short-lived connections too: by default when you (the client) close a connection, it goes into a ‘TIME_WAIT’ state that locks up the {src,dst}_{ip,port} quadruple for (by default) 2 minutes to protect new connections with that quadruple getting packets that were meant for the previous connection. This bit hard at a previous job. A very important PHP script would open DB connections (no pooling) and when it’s load hit the magic threshold we started running out of ports. But requests would succeed as TIME_WAIT expired so it just looked like some requests would hang with no reason. It was tough for us to figure out. The final solution was obvious (pooling) but that took some doing because the script was quite old and it took a bit of an overhaul for that to work. Don’t remember exactly why. Tech debt is fun. But we learned a new thing we had to be careful of, and a new failure mode we never expected.
stepchowfun 5y ago
> I think one thing that weakly surprised me about the article was that cloud flare seem to be mostly using standard BSD sockets apis and, presumably, the kernel interface to the network cards. I would have expected their main products (ie being a CDN) to use more specialised network cards and userspace networking (eg the final function they give for UDP does a ton of syscalls, and for tcp it still does several) with a non-BSD api. You might be interested in another article from them: "Why we use the Linux kernel's TCP stack" [1] [1] https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
graderjs 5y ago
> the OS thinks you might call listen after bind instead of connect and listen requires the src port/ip to be unique. They specifically mentioned this, and it was one reason for their custom solution.
mrspuratic 5y ago
Back in the HTTP/0.9 era TIME_WAIT was a big problem when running a busy [reverse] proxy. I remember using ndd to "tune" tcp_time_wait_interval on Solaris for this, as well as the anon (ephemeral) port range. This is my go to that topic now: https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-l... "The solution is more quadruplets" is for me one those things that's blindingly obvious once pointed out. See also the other link Cloudflare "Why we use the Linux kernel's TCP stack" link in a sibling - about half-way through they reveal they use both kernel bypass (DOS can max out iptables at ~1Mpps) and Solarflare NICs (offload options not stated), as well as Intel NICs with netmap bypass.
- lmb 5y ago
  We don't use kernel bypass anymore, it's all XDP: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/ https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...

MBCook 5y ago

> Running out of ports may affect short-lived connections too: by default when you (the client) close a connection, it goes into a ‘TIME_WAIT’ state that locks up the {src,dst}_{ip,port} quadruple for (by default) 2 minutes to protect new connections with that quadruple getting packets that were meant for the previous connection. This bit hard at a previous job. A very important PHP script would open DB connections (no pooling) and when it’s load hit the magic threshold we started running out of ports. But requests would succeed as TIME_WAIT expired so it just looked like some requests would hang with no reason. It was tough for us to figure out. The final solution was obvious (pooling) but that took some doing because the script was quite old and it took a bit of an overhaul for that to work. Don’t remember exactly why. Tech debt is fun. But we learned a new thing we had to be careful of, and a new failure mode we never expected.

stepchowfun 5y ago

> I think one thing that weakly surprised me about the article was that cloud flare seem to be mostly using standard BSD sockets apis and, presumably, the kernel interface to the network cards. I would have expected their main products (ie being a CDN) to use more specialised network cards and userspace networking (eg the final function they give for UDP does a ton of syscalls, and for tcp it still does several) with a non-BSD api. You might be interested in another article from them: "Why we use the Linux kernel's TCP stack" [1] [1] https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...

graderjs 5y ago

> the OS thinks you might call listen after bind instead of connect and listen requires the src port/ip to be unique. They specifically mentioned this, and it was one reason for their custom solution.

mrspuratic 5y ago

Back in the HTTP/0.9 era TIME_WAIT was a big problem when running a busy [reverse] proxy. I remember using ndd to "tune" tcp_time_wait_interval on Solaris for this, as well as the anon (ephemeral) port range. This is my go to that topic now: https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-l... "The solution is more quadruplets" is for me one those things that's blindingly obvious once pointed out. See also the other link Cloudflare "Why we use the Linux kernel's TCP stack" link in a sibling - about half-way through they reveal they use both kernel bypass (DOS can max out iptables at ~1Mpps) and Solarflare NICs (offload options not stated), as well as Intel NICs with netmap bypass.

lmb 5y ago
We don't use kernel bypass anymore, it's all XDP: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/ https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitig...