3 ms·
I'm not at Netflix, just summarizing. Check out the Netflix presentations like [1]; at 400gbps, the bottleneck is RAM and NUMA fabric. Doing bulk crypto on the
by toast0 5y ago
I'm not at Netflix, just summarizing. Check out the Netflix presentations like [1]; at 400gbps, the bottleneck is RAM and NUMA fabric. Doing bulk crypto on the CPU requires double the RAM bandwidth, and that kills the throughput regardless of the cost of cipher computation.
TLS termination costs are a large factor for Netflix because their CDN appliances want to have as much throughput as possible in a small space. At 'web scale', it wouldn't be a big deal to be happy with 100gbps througput and use 4 boxes; but getting ISPs to install more boxes is hard, it's better to get one box to work as fast as possible.
Personally, I've seen it be an issue; dual xeon 2690 v1 or maybe v2 couldn't push 10G via TLS, but could via plain text. But the v3 chips had better acceleration and could do 2x10G no problem. I never got access to more than 2x10G networking, so no idea what the limit was. That was under managed hosting (dedicated bare metal), so no ability to do networking beyond what was offered. At Facebook, they tended to use smaller servers and more of them, and I didn't deal with TLS as much.
[1] https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf
- drewg123 5y agoThe limit for FreeBSD software kTLS with our workload is a little over 100Gb/s (using a 2x100G NIC) on a 2697a v4. I think most people only get 25-30Gb/s out of the same CPU when not using kTLS.