4 ms·

So, this is an honest question. What kind of performance do Linux based CDNs get out of a single box? At Netflix, we can serve over 90Gb/s of 100% TLS encrypt

So, this is an honest question. What kind of performance do Linux based CDNs get out of a single box? At Netflix, we can serve over 90Gb/s of 100% TLS encrypted traffic using a single-socket E5-2697A and Mellanox (or Chelsio) 100GbE NICs using software crypto (Intel ISA-L). This is distributed across tens of thousands of connections, and all is done in-kernel, using our "ssl sendfile". Eg, no dpdk, no crypo accelerators. I'm working on a tech blog about the changes we've needed to make to the FreeBSD kernel to get this kind of performance (and working on getting some of them in shape to upstream).

zaroth 9y ago
That would be the part of Netflix not running on AWS. :-) So the library feed can serve tens of thousands of streams, an aggregate 100Gbps, on a single node. And then... how many nodes to support the front-end UI operations to get to that point? It's funny how amazingly efficient we can be moving encrypted bits, but to support the APIs for login, browsing titles, updating account info, and setting up a stream; I'm going to guess ~100 of those nodes for every one of your stream-tanks?
peterwwillis 9y ago
It doesn't hurt to get a performance boost from your processor's crypto instructions, assuming you optimized your cipher lists to prefer crypto with a modern hw implementation (AES128-NI is 179% faster than RC4). But is this traffic ongoing connections, new connections, a mix? They have different penalties, and result in different numbers: 90Gbps of ongoing connections might be, like, 100,000hps, but 90Gbps of new connections during primetime might only net you 50,000hps. And are you using Google's UDP TLS stuff? Google also hacked on the kernel a lot to improve their performance, I don't know if any of that's upstream currently though. Maybe Cloudflare can answer you, as they seem to support the most HTTPS wizardry of the big CDNs.
- drewg123 9y ago
  Yes, as I said, we're using ISA-L. This is Intel's library with thin wrappers around hand-tuned assembly routines for crypto. The traffic is mostly long-ish lived connections. Eg, the duration of a TV show or movie. So there is some churn, but not a lot. This is all TCP. By "UDP TLS", I assume you mean Quic?

zaroth 9y ago

That would be the part of Netflix not running on AWS. :-) So the library feed can serve tens of thousands of streams, an aggregate 100Gbps, on a single node. And then... how many nodes to support the front-end UI operations to get to that point? It's funny how amazingly efficient we can be moving encrypted bits, but to support the APIs for login, browsing titles, updating account info, and setting up a stream; I'm going to guess ~100 of those nodes for every one of your stream-tanks?

peterwwillis 9y ago

It doesn't hurt to get a performance boost from your processor's crypto instructions, assuming you optimized your cipher lists to prefer crypto with a modern hw implementation (AES128-NI is 179% faster than RC4). But is this traffic ongoing connections, new connections, a mix? They have different penalties, and result in different numbers: 90Gbps of ongoing connections might be, like, 100,000hps, but 90Gbps of new connections during primetime might only net you 50,000hps. And are you using Google's UDP TLS stuff? Google also hacked on the kernel a lot to improve their performance, I don't know if any of that's upstream currently though. Maybe Cloudflare can answer you, as they seem to support the most HTTPS wizardry of the big CDNs.

drewg123 9y ago
Yes, as I said, we're using ISA-L. This is Intel's library with thin wrappers around hand-tuned assembly routines for crypto. The traffic is mostly long-ish lived connections. Eg, the duration of a TV show or movie. So there is some churn, but not a lot. This is all TCP. By "UDP TLS", I assume you mean Quic?