5 ms·

The first thing that came to mind was @agl's discussion of TLS's overhead with Gmail: https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html https://ww

by semenko 11y ago

The first thing that came to mind was @agl's discussion of TLS's overhead with Gmail: https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html https://www.imperialviolet.org/2010/06/25/overclocking-ssl.h... It's a bit hard for me to reconcile "<1% of the CPU load" (Google) with "up to 50% capacity hit" (Netflix).

wmf 11y ago
Gmail servers probably do actual work. Netflix's CDN just copies bytes from disk to network, so TLS overhead is a greater percentage.
- nemothekid 11y ago
  I'd also the content type has something to do with it. Encrypting a 10kb email is a lot faster than encrypting an 3GB movie.
  - aw3c2 11y ago
    Isn't it just compressed "on the stream"? It's a matter of throughput, not size of individual files.
jewel 11y ago
It's surprising. They say that they are using the hardware AES support in the CPU too. They do mention that they are no longer able to take advantage of sendfile() which was letting the bulk of the data transfer to happen completely in kernel space. So, in their case, they had already optimized things to such a degree that they were having a lot of throughput and very little CPU usage per byte. In fact, it's possible that on their old system the data streams wouldn't have to be touched a byte at a time, as they could be loaded into main memory via DMA from the disk and once that finished they could be transferred to the network card, again via DMA. Once sendfile() is called, no further context switches are necessary for that request. Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI. I tried running two concurrent copies and they each got that speed, which tells me that it's per-core. That's easily enough speed to saturate a 10G connection. It's quite possible that they have much older or slower hardware involved, since when you're building a server like this normally the network or disks are the limiting factor.
- Freaky 11y ago
  > Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI Bit of a crappy test, though - tight loop over a single 8k buffer (used in-place). It probably executes entirely in L1/L2. They provide details of the hardware they're using: https://openconnect.itp.netflix.com/hardware/ https://openconnect.itp.netflix.com/hardware/ The fanciest in their "IO-optimized" SSD-driven beast is a 2.7GHz 12 core Ivy Bridge. You've got a 20% clock advantage, and Haswell reportedly reduced AES-NI clock latency from 8 to 7 cycles for a ~14% improvement. Doing the math I get 55Gbps for the lot. That's without doing any packet processing, no waiting on memory accesses, no interrupts, no context switching, just perfect scaling with a tight loop over the same 8k buffer - and they're driving 4 10Gbps NICs. Not much left over.
  - xxs 11y ago
    >> It probably executes entirely in L1/L2. L1!
Freaky 11y ago
If you're just pushing raw bytes down a socket, you can use sendfile(2) to direct cached data directly to the NIC without copying it to a private buffer first. This is a pretty big deal at 40Gbps, and Netflix have funded quite a bit of work towards improving it: https://secure.freshbsd.org/search?project=freebsd&q=netflix+sendfile TLS kicks all that in the teeth - each encrypted stream needs its own private buffer which has to go in and out of the CPU before it gets anywhere near the NIC. That costs memory bandwidth (a mere 400Gbps with quad-channel DDR3), memory you could have otherwise used as cache (keeping in mind you need to hold onto it until the client ACKs it), and CPU cycles (probably still a decent chunk of overhead even with AES-NI).
pwaring 11y ago
They are completely different use cases though, so it shouldn't be too surprising that the results are different. Gmail: Lots of small requests which each last a short time. Netflix: Small number of large requests with connections that stay open a long time (possibly hours).

by semenko 11y ago

wmf 11y ago

Gmail servers probably do actual work. Netflix's CDN just copies bytes from disk to network, so TLS overhead is a greater percentage.

nemothekid 11y ago
I'd also the content type has something to do with it. Encrypting a 10kb email is a lot faster than encrypting an 3GB movie.
- aw3c2 11y ago
  Isn't it just compressed "on the stream"? It's a matter of throughput, not size of individual files.

jewel 11y ago

It's surprising. They say that they are using the hardware AES support in the CPU too. They do mention that they are no longer able to take advantage of sendfile() which was letting the bulk of the data transfer to happen completely in kernel space. So, in their case, they had already optimized things to such a degree that they were having a lot of throughput and very little CPU usage per byte. In fact, it's possible that on their old system the data streams wouldn't have to be touched a byte at a time, as they could be loaded into main memory via DMA from the disk and once that finished they could be transferred to the network card, again via DMA. Once sendfile() is called, no further context switches are necessary for that request. Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI. I tried running two concurrent copies and they each got that speed, which tells me that it's per-core. That's easily enough speed to saturate a 10G connection. It's quite possible that they have much older or slower hardware involved, since when you're building a server like this normally the network or disks are the limiting factor.

Freaky 11y ago
> Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI Bit of a crappy test, though - tight loop over a single 8k buffer (used in-place). It probably executes entirely in L1/L2. They provide details of the hardware they're using: https://openconnect.itp.netflix.com/hardware/ https://openconnect.itp.netflix.com/hardware/ The fanciest in their "IO-optimized" SSD-driven beast is a 2.7GHz 12 core Ivy Bridge. You've got a 20% clock advantage, and Haswell reportedly reduced AES-NI clock latency from 8 to 7 cycles for a ~14% improvement. Doing the math I get 55Gbps for the lot. That's without doing any packet processing, no waiting on memory accesses, no interrupts, no context switching, just perfect scaling with a tight loop over the same 8k buffer - and they're driving 4 10Gbps NICs. Not much left over.
- xxs 11y ago
  >> It probably executes entirely in L1/L2. L1!

Freaky 11y ago

If you're just pushing raw bytes down a socket, you can use sendfile(2) to direct cached data directly to the NIC without copying it to a private buffer first. This is a pretty big deal at 40Gbps, and Netflix have funded quite a bit of work towards improving it: https://secure.freshbsd.org/search?project=freebsd&q=netflix+sendfile TLS kicks all that in the teeth - each encrypted stream needs its own private buffer which has to go in and out of the CPU before it gets anywhere near the NIC. That costs memory bandwidth (a mere 400Gbps with quad-channel DDR3), memory you could have otherwise used as cache (keeping in mind you need to hold onto it until the client ACKs it), and CPU cycles (probably still a decent chunk of overhead even with AES-NI).

pwaring 11y ago

They are completely different use cases though, so it shouldn't be too surprising that the results are different. Gmail: Lots of small requests which each last a short time. Netflix: Small number of large requests with connections that stay open a long time (possibly hours).