5 ms·

Memory bandwidth and latency is a real problem. As the referenced article points out, both have gotten worse as compared to CPU speeds as time goes on. The bi

Memory bandwidth and latency is a real problem. As the referenced article points out, both have gotten worse as compared to CPU speeds as time goes on. The biggest bottleneck in my favorite workload (Netflix CDN server) is memory bandwidth. When doing software kTLS, we do 4 access per byte served to customers. One access to DMA the contents of a video file to DRAM from storage. A second access to read the byte for encryption. A third to write the encrypted data to DRAM. And a fourth to DMA the encrypted data to the NIC. That means we're bottlenecked serving Netflix video at ~240Gb/s on AMD Rome servers (quad channel DDR4-3200, single socket). The theoretical performance, assuming everything besides serving the video is "free" is ~260-300Gb/s (depending on numa config, etc). Every cache miss that I can eliminate leads to a small performance increase by both decreasing the memory system load (and leaving more headroom for the workload), and by eliminating a CPU stall on DRAM access. I'm thrilled that next generation servers will have DDR5. That should help a lot with bandwidth, but the PCIe and network speeds will go up as well (Gen4->Gen5, 100GbE -> 400GbE), so we'll still be fighting the same battle. Things like inline-kTLS (where the NIC does the TLS encryption) reduce the memory bandwidth requirements by roughly half, and is how we're able to serve up over 350Gb/s from a single socket.

mhh__ 5y ago
> real problem I'd argue it's the only problem more times than not, although I don't work at a particularly high level of abstraction so YMMV
PragmaticPulp 5y ago
That's a fascinating project to work on. > That means we're bottlenecked serving Netflix video at ~240Gb/s on AMD Rome servers (quad channel DDR4-3200, single socket). AMD Rome supports octa-channel DDR4 even in single-socket servers. Did you mean octa-channel DDR4, or is there some other limitation in the system that limits you to quad-channel configurations?
magicalhippo 5y ago
> Things like inline-kTLS (where the NIC does the TLS encryption) reduce the memory bandwidth requirements by roughly half If storage is accessed via PCI-e, couldn't one just pipe it straight from the storage to the NIC, using p2pdma[1][2] or similar? Seems you could skip all of the main DRAM round-trip this way. Or aren't things that simple? [1]: https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma.html https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma... [2]: https://youtu.be/LDOlqgUZtHE?t=612 https://youtu.be/LDOlqgUZtHE?t=612
- throwawaylinux 5y ago
  I imagine for some reason their encryption or some other processing step is not performed by their hardware (you can get NICs with crypto offload) or they've found it cheaper or more flexible to go with doing it on the CPU. But for other, this is what is done. Which is probably why their use case here isn't optimized all that well. There have been ongoing efforts on and off to make DMA directly to/from CPU cache work well. And it can be made to work with some hardware, but I don't know if it's performed to potential. In theory his workload requires no DRAM accesses for the data if the working set could fit in cache. DMA from disk to dirty cache line, CPU loads from there encrypts, stores it in place or to new dirty line, NIC loads from that dirty data, and the dirty cache lines are invalidated or overwritten in a ring buffer with new data to avoid the mandatory writeback to DRAM. Some of the new CXL protocols coming down the pipeline might improve this situation, but they also might not.
- drewg123 5y ago
  I've been looking at this. When using inline hardware kTLS offload, this is an amazing solution on paper. However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. According to a colleague who knows the internal architecture of NVME drives well, making 1400b requests is a non-starter. So to use P2P DMA, you need some place to cache things. There turn out to be 3 places to cache things w/o going to the host: 1) On the NVME drive itself. This is known as controller memory buffer (CMB). Unfortunately, in order to get a CMB that's large enough to be useful, you need to buy gold-plated enterprise drives, which would be fine for a science experiment, but does not scale. 2) On the NIC. We use Mellanox CX6-DX NICs, and they have essentially no storage. 3) On something on the PCIe bus that has memory, like a GPU or custom "box of ram". The GPU would blow our thermal budget, and I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe.
  - magicalhippo 5y ago
    > However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. Ah, I had imagined the NIC TLS stuff to be more stream based, clearly out of my comfort zone this stuff. > I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe. Yeah, I mean some FPGA cards could probably do the trick, but would not be a low cost option and I guess not low power if you want full speed PCIe. Close but no cigar for p2pdma then...
peterburkimsher 5y ago
"There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors" - Phil Karlton It's the same problem when solving technical difficulties in life: I fill my pockets and backpack with adaptors, because the cost of a cache miss (going back to my room to get it) means it's no longer useful (for plugging in the laptop to the projector so these people can see the slides). If only we could predict the future, then the cache could pre-load! And usually things do happen as planned, but sometimes free will gets involved and we branch another way, and need to fetch something else from memory, and that takes a while.
gautamcgoel 5y ago
Dumb question: since memory bandwidth is clearly a bottleneck in many server workloads, why don't chip manufacturers make CPUs with 20 memory channels instead of just 6-10, as in current CPUs?
- PragmaticPulp 5y ago
  > Dumb question: since memory bandwidth is clearly a bottleneck in many server workloads, why don't chip manufacturers make CPUs with 20 memory channels instead of just 6-10, as in current CPUs? It gets difficult to fit it all and make it work. An 8-channel DDR4 memory controller already has 512 pins just for data bits, and that's before you deal with all of the necessary grounding pins and all of the other signal lines required to interface with memory chips. There are rumors that AMD's next generation server CPUs will support 12-channel DDR5. The expanded memory channels and other features require a socket with 6096 pins. That's a lot of interfaces to squeeze through a socket. A hypothetical 20-channel CPU would also require you to install RAM in multiples of 20 sticks to populate every channel. You can theoretically get 16 channels of memory bandwidth in a single, dual-socket system if each socket has an 8-channel memory controller. Dual-socket AMD systems have a theoretical upper limit memory bandwidth around 300GB (big B) per second. In reality, not as many applications are constrained my memory bandwidth as you might expect. CPU vendors are adding additional cache to work around the memory issue somewhat. As long as the application isn't constantly streaming bytes into and out of memory (as in the CDN example), it will eventually have to stop to work on the data, allowing the CPU a chance to cache some additional data from main memory and hide the memory bandwidth bottleneck somewhat.
- wmf 5y ago
  Here you go; just don't ask what it costs. https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6070&year=2019&tag=3 https://www.linleygroup.com/newsletters/newsletter_detail.ph... https://fuse.wikichip.org/news/2893/ibm-adds-power9-aio-pushes-for-an-open-memory-agnostic-interface/ https://fuse.wikichip.org/news/2893/ibm-adds-power9-aio-push...
- Dylan16807 5y ago
  Most of the people that really need a certain amount of bandwidth per core are probably better off buying CPUs with fewer cores.
shadilay 5y ago
>AMD Rome servers (quad channel DDR4-3200, single socket) Rome has 8 memory channels. https://en.wikichip.org/wiki/amd/cores/rome https://en.wikichip.org/wiki/amd/cores/rome
- drewg123 5y ago
  Yes, sorry, I was off by a factor of 2 :)
fragmede 5y ago
Fascinating! Have you looked at other ways to get more TLS performance out of the system as a whole? How does that performance even compare to the Netflix services side of things (aka in AWS)? How do various TLS offload strategies stack up?
perfmode 5y ago
how do you eliminate cache misses in this workload? seems like long runs of sequential reads/writes will result in eviction by the time the “byte” is ready for the next stage of the process.

mhh__ 5y ago

> real problem I'd argue it's the only problem more times than not, although I don't work at a particularly high level of abstraction so YMMV

PragmaticPulp 5y ago

That's a fascinating project to work on. > That means we're bottlenecked serving Netflix video at ~240Gb/s on AMD Rome servers (quad channel DDR4-3200, single socket). AMD Rome supports octa-channel DDR4 even in single-socket servers. Did you mean octa-channel DDR4, or is there some other limitation in the system that limits you to quad-channel configurations?

magicalhippo 5y ago

> Things like inline-kTLS (where the NIC does the TLS encryption) reduce the memory bandwidth requirements by roughly half If storage is accessed via PCI-e, couldn't one just pipe it straight from the storage to the NIC, using p2pdma[1][2] or similar? Seems you could skip all of the main DRAM round-trip this way. Or aren't things that simple? [1]: https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma.html https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma... [2]: https://youtu.be/LDOlqgUZtHE?t=612 https://youtu.be/LDOlqgUZtHE?t=612

throwawaylinux 5y ago
I imagine for some reason their encryption or some other processing step is not performed by their hardware (you can get NICs with crypto offload) or they've found it cheaper or more flexible to go with doing it on the CPU. But for other, this is what is done. Which is probably why their use case here isn't optimized all that well. There have been ongoing efforts on and off to make DMA directly to/from CPU cache work well. And it can be made to work with some hardware, but I don't know if it's performed to potential. In theory his workload requires no DRAM accesses for the data if the working set could fit in cache. DMA from disk to dirty cache line, CPU loads from there encrypts, stores it in place or to new dirty line, NIC loads from that dirty data, and the dirty cache lines are invalidated or overwritten in a ring buffer with new data to avoid the mandatory writeback to DRAM. Some of the new CXL protocols coming down the pipeline might improve this situation, but they also might not.
drewg123 5y ago
I've been looking at this. When using inline hardware kTLS offload, this is an amazing solution on paper. However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. According to a colleague who knows the internal architecture of NVME drives well, making 1400b requests is a non-starter. So to use P2P DMA, you need some place to cache things. There turn out to be 3 places to cache things w/o going to the host: 1) On the NVME drive itself. This is known as controller memory buffer (CMB). Unfortunately, in order to get a CMB that's large enough to be useful, you need to buy gold-plated enterprise drives, which would be fine for a science experiment, but does not scale. 2) On the NIC. We use Mellanox CX6-DX NICs, and they have essentially no storage. 3) On something on the PCIe bus that has memory, like a GPU or custom "box of ram". The GPU would blow our thermal budget, and I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe.
- magicalhippo 5y ago
  > However, NVME speaks in 4KB blocks, but NICs speak in ~1400 byte MSS chunks. Ah, I had imagined the NIC TLS stuff to be more stream based, clearly out of my comfort zone this stuff. > I can't think of a low cost, low-power "box of RAM" that can be attached at full speed to PCIe. Yeah, I mean some FPGA cards could probably do the trick, but would not be a low cost option and I guess not low power if you want full speed PCIe. Close but no cigar for p2pdma then...

peterburkimsher 5y ago

"There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors" - Phil Karlton It's the same problem when solving technical difficulties in life: I fill my pockets and backpack with adaptors, because the cost of a cache miss (going back to my room to get it) means it's no longer useful (for plugging in the laptop to the projector so these people can see the slides). If only we could predict the future, then the cache could pre-load! And usually things do happen as planned, but sometimes free will gets involved and we branch another way, and need to fetch something else from memory, and that takes a while.

gautamcgoel 5y ago

Dumb question: since memory bandwidth is clearly a bottleneck in many server workloads, why don't chip manufacturers make CPUs with 20 memory channels instead of just 6-10, as in current CPUs?

PragmaticPulp 5y ago
> Dumb question: since memory bandwidth is clearly a bottleneck in many server workloads, why don't chip manufacturers make CPUs with 20 memory channels instead of just 6-10, as in current CPUs? It gets difficult to fit it all and make it work. An 8-channel DDR4 memory controller already has 512 pins just for data bits, and that's before you deal with all of the necessary grounding pins and all of the other signal lines required to interface with memory chips. There are rumors that AMD's next generation server CPUs will support 12-channel DDR5. The expanded memory channels and other features require a socket with 6096 pins. That's a lot of interfaces to squeeze through a socket. A hypothetical 20-channel CPU would also require you to install RAM in multiples of 20 sticks to populate every channel. You can theoretically get 16 channels of memory bandwidth in a single, dual-socket system if each socket has an 8-channel memory controller. Dual-socket AMD systems have a theoretical upper limit memory bandwidth around 300GB (big B) per second. In reality, not as many applications are constrained my memory bandwidth as you might expect. CPU vendors are adding additional cache to work around the memory issue somewhat. As long as the application isn't constantly streaming bytes into and out of memory (as in the CDN example), it will eventually have to stop to work on the data, allowing the CPU a chance to cache some additional data from main memory and hide the memory bandwidth bottleneck somewhat.
wmf 5y ago
Here you go; just don't ask what it costs. https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6070&year=2019&tag=3 https://www.linleygroup.com/newsletters/newsletter_detail.ph... https://fuse.wikichip.org/news/2893/ibm-adds-power9-aio-pushes-for-an-open-memory-agnostic-interface/ https://fuse.wikichip.org/news/2893/ibm-adds-power9-aio-push...
Dylan16807 5y ago
Most of the people that really need a certain amount of bandwidth per core are probably better off buying CPUs with fewer cores.

shadilay 5y ago

>AMD Rome servers (quad channel DDR4-3200, single socket) Rome has 8 memory channels. https://en.wikichip.org/wiki/amd/cores/rome https://en.wikichip.org/wiki/amd/cores/rome

drewg123 5y ago
Yes, sorry, I was off by a factor of 2 :)

fragmede 5y ago

Fascinating! Have you looked at other ways to get more TLS performance out of the system as a whole? How does that performance even compare to the Netflix services side of things (aka in AWS)? How do various TLS offload strategies stack up?

perfmode 5y ago

how do you eliminate cache misses in this workload? seems like long runs of sequential reads/writes will result in eviction by the time the “byte” is ready for the next stage of the process.