16 ms·
> Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely?
by dragontamer 5y ago
> Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely? This is what new generation consoles are using to load static scene data from NVMe drives to GPUs directly over PCIe, your workload also seems to match this model well.
Lets say an Ethernet Port on NUMA#1 is asking for a movie found on a NVMe SSD on NUMA#3.
There's simply no way for you to get the data unless you traverse the NUMA-fabric somehow. Either NUMA#1 tells NUMA#3 "Yo, start sending me data so that I can reply to this TCP/TLS/HTTP message"... or NUMA#1 offloads the job to NUMA#3 somehow (Its now NUMA#3's problem to figure out how to talk back).
----------
There are many ways to implement the system, each with apparently their own pros and cons. There are further issues: in an earlier post a few weeks ago, drewg123 explained that his system / bosses demanded that each server was only allowed to have 1 IP Address (meaning all 4 NICs had to cooperate using link-aggregation).
You can effectively think of this as "each Netflix stream comes in on one of the 4-random ports" (which may be on NUMA#1 or NUMA#2). While each bit of data is scattered across NUMA#1, #2, #3, and #4.
---------
Note: NUMA#1 and NUMA#2 have the NICs, which means that they have fewer remaining PCIe lanes for storage (The ConnectX NIC uses multiple PCIe lanes). As a result, most of the movie's data will be on NUMA#3 or NUMA#4.
-------
Note: I expect the NUMA-fabric to be faster than PCIe fabric. NUMA-fabric is designed so that different chips can pretend each other's RAM at roughly 50+GBps bandwidth at something like 300ns latencies. In contrast, PCIe 4.0 x16 is only 30GBps or so... and a lot of these NVMe SSDs are only 4x lanes (aka: 8GBps).
You also need to handle all the networking tidbits: HTTPS has state (not just the TLS state, but also the state of the stream: who's turn is it to talk and such). Which means the CPU / Application data needs to be in the loop somehow. I know that the most recent talk offloads the process as much as possible thanks to the "sendfile" interface on FreeBSD (also available on Linux), which allows you to "pipe" data from one file-descriptor to another.
- toast0 5y agoFrom drewg123's other threads, it seems like their machines are also hitting (or approaching) memory bandwidth limits, so being able to reduce some of the memory write/read requirements should help with that. I think what was working best was having the disk I/O write to RAM that was NUMA aligned with the NIC, so disk -> PCIe -> NUMA Fabric -> RAM -> PCIe -> NIC. If instead you could do disk -> PCIe -> NUMA Fabric -> PCIe -> NIC, at least for a portion of the disk reads, that would still be the same amount of traffic on the NUMA Fabric, but less traffic on the memory bus. This probably means that the NIC would be doing more high latency reads though, so you need more in-flight sends to keep throughput up.
- infogulch 5y agoBoth NICs and SSDs are connected to the CPU via PCIe: all SSD and NIC traffic already go through PCIe before they hit main memory today. PCIe 4.0 x16 is 64GB/s, and there are 8 groups of x16 lanes in a 128 lane EPYC, for a total of 512GB/s or 4Tb/s, more than double the max fabric bandwidth of 200GB/s. Lets take the original diagram from the video: CPU ↑ ↓ Storage → Memory → NIC He was able to use NIC-kTLS offloading to simplify it to this: Storage → Memory → NIC Now lets add a bit more detail to the second diagram, expanding it to include PCIe: Memory ↑ ↓ Storage → PCIe → NIC This third diagram is the same as the diagram above it, except it explicitly describes how data gets from storage to memory and from memory to NIC. So the story for a request is something like: 1. request comes in for a file, loop { 2. CPU requests chunks of data, 3. data is delivered to memory via PCIe and signals CPU to handle it, 4. CPU tells NIC to send data from memory, 5. NIC requests data from memory via PCIe, sends it out on port } If you squint this looks kinda like the first diagram where there were extra unnecessary data transfers going up through the CPU, except now they're going up through main memory. My proposal is to skip main memory and go straight from storage to NIC as in: Storage → PCIe → NIC The story for serving a request would now be: 1. request comes in for a file, 2. CPU tells NIC: "ok get it from SSD 13", 3. NIC requests data from SSD via PCIe, sends it out on port, 4. cpu & main memory: crickets