6 ms·

NUMA Siloing in the FreeBSD Network Stack (2019) [pdf]

70 points by harporoeder 5y ago

thinkingkong 5y ago
Previous discussion https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738
- detaro 5y ago
  this is not a previous discussion of this talk.
  - thinkingkong 5y ago
    Youre totally right. The talk I linked simply references NUMA siloing as a way of boosting throughput for their usecase.
perihelions 5y ago
Related (from a more recent talk): https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738 ("Serving Netflix Video at 400Gb/s on FreeBSD [pdf]") There's also an AMA by the author (in the HN thread).
throw0101a 5y ago
Video of the talk: * https://www.youtube.com/watch?v=_o-HcG8QxPc https://www.youtube.com/watch?v=_o-HcG8QxPc * https://2021.eurobsdcon.org/home/speakers/#serving https://2021.eurobsdcon.org/home/speakers/#serving Playlist of all the 2021 EuroBSDCon videos: * https://www.youtube.com/playlist?list=PLskKNopggjc4dadqaCDmctW-swHPD49td https://www.youtube.com/playlist?list=PLskKNopggjc4dadqaCDmc...
jeffbee 5y ago
I live for getting downvoted on HN so I'd just like to point out that this deck supports my previously-expressed opinion that the AMD EPYC architecture is harder to use. Out of the box, the Intel machine that is obsolete on paper was beating the EPYC machine by more than 50%.
- monocasa 5y ago
  EPYC isn't NUMA anymore.
  - MisterTea 5y ago
    I can not find anything to back up this claim. How else is AMD linking the multiple dies/sockets together?
    - monocasa 5y ago
      The DDR phys are on the I/O die, so all of the core complexes have the same length path to DRAM. Multi socket is still NUMA, but that's true of Intel as well.
      - dragontamer 5y ago
        > The DDR phys are on the I/O die, so all of the core complexes have the same length path to DRAM. The I/O die has 4 quadrants. The 2 chips in the 1st quadrant access the 1st quadrant's 2-memory channels slightly faster than the 4th quadrant. > Multi socket is still NUMA, but that's true of Intel as well. Intel has 6-memory controllers split into pairs of 3 IIRC (I'm going off of my memory here). The "left" 3 memory channels access the "left 9 cores" a bit faster than the "right 9 cores" in an 18-core Intel Skylake-X chip. -------- Both AMD and Intel have non-uniform latency/bandwidth even within the chips that they make.
        monocasa 5y ago
        There's a few cycles difference based on how the on chip network works, but variability in the number of off chip links between you and memory is what dominates the design. And in the context of what the parent said (but has since edited out), was what was being discussed.
  - dragontamer 5y ago
    Both EPYC and Intel Skylake-X are NUMA. Yes, Skylake-X. It turns out that the placement of those L3 caches matter, and some cores are closer to some memory controllers than others. https://software.intel.com/content/www/us/en/develop/articles/intel-xeon-processor-scalable-family-technical-overview.html https://software.intel.com/content/www/us/en/develop/article... ------------ Some cores have lower latency access to some memory channels than other memory channels. Our modern CPUs are so big, that even if everything is on a singular chip, the difference in latency can be measured. The only question that matters is: what is the bandwidth and latencies of _EACH_ core compared to _EACH_ memory channel? The answer is "it varies". "It varies" a bit for Skylake, and "it varies a bit more" for Rome (Zen 2), and "it varies a lot lot more" for Naples (Zen1). --------- For simplicity, both AMD and Intel offer memory-layouts (usually round-robin) that "mixes" the memory channels between the cores, causing an average latency. But for complexity / slightly better performance, both AMD and Intel also offer NUMA-modes. Either 4-NUMA for AMD's EPYCs, or SNC (Sub-numa clustering) for Intel chips. There are always a set of programmers who care enough about latency/bandwidth to drop down to this level.
drewg123 5y ago
This is my EuroBSDcon talk from 2 years ago. The more recent one (about getting to 400G) is up now at https://www.youtube.com/watch?v=_o-HcG8QxPc https://www.youtube.com/watch?v=_o-HcG8QxPc
- throw0101a 5y ago
  Looking to the 800Gb/s next year. :) Is that with the ConnectX-7? * https://www.nvidia.com/content/dam/en-zz/Solutions/networking/ethernet-adapters/connectx-7-datasheet-Final.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/networkin...
  - drewg123 5y ago
    No, just multiple CX6-DX I'm currently stuck because the only boards we can find that have enough PCIe lanes exposed to do 800g (64 for NICs + close to 64 for NVME, with them more or less equally divided between sockets) have only 3 xgmi links to the 2nd socket, rather than 4. That causes uneven loading on the xgmi links, which causes saturation issues well below 800g EDIT: I didn't realize the CX7 specs were public; I no longer feel quite so special :)
- infogulch 5y ago
  Great talk, thanks for sharing! Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely? This is what new generation consoles are using to load static scene data from NVMe drives to GPUs directly over PCIe, your workload also seems to match this model well. Edit to add: Nvidia branded this as GPUDirect Storage in 2019: https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/ Edit2: Oh neat this is mentioned in the connectx-7 doc linked in sibling comment.
  - dragontamer 5y ago
    > Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely? This is what new generation consoles are using to load static scene data from NVMe drives to GPUs directly over PCIe, your workload also seems to match this model well. Lets say an Ethernet Port on NUMA#1 is asking for a movie found on a NVMe SSD on NUMA#3. There's simply no way for you to get the data unless you traverse the NUMA-fabric somehow. Either NUMA#1 tells NUMA#3 "Yo, start sending me data so that I can reply to this TCP/TLS/HTTP message"... or NUMA#1 offloads the job to NUMA#3 somehow (Its now NUMA#3's problem to figure out how to talk back). ---------- There are many ways to implement the system, each with apparently their own pros and cons. There are further issues: in an earlier post a few weeks ago, drewg123 explained that his system / bosses demanded that each server was only allowed to have 1 IP Address (meaning all 4 NICs had to cooperate using link-aggregation). You can effectively think of this as "each Netflix stream comes in on one of the 4-random ports" (which may be on NUMA#1 or NUMA#2). While each bit of data is scattered across NUMA#1, #2, #3, and #4. --------- Note: NUMA#1 and NUMA#2 have the NICs, which means that they have fewer remaining PCIe lanes for storage (The ConnectX NIC uses multiple PCIe lanes). As a result, most of the movie's data will be on NUMA#3 or NUMA#4. ------- Note: I expect the NUMA-fabric to be faster than PCIe fabric. NUMA-fabric is designed so that different chips can pretend each other's RAM at roughly 50+GBps bandwidth at something like 300ns latencies. In contrast, PCIe 4.0 x16 is only 30GBps or so... and a lot of these NVMe SSDs are only 4x lanes (aka: 8GBps). You also need to handle all the networking tidbits: HTTPS has state (not just the TLS state, but also the state of the stream: who's turn is it to talk and such). Which means the CPU / Application data needs to be in the loop somehow. I know that the most recent talk offloads the process as much as possible thanks to the "sendfile" interface on FreeBSD (also available on Linux), which allows you to "pipe" data from one file-descriptor to another.

70 points by harporoeder 5y ago

thinkingkong 5y ago

Previous discussion https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738

detaro 5y ago
this is not a previous discussion of this talk.
- thinkingkong 5y ago
  Youre totally right. The talk I linked simply references NUMA siloing as a way of boosting throughput for their usecase.

perihelions 5y ago

Related (from a more recent talk): https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738 ("Serving Netflix Video at 400Gb/s on FreeBSD [pdf]") There's also an AMA by the author (in the HN thread).

throw0101a 5y ago

Video of the talk: * https://www.youtube.com/watch?v=_o-HcG8QxPc https://www.youtube.com/watch?v=_o-HcG8QxPc * https://2021.eurobsdcon.org/home/speakers/#serving https://2021.eurobsdcon.org/home/speakers/#serving Playlist of all the 2021 EuroBSDCon videos: * https://www.youtube.com/playlist?list=PLskKNopggjc4dadqaCDmctW-swHPD49td https://www.youtube.com/playlist?list=PLskKNopggjc4dadqaCDmc...

jeffbee 5y ago

I live for getting downvoted on HN so I'd just like to point out that this deck supports my previously-expressed opinion that the AMD EPYC architecture is harder to use. Out of the box, the Intel machine that is obsolete on paper was beating the EPYC machine by more than 50%.

monocasa 5y ago
EPYC isn't NUMA anymore.
- MisterTea 5y ago
  I can not find anything to back up this claim. How else is AMD linking the multiple dies/sockets together?
  - monocasa 5y ago
    The DDR phys are on the I/O die, so all of the core complexes have the same length path to DRAM. Multi socket is still NUMA, but that's true of Intel as well.
    - dragontamer 5y ago
      > The DDR phys are on the I/O die, so all of the core complexes have the same length path to DRAM. The I/O die has 4 quadrants. The 2 chips in the 1st quadrant access the 1st quadrant's 2-memory channels slightly faster than the 4th quadrant. > Multi socket is still NUMA, but that's true of Intel as well. Intel has 6-memory controllers split into pairs of 3 IIRC (I'm going off of my memory here). The "left" 3 memory channels access the "left 9 cores" a bit faster than the "right 9 cores" in an 18-core Intel Skylake-X chip. -------- Both AMD and Intel have non-uniform latency/bandwidth even within the chips that they make.
      - monocasa 5y ago
        There's a few cycles difference based on how the on chip network works, but variability in the number of off chip links between you and memory is what dominates the design. And in the context of what the parent said (but has since edited out), was what was being discussed.
- dragontamer 5y ago
  Both EPYC and Intel Skylake-X are NUMA. Yes, Skylake-X. It turns out that the placement of those L3 caches matter, and some cores are closer to some memory controllers than others. https://software.intel.com/content/www/us/en/develop/articles/intel-xeon-processor-scalable-family-technical-overview.html https://software.intel.com/content/www/us/en/develop/article... ------------ Some cores have lower latency access to some memory channels than other memory channels. Our modern CPUs are so big, that even if everything is on a singular chip, the difference in latency can be measured. The only question that matters is: what is the bandwidth and latencies of _EACH_ core compared to _EACH_ memory channel? The answer is "it varies". "It varies" a bit for Skylake, and "it varies a bit more" for Rome (Zen 2), and "it varies a lot lot more" for Naples (Zen1). --------- For simplicity, both AMD and Intel offer memory-layouts (usually round-robin) that "mixes" the memory channels between the cores, causing an average latency. But for complexity / slightly better performance, both AMD and Intel also offer NUMA-modes. Either 4-NUMA for AMD's EPYCs, or SNC (Sub-numa clustering) for Intel chips. There are always a set of programmers who care enough about latency/bandwidth to drop down to this level.

drewg123 5y ago

This is my EuroBSDcon talk from 2 years ago. The more recent one (about getting to 400G) is up now at https://www.youtube.com/watch?v=_o-HcG8QxPc https://www.youtube.com/watch?v=_o-HcG8QxPc

throw0101a 5y ago
Looking to the 800Gb/s next year. :) Is that with the ConnectX-7? * https://www.nvidia.com/content/dam/en-zz/Solutions/networking/ethernet-adapters/connectx-7-datasheet-Final.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/networkin...
- drewg123 5y ago
  No, just multiple CX6-DX I'm currently stuck because the only boards we can find that have enough PCIe lanes exposed to do 800g (64 for NICs + close to 64 for NVME, with them more or less equally divided between sockets) have only 3 xgmi links to the 2nd socket, rather than 4. That causes uneven loading on the xgmi links, which causes saturation issues well below 800g EDIT: I didn't realize the CX7 specs were public; I no longer feel quite so special :)
infogulch 5y ago
Great talk, thanks for sharing! Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely? This is what new generation consoles are using to load static scene data from NVMe drives to GPUs directly over PCIe, your workload also seems to match this model well. Edit to add: Nvidia branded this as GPUDirect Storage in 2019: https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/ Edit2: Oh neat this is mentioned in the connectx-7 doc linked in sibling comment.
- dragontamer 5y ago
  > Have you looked into something like DirectStorage to allow the nic to request data directly over PCIe, thus cutting out memory bandwidth limitations entirely? This is what new generation consoles are using to load static scene data from NVMe drives to GPUs directly over PCIe, your workload also seems to match this model well. Lets say an Ethernet Port on NUMA#1 is asking for a movie found on a NVMe SSD on NUMA#3. There's simply no way for you to get the data unless you traverse the NUMA-fabric somehow. Either NUMA#1 tells NUMA#3 "Yo, start sending me data so that I can reply to this TCP/TLS/HTTP message"... or NUMA#1 offloads the job to NUMA#3 somehow (Its now NUMA#3's problem to figure out how to talk back). ---------- There are many ways to implement the system, each with apparently their own pros and cons. There are further issues: in an earlier post a few weeks ago, drewg123 explained that his system / bosses demanded that each server was only allowed to have 1 IP Address (meaning all 4 NICs had to cooperate using link-aggregation). You can effectively think of this as "each Netflix stream comes in on one of the 4-random ports" (which may be on NUMA#1 or NUMA#2). While each bit of data is scattered across NUMA#1, #2, #3, and #4. --------- Note: NUMA#1 and NUMA#2 have the NICs, which means that they have fewer remaining PCIe lanes for storage (The ConnectX NIC uses multiple PCIe lanes). As a result, most of the movie's data will be on NUMA#3 or NUMA#4. ------- Note: I expect the NUMA-fabric to be faster than PCIe fabric. NUMA-fabric is designed so that different chips can pretend each other's RAM at roughly 50+GBps bandwidth at something like 300ns latencies. In contrast, PCIe 4.0 x16 is only 30GBps or so... and a lot of these NVMe SSDs are only 4x lanes (aka: 8GBps). You also need to handle all the networking tidbits: HTTPS has state (not just the TLS state, but also the state of the stream: who's turn is it to talk and such). Which means the CPU / Application data needs to be in the loop somehow. I know that the most recent talk offloads the process as much as possible thanks to the "sendfile" interface on FreeBSD (also available on Linux), which allows you to "pipe" data from one file-descriptor to another.