4 ms·

> at triple to quadruple the bandwidth at the cost of 5-10 ns higher load-to-use You need to make full use of the bandwidth for this to be an overall win.

drewg123 7y ago
Once you use more than 75% of the available DRAM bandwidth, things start to get wonky, with large unpredictable CPU spikes, as the CPU stalls increase. So I think you'd see a definite benefit if you use more than 75% of the available bandwidth. We (Netflix CDN) are currently bandwidth limited for serving TLS encrypted video using software ktls. (~100Gb/s on broadwell, 150-160Gb/s on skylake/cascade lake, 220Gb/s on EPYC Rome). So I'd love to have twice the bandwidth. With twice the bandwidth, I think we could probably serve 400Gb/s from a single Rome based box. Note that I'm also looking forward to trying new kTLS offload NICs that should cut the bandwidth requirements.. those are a much more viable short-term path forward.
- e12e 7y ago
  Wow. What kind of networking hardware do you push to/through? I have a hard time visualizing where/how you would "put" 200 gbps worth of data over (any) network...
  - joaovictortr 7y ago
    There is a presentation from EuroBSDCon 2019, that describes the techniques and hardware used to achieve that figure [0]. There is also a thread in HN if you're interested [1]. [0] https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf [1] https://news.ycombinator.com/item?id=21043765 https://news.ycombinator.com/item?id=21043765
  - drewg123 7y ago
    Multiple Mellanox or Chelsio 100GbE NICs in an LACP bundle. The 220g is from a prototype with a Rome CPU in a Naples-designed motherboard, with 4 Mellanox ConnectX-4 100GbE NICs. 2 of them are limited to Gen 3 x8 (since it is a Naples generation mobo).
    - Teknoman117 7y ago
      I'm curious about you being limited to 220 Gb/s on an AMD Rome box. The maximum theoretical DDR4 bandwidth on such a system with 3200 MT DDR4 is 204 GiB/s, so you'd only be using ~15% of the DRAM bandwidth assuming the block/record/file size of the cached data you're serving is enough to mask the DRAM latency.
  - rasz 7y ago
    as to where, is internet infrastructure this bad in US? Even something like 35C3 had 500gbit pipe last year https://media.ccc.de/v/35c3-9576-35c3_infrastructure_review https://media.ccc.de/v/35c3-9576-35c3_infrastructure_review
    - kev009 7y ago
      No, the US is one of the most densely connected countries in the world. Fiber was over-provisioned by the functional Bell monopoly, railroads, and a smattering of upstarts in the 1980s and massive infrastructure capital bubble in the 2000 dotcom crash which was more about this infrastructure than software or computer hardware. If you hear people in the US complaining about their internet connection, it is because the local duopoly of telephone and cable company is failing them. In terms of industrial fiber, the competition is fierce and you can light up Nx100G to almost anywhere in the country.
    - Teknoman117 7y ago
      The industrial fiber capacity is fantastic here. What sucks is the consumer internet service. In most places in the US, there is a monopoly/duopoly for consumer ISPs and they certainly abuse their positions.
- baybal2 7y ago
  Wow, you still handle TLS in software? I worked on a few CDN projects, and on a DC project for Alibaba. Even in 2013, I think, all big CDNs used Intel hardware accelerators. The beauty of this approach is that if you need more TLS bandwidth, you just plug more TLS cards, and you are done. Otherwise, it's impressive to see how much TLS throughput people can get on CPUs today. Back in 2013, the best accelerator was only able to pull like 30Gb/s per card
  - rasz 7y ago
    How much are the accelerators? something tells me more expensive than delivering comparable throughput, highly flexible off the shelf x86 hardware.
    - Unklejoe 7y ago
      You're also not tied to any particular vendor with a pure software approach.
      - wmf 7y ago
        Realistically with something well-defined like TLS you use a library that supports pure software or hardware acceleration. Then you can use acceleration without any lock-in.
    - baybal2 7y ago
      Back in 2013, Intel 8920 did cost $750 wholesale, and similar Broadcom card a little bit cheaper. Looks a lot now, but back then you didn't have server CPUs with gazillion cores on the cheap. It was also about getting more bandwidth to a single server. 20Gb/s was a practical limit per mainstream server, but with accelerators you could've multiplied it many times more than the fastest CPU on the market.
      - snark42 7y ago
        > Looks a lot now, but back then you didn't have server CPUs with gazillion cores on the cheap. You also didn't have the AES instruction set to improve the SSL performance.
- fluffything 7y ago
  > We (Netflix CDN) are currently bandwidth limited for serving TLS encrypted video using software ktls. (~100Gb/s on broadwell, 150-160Gb/s on skylake/cascade lake, 220Gb/s on EPYC Rome). So I'd love to have twice the bandwidth. With twice the bandwidth, I think we could probably serve 400Gb/s from a single Rome based box. When I commented, I was assuming that people would be moving memory from RAM to the CPU to do something useful with it. If you are just dumping memory to the network, it would be more efficient to skip moving the memory to the CPU all together, or to even skip a normal CPU completely and just have network cards with a lot of RAM.

drewg123 7y ago

Once you use more than 75% of the available DRAM bandwidth, things start to get wonky, with large unpredictable CPU spikes, as the CPU stalls increase. So I think you'd see a definite benefit if you use more than 75% of the available bandwidth. We (Netflix CDN) are currently bandwidth limited for serving TLS encrypted video using software ktls. (~100Gb/s on broadwell, 150-160Gb/s on skylake/cascade lake, 220Gb/s on EPYC Rome). So I'd love to have twice the bandwidth. With twice the bandwidth, I think we could probably serve 400Gb/s from a single Rome based box. Note that I'm also looking forward to trying new kTLS offload NICs that should cut the bandwidth requirements.. those are a much more viable short-term path forward.

e12e 7y ago
Wow. What kind of networking hardware do you push to/through? I have a hard time visualizing where/how you would "put" 200 gbps worth of data over (any) network...
- joaovictortr 7y ago
  There is a presentation from EuroBSDCon 2019, that describes the techniques and hardware used to achieve that figure [0]. There is also a thread in HN if you're interested [1]. [0] https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf [1] https://news.ycombinator.com/item?id=21043765 https://news.ycombinator.com/item?id=21043765
- drewg123 7y ago
  Multiple Mellanox or Chelsio 100GbE NICs in an LACP bundle. The 220g is from a prototype with a Rome CPU in a Naples-designed motherboard, with 4 Mellanox ConnectX-4 100GbE NICs. 2 of them are limited to Gen 3 x8 (since it is a Naples generation mobo).
  - Teknoman117 7y ago
    I'm curious about you being limited to 220 Gb/s on an AMD Rome box. The maximum theoretical DDR4 bandwidth on such a system with 3200 MT DDR4 is 204 GiB/s, so you'd only be using ~15% of the DRAM bandwidth assuming the block/record/file size of the cached data you're serving is enough to mask the DRAM latency.
- rasz 7y ago
  as to where, is internet infrastructure this bad in US? Even something like 35C3 had 500gbit pipe last year https://media.ccc.de/v/35c3-9576-35c3_infrastructure_review https://media.ccc.de/v/35c3-9576-35c3_infrastructure_review
  - kev009 7y ago
    No, the US is one of the most densely connected countries in the world. Fiber was over-provisioned by the functional Bell monopoly, railroads, and a smattering of upstarts in the 1980s and massive infrastructure capital bubble in the 2000 dotcom crash which was more about this infrastructure than software or computer hardware. If you hear people in the US complaining about their internet connection, it is because the local duopoly of telephone and cable company is failing them. In terms of industrial fiber, the competition is fierce and you can light up Nx100G to almost anywhere in the country.
  - Teknoman117 7y ago
    The industrial fiber capacity is fantastic here. What sucks is the consumer internet service. In most places in the US, there is a monopoly/duopoly for consumer ISPs and they certainly abuse their positions.
baybal2 7y ago
Wow, you still handle TLS in software? I worked on a few CDN projects, and on a DC project for Alibaba. Even in 2013, I think, all big CDNs used Intel hardware accelerators. The beauty of this approach is that if you need more TLS bandwidth, you just plug more TLS cards, and you are done. Otherwise, it's impressive to see how much TLS throughput people can get on CPUs today. Back in 2013, the best accelerator was only able to pull like 30Gb/s per card
- rasz 7y ago
  How much are the accelerators? something tells me more expensive than delivering comparable throughput, highly flexible off the shelf x86 hardware.
  - Unklejoe 7y ago
    You're also not tied to any particular vendor with a pure software approach.
    - wmf 7y ago
      Realistically with something well-defined like TLS you use a library that supports pure software or hardware acceleration. Then you can use acceleration without any lock-in.
  - baybal2 7y ago
    Back in 2013, Intel 8920 did cost $750 wholesale, and similar Broadcom card a little bit cheaper. Looks a lot now, but back then you didn't have server CPUs with gazillion cores on the cheap. It was also about getting more bandwidth to a single server. 20Gb/s was a practical limit per mainstream server, but with accelerators you could've multiplied it many times more than the fastest CPU on the market.
    - snark42 7y ago
      > Looks a lot now, but back then you didn't have server CPUs with gazillion cores on the cheap. You also didn't have the AES instruction set to improve the SSL performance.
fluffything 7y ago
> We (Netflix CDN) are currently bandwidth limited for serving TLS encrypted video using software ktls. (~100Gb/s on broadwell, 150-160Gb/s on skylake/cascade lake, 220Gb/s on EPYC Rome). So I'd love to have twice the bandwidth. With twice the bandwidth, I think we could probably serve 400Gb/s from a single Rome based box. When I commented, I was assuming that people would be moving memory from RAM to the CPU to do something useful with it. If you are just dumping memory to the network, it would be more efficient to skip moving the memory to the CPU all together, or to even skip a normal CPU completely and just have network cards with a lot of RAM.