3 ms·

Netflix employee here: FreeBSD generally scales well for our workload, and where it doesn't, we can improve it and upstream the fixes without a lot of friction

Netflix employee here: FreeBSD generally scales well for our workload, and where it doesn't, we can improve it and upstream the fixes without a lot of friction. Right now, we're serving 200Gb/s of kTLS encrypted video on a single box (see talk at https://youtu.be/8NSzkYSX5nY https://youtu.be/8NSzkYSX5nY), and looking at 400Gb/s. I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it. In fact, as far as I know, some other big CDN providers that use Linux are just now moving to 100GbE, where as we've been serving at 100Gb/s in production for almost 4 years now on FreeBSD.

ksec 7y ago
Yes! I have been following your crazy work pushing the kLTS limit running into Memory bandwidth problems, and things get wonky once it reaches 75% of max bandwidth. I have always assume FreeBSD's performance was sort of a pride within, so I was surprised another Netflix employees being rather dismissive. ( Edit: I just spend 15min searching for it and couldn't find the comment. I should have favourite that. May be it was my false memory ) Which is one reason why I wish there are CDN running on FreeBSD. So you have an ecosystem / business around FreeBSD ( partly to save guard its future ). AFAIK the only CDN running on FreeBSD is Limelight Network, and they dont really do any pay as you go pricing, everything is ask for quote. I think not too long ago I floated the idea that Netflix should open a Video Delivery CDN with the Fast.com branding. ( In hindsight it was a silly idea ) And it was for the same reason, more FreeBSD usage = more vested interest = long term sustainability.
pdimitar 7y ago
Would you recommend a few specific server parts that help you achieve those numbers (in addition to FreeBSD). I am curious about constructing a 100GbE network in my neighbourhood.
- drewg123 7y ago
  We use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4 x8). The Mellanox (and Chelsio) NICs are helpful for us because they support RSS assisted TCP LRO. If you're doing just plain packet forwarding, that would not matter.
  - pdimitar 7y ago
    Thanks a lot! I am not sure about the plain packet forwarding or if I am gonna do more, but a few hundred (or thousand) bucks more sound pretty good for future-proofing a neighbourhood level of networking. Pretty valuable info for a hobbyist looking to get better (and trying to get into community work). Thank you.
  - gonzo 7y ago
    Well, the RSS does, but LRO does not.
    - drewg123 7y ago
      RSS assisted LRO is a FreeBSD feature where LRO holds batches of hundreds or thousands of packets, then sorts packets by arrival time and LRO hash result. That puts packets from the same connection adjacent to each other, and lets LRO combine them. Delivering packets in batches of hundreds requires some work from the driver (and requires the driver use a new LRO API). At least on our (Netflix) workload with tens of thousands of active connections, RSS assisted LRO increases our aggregation rate from almost nothing to about 2:1, and saves about 10% CPU. Its been on my TODO list to add RSS-assisted LRO support to iflib, which would give the intel and broadcom 100g drivers access to it, along with assorted Intel and other 1g, 10g, and 40g devices. But it still hasn't happened yet.
      - cthalupa 7y ago
        I think he was saying that RSS would still matter for plain packet forwarding.
cthalupa 7y ago
>I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it. You guys have had to do some considerable (from my interpretation of your presentations, at least) work to make this happen even in FreeBSD - do you think that a similar level of effort working in Linux would put it within reach there, too? It seems like there's been a lot of people doing some serious packet pushing with XDP.

ksec 7y ago

Yes! I have been following your crazy work pushing the kLTS limit running into Memory bandwidth problems, and things get wonky once it reaches 75% of max bandwidth. I have always assume FreeBSD's performance was sort of a pride within, so I was surprised another Netflix employees being rather dismissive. ( Edit: I just spend 15min searching for it and couldn't find the comment. I should have favourite that. May be it was my false memory ) Which is one reason why I wish there are CDN running on FreeBSD. So you have an ecosystem / business around FreeBSD ( partly to save guard its future ). AFAIK the only CDN running on FreeBSD is Limelight Network, and they dont really do any pay as you go pricing, everything is ask for quote. I think not too long ago I floated the idea that Netflix should open a Video Delivery CDN with the Fast.com branding. ( In hindsight it was a silly idea ) And it was for the same reason, more FreeBSD usage = more vested interest = long term sustainability.

pdimitar 7y ago

Would you recommend a few specific server parts that help you achieve those numbers (in addition to FreeBSD). I am curious about constructing a 100GbE network in my neighbourhood.

drewg123 7y ago
We use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4 x8). The Mellanox (and Chelsio) NICs are helpful for us because they support RSS assisted TCP LRO. If you're doing just plain packet forwarding, that would not matter.
- pdimitar 7y ago
  Thanks a lot! I am not sure about the plain packet forwarding or if I am gonna do more, but a few hundred (or thousand) bucks more sound pretty good for future-proofing a neighbourhood level of networking. Pretty valuable info for a hobbyist looking to get better (and trying to get into community work). Thank you.
- gonzo 7y ago
  Well, the RSS does, but LRO does not.
  - drewg123 7y ago
    RSS assisted LRO is a FreeBSD feature where LRO holds batches of hundreds or thousands of packets, then sorts packets by arrival time and LRO hash result. That puts packets from the same connection adjacent to each other, and lets LRO combine them. Delivering packets in batches of hundreds requires some work from the driver (and requires the driver use a new LRO API). At least on our (Netflix) workload with tens of thousands of active connections, RSS assisted LRO increases our aggregation rate from almost nothing to about 2:1, and saves about 10% CPU. Its been on my TODO list to add RSS-assisted LRO support to iflib, which would give the intel and broadcom 100g drivers access to it, along with assorted Intel and other 1g, 10g, and 40g devices. But it still hasn't happened yet.
    - cthalupa 7y ago
      I think he was saying that RSS would still matter for plain packet forwarding.

cthalupa 7y ago

>I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it. You guys have had to do some considerable (from my interpretation of your presentations, at least) work to make this happen even in FreeBSD - do you think that a similar level of effort working in Linux would put it within reach there, too? It seems like there's been a lot of people doing some serious packet pushing with XDP.