6 ms·

This release includes most of the kTLS and NUMA work that I talked about at EuroBSDCon in 2019, including hw kTLS offload support for some NICs. This allows Ne

This release includes most of the kTLS and NUMA work that I talked about at EuroBSDCon in 2019, including hw kTLS offload support for some NICs. This allows Netflix to serve over 350Gb/s of TLS encrypted video streams from a single 32-core AMD based server (at about 50% CPU).

aduitsis 6y ago
Drew, thank you kindly (and Netflix) for your contributions to FreeBSD. They do not go unnoticed. I, for one, learned quite a few things too out of your presentation, which is here: https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in%20the%20FreeBSD%20Network%20Stack%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in...
- drewg123 6y ago
  Thanks so much. I love that my "war stories" are interesting to people other than me :)
  - ta988 6y ago
    They absolutely are, thank you!
  - csdreamer7 6y ago
    Same. They are excellent.
- drewg123 6y ago
  And one more thing.. A lot of the core NUMA work, especially in the VM system, was done by Jeff Roberson, not me. I did a lot of hacks and Jeff turned them into production quality code. Without that work, my network siloing would have been useless. More people will benefit from the core work, since its not just geared to a web workload. And we have John Baldwin to thank for upstreaming the Netflix kTLS.
jiggawatts 6y ago
One thing I haven't quite understood is why Netflix even bothers to encrypt the video streams. It seems like a waste of effort, since Netflix isn't exactly serving content that would qualify as "secret". There's not even any porn on there, so it's not the biggest concern if someone were to find out what some people were watching. This data is being collected anyway via endpoint technologies like Samsung's Automated Content Recognition (ACR): https://www.samsung.com/us/business/samsungads/resources/tv-ad-retargeting/ https://www.samsung.com/us/business/samsungads/resources/tv-... It seems like encrypting the video stream traffic at this kind of scale requires a substantial engineering effort and special hardware. Encryption also means that shared downstream caches at ISPs can't do anything to reduce their bandwidth consumption. The only option is to contact Netflix and physically install one of their edge caching boxes. Content tampering protection could be obtained by simply computing content hashes offline once, which the clients could verify.
- freedomben 6y ago
  I would guess it's for DRM.
  - paulgdp 6y ago
    Encryption for DRM needs to happen only once per media, and then can be saved to storage and shared infinitely many times. So that's probably something else
- stock_toaster 6y ago
  My guess would be to prevent middlebox fiddling. Such as an end user device asking for 4k, but a middlebox somewhere between the customer and Netflix inspecting the http content and blocking it (or modifying the client request in flight), because "only 480p allowed here" or something like that.
  - mrpippy 6y ago
    US cellular carriers do that by throttling bandwidth to the network ranges used by video providers
forty 6y ago
Hello, I'm genuinely curious: why did you implement it in FreeBSD rather than Linux? Isn't that kind of offload something that already exists in Linux? Or does something in the way Linux is implemented/designed that makes it impractical / impossible to implement as efficiently? Or is somehow Linux not secure enough for your use case? Or not something else enough? I'm asking because I'm always interested to know what are the practical limitations of Linux and also because, while I think it's nice that Linux had some serious competitors, I cannot help being a bit sad that contribution like this will only benefit the few people that use FreeBSD (even if to be fair, it seems that it will benefit all netflix customers at least, which is quite a few people ;) )
- nix23 6y ago
  >I cannot help being a bit sad that contribution like this will only benefit the few people that use FreeBSD Same could be said the other way around, the thing is Linux can implement it and even take the code, the other way is rather difficult....so much for freedom. And not sure what you mean with "few" people, the few Playstation 3/4/5 users? Juniper? OpnSense/Pfsense installations? Or the few Truenas/Freenas users? >Or not something else enough? https://www.youtube.com/watch?v=KP_bKvXkoC4 https://www.youtube.com/watch?v=KP_bKvXkoC4
  - forty 6y ago
    I still expect the number to be pretty small compared to the number of Linux users, no matter how you define user (if playstation users counts as freebsd users, then I guess nearly everyone is a Linux user :) ) The point of the licence is a good point. Is the kind of code drewg123 is referring to regularly copied from BSD to Linux? (i understand it should be possible in this direction?). Thanks for the video, looks like exactly what I was looking for, I will watch it later.
    - nix23 6y ago
      >small compared to the number of Linux users Not to mention all the Minix and NucleusRTOS Users ;)

aduitsis 6y ago

Drew, thank you kindly (and Netflix) for your contributions to FreeBSD. They do not go unnoticed. I, for one, learned quite a few things too out of your presentation, which is here: https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in%20the%20FreeBSD%20Network%20Stack%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/NUMA%20Optimizations%20in...

drewg123 6y ago
Thanks so much. I love that my "war stories" are interesting to people other than me :)
- ta988 6y ago
  They absolutely are, thank you!
- csdreamer7 6y ago
  Same. They are excellent.
drewg123 6y ago
And one more thing.. A lot of the core NUMA work, especially in the VM system, was done by Jeff Roberson, not me. I did a lot of hacks and Jeff turned them into production quality code. Without that work, my network siloing would have been useless. More people will benefit from the core work, since its not just geared to a web workload. And we have John Baldwin to thank for upstreaming the Netflix kTLS.

jiggawatts 6y ago

One thing I haven't quite understood is why Netflix even bothers to encrypt the video streams. It seems like a waste of effort, since Netflix isn't exactly serving content that would qualify as "secret". There's not even any porn on there, so it's not the biggest concern if someone were to find out what some people were watching. This data is being collected anyway via endpoint technologies like Samsung's Automated Content Recognition (ACR): https://www.samsung.com/us/business/samsungads/resources/tv-ad-retargeting/ https://www.samsung.com/us/business/samsungads/resources/tv-... It seems like encrypting the video stream traffic at this kind of scale requires a substantial engineering effort and special hardware. Encryption also means that shared downstream caches at ISPs can't do anything to reduce their bandwidth consumption. The only option is to contact Netflix and physically install one of their edge caching boxes. Content tampering protection could be obtained by simply computing content hashes offline once, which the clients could verify.

freedomben 6y ago
I would guess it's for DRM.
- paulgdp 6y ago
  Encryption for DRM needs to happen only once per media, and then can be saved to storage and shared infinitely many times. So that's probably something else
stock_toaster 6y ago
My guess would be to prevent middlebox fiddling. Such as an end user device asking for 4k, but a middlebox somewhere between the customer and Netflix inspecting the http content and blocking it (or modifying the client request in flight), because "only 480p allowed here" or something like that.
- mrpippy 6y ago
  US cellular carriers do that by throttling bandwidth to the network ranges used by video providers

forty 6y ago

Hello, I'm genuinely curious: why did you implement it in FreeBSD rather than Linux? Isn't that kind of offload something that already exists in Linux? Or does something in the way Linux is implemented/designed that makes it impractical / impossible to implement as efficiently? Or is somehow Linux not secure enough for your use case? Or not something else enough? I'm asking because I'm always interested to know what are the practical limitations of Linux and also because, while I think it's nice that Linux had some serious competitors, I cannot help being a bit sad that contribution like this will only benefit the few people that use FreeBSD (even if to be fair, it seems that it will benefit all netflix customers at least, which is quite a few people ;) )

nix23 6y ago
>I cannot help being a bit sad that contribution like this will only benefit the few people that use FreeBSD Same could be said the other way around, the thing is Linux can implement it and even take the code, the other way is rather difficult....so much for freedom. And not sure what you mean with "few" people, the few Playstation 3/4/5 users? Juniper? OpnSense/Pfsense installations? Or the few Truenas/Freenas users? >Or not something else enough? https://www.youtube.com/watch?v=KP_bKvXkoC4 https://www.youtube.com/watch?v=KP_bKvXkoC4
- forty 6y ago
  I still expect the number to be pretty small compared to the number of Linux users, no matter how you define user (if playstation users counts as freebsd users, then I guess nearly everyone is a Linux user :) ) The point of the licence is a good point. Is the kind of code drewg123 is referring to regularly copied from BSD to Linux? (i understand it should be possible in this direction?). Thanks for the video, looks like exactly what I was looking for, I will watch it later.
  - nix23 6y ago
    >small compared to the number of Linux users Not to mention all the Minix and NucleusRTOS Users ;)