16 ms·

Netflix and FreeBSD: Using Open Source to Deliver Streaming Video [pdf]

132 points by vermaden 8y ago

2trill2spill 8y ago
I really like the work Netflix has done and open sourced back into FreeBSD [1], But I do take issue with one of the slides, where it says "Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections". Sorry but that's not FreeBSD, that's a Netflix proprietary fork of FreeBSD serving all that, until me and the rest of FreeBSD community can actually download and use TLS sendfile, Netflix should probably stop saying FreeBSD serves all this TLS traffic. [1]: https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s
- marktangotango 8y ago
  So the implication is sendfile does not use tls over an https connection for example? That is a shocking revelation to me if true!
  - runlevel1 8y ago
    TLS libraries like OpenSSL operate in user space (in-kernel TLS is very a recent addition to Linux). sendfile(2)'s benefit is that it doesn't require transferring the data back and forth between kernel space and user space. You point it to a file descriptor and a socket, and it simply copies the bytes -- all in kernel space.
    - messe 8y ago
      > in-kernel TLS is very a recent addition to Linux Linux has gotten in-kernel TLS? That's slightly horrifying. Seems like it's hard enough to get right without being allowed to overwrite any user page you'd like.
      - philjohn 8y ago
        Doesn't a lot of that come from the very long development history of OpenSSL (so best practices have developed over its lifetime), and the many, many compatibility hacks they still include in the codebase? It also contains a LOT of different things in there, whereas "just" a TLS stream with limited ciphers should be a lot simpler.
        toast0 8y ago
        Supporting TLS directly in a kernel is horrifying regardless of the legacy of OpenSSL. Having userspace do the session establishment (where all the weird stuff happens) and letting the kernel do the bulk ciphering is reasonable, especially if you're already using sendfile so the kernel can handle the bulk data sending. If something goes sideways, punt the socket back to userspace. Lots of kernels already have bulk ciphering for encrypted disks or ipsec (which is often very similar, userspace negotiates sessions and kernel handles per packet ciphering)
        ronsor 8y ago
        AFAIK that's all Linux's TLS implementation does -- ciphering & encryption, not session setup
      - runlevel1 8y ago
        Yes, but only the symmetric part. You handshake in userspace, then you pass the negotiated cipher keys/IV to the TCP socket with `setsockopt`. Last I looked, the only cipher supported was AES-128-GCM.
        caf 8y ago
        That's right. And if anything else happens on the connection (like a renegotiation) then it is just punted back to userspace.
        alacombe 8y ago
- brobdingnagians 8y ago
  To be fair, they do say "It is our intention to upstream any code which we can." and give the 3 reasons they have local commits, which all seem reasonable. I expect they will continue making contributions as they make finish features and generalize the features. Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. So the fact that FreeBSD has made an amazing code base that can and does serve as the foundation of Netflix's distribution network is great; it gives visibility to the OS and also meaningful code contributions from a large player who pushes it to the limits. We may not get all of those _right_now_ because of NDAs and the realities of life, but it is all helping to make it more awesome.
  - nordsieck 8y ago
    > Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. There is no "distribution" happening; all of Netflix's use of BSD is internal. Even if they had chosen to modify Linux instead, there would be no requirement to share their changes.
    - empthought 8y ago
      Pretty sure they distribute these appliances to ISPs...
      - nordsieck 8y ago
        > Pretty sure they distribute these appliances to ISPs... They co-locate them at ISP as a form of extremely distributed peering. Netflix, however, owns the boxes. In the context of licenses such as the GPLv2, distribution happens when software transfers from one party to another. This doesn't happen: the software always stays on Netflix owned devices.
- kev009 8y ago
  The most important part IMHO is here https://github.com/freebsd/freebsd/tree/master/sys/netinet/tcp_stacks https://github.com/freebsd/freebsd/tree/master/sys/netinet/t.... The TLS sendfile piece is a bit specialized, mainly interesting if you are doing the exact same workload i.e. a CDN on UFS disks. I can say on some authority (I had all the relevant code at my last job) NF will give it to you if you ask and have a need/competence. The reason the TLS bits aren't upstream is that they aren't general purpose or "finished" yet. It relies on some third party code outside of src like boringssl and Intel's isa-l.
- fefe23 8y ago
  I clicked on that video and at the timestamp in the link the guy says they like FreeBSD because it's not GPL. Not that we have anything against GPL or the Free Software Foundation, but the BSD license is more collaborative. I'm paraphrasing here. Is there any actual evidence for that claim? I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Why does FreeBSD then not have your TLS sendfile code? Linux has TLS sendfile. It was contributed by a corporation, as far as I know. Personally, I have always really liked the BSD projects. The people working on them have been nothing but professional and yet courteous, they tend to fix their security issues quickly. I would love for them to get paid properly. With money. Not with "exposure" or "we'll test your code".
  - boomboomsubban 8y ago
    >I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Where are you seeing these corporations ripping off FreeBSD? Netflix has contributed code in the past, currently over a thousand commits in FreeBSD, and there's no reason to doubt their intention to continue doing so.
    - alacombe 8y ago
      I worked for 5 years for a company taking from *BSD and not contributing much. Lately they started monetary donation, but no code.
      - protomyth 8y ago
        Weirdly, I actually think that's a feature not a bug. It keep ill intentioned idiots from gumming up the works because they are forced to contribute and really don't have the corporate beliefs or culture that would make them productive and not a distraction.
betaby 8y ago
There is ZERO technical details in this PDF beside 'we track head closely'
_eht 8y ago
Absolute shame flaunting open source as a backbone for DRM delivery.
kgwxd 8y ago
Using Open Source to Deliver Streaming Bytes You Hope Are Just Video
- _eht 8y ago
  ^ better title, please update OP
mikece 8y ago
Why is there such a visceral hatred for non-GPL software? Seems that every time the GPL vs. BSD (or equivalent) issue comes up there is much anger about companies doing proprietary work on top of FreeBSD. I'm curious about the opposite: code taken from BSD (or equivalent) projects and put into GPL code bases and then extended such that those improvements cannot be ported back to the BSD source. GPL proponents say this never happens but in multiple forums I've heard from folks who have had their permissive-licensed projects killed because their work was effectively GPLed and "extended" against their will. To me this boils down to how you see humanity: if you think we're essentially good and willing to help each other, permissive licensing makes more sense. If you think morality has to be enforced by contract because we cannot trust one another then GPL is the only choice. I chose permissive licensing because I don't see my fellow humans as essentially evil.
- pjmlp 8y ago
  As user of commercial software and ex-Linux zealot, I have my own glass ceiling, so take it with a grain of salt. Had Linux and GNU not been GPL, and we would all be using Aix, HP-UX, Solaris, ..., because none of the BSDs would have gotten back the contributions like it happened to Linux, and BSDs would still be just another UNIX derivative among many. Just look at the optimizations that some embedded OEMs, or even Sony, don't contribute back to LLVM. And speaking of Sony, not everything from PS4 OS has gotten back to BSD.
  - IntelMiner 8y ago
    The Playstation 3 and Vita also run/ran NetBSD, as well Sony also helped spawn "ToyBox". A BSD licensed replacement for the GPL "BusyBox" https://en.wikipedia.org/wiki/Toybox https://en.wikipedia.org/wiki/Toybox
- trumped 8y ago
  The problem might be that companies are considered to be people and it seems easier for companies to be evil.
gigatexal 8y ago
All of the discussion points in this thread are valid but what the slides omit is whether they using ZFS?
- dehrmann 8y ago
  Is ZFS that interesting for their use case?
  - gigatexal 8y ago
    Built in compression (probably not given the uncompressibility of video) but the protection against bit rot would be huge imo
- _emacsomancer_ 8y ago
  The last I'd heard, they were using UFS.
- m11r 8y ago
  Pretty sure it's still UFS+J. They've mentioned in the past that the benefits of ZFS don't really apply to the CDN boxes: if a drive dies, it's simply removed from the local cache, and files are re-downloaded/redistributed as necessary across the remaining cache drives the next time the appliance refreshes from its upstream origin. Additionally, ZFS prefetching was actually less performant in some cases from what I remember -- or at least providing no benefit -- because of how it couldn't really anticipate the I/O patterns of video segment requests from users. Additional reference: - https://news.ycombinator.com/item?id=8077594 https://news.ycombinator.com/item?id=8077594 ("…but caching is futile.") - https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4
- Cyclenerd 8y ago
  UFS. Source: 47:50min @ https://ftp.fau.de/fosdem/2019/Janson/netflix_freebsd.webm https://ftp.fau.de/fosdem/2019/Janson/netflix_freebsd.webm
  - gigatexal 8y ago
ksec 8y ago
I was a bit naive when I said I wish Netflix had use Fast.com as their brand of CDN. Turns out their whole CDN are tailored to their special needs of Video and Video only. So it wouldn't work nearly as well if there were many small files or other CDN use cases. What I really wished was I could have a CDN based on BSD. Even though you can't actually use BSD for lots of other reason, you could at last contribute a little by having some infrastructure that is built on top of BSD and you are paying for it. As far as I know only Limestone CDN are on FreeBSD, but unfortunately they only offer business to Enterprise.

132 points by vermaden 8y ago

I really like the work Netflix has done and open sourced back into FreeBSD [1], But I do take issue with one of the slides, where it says "Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections". Sorry but that's not FreeBSD, that's a Netflix proprietary fork of FreeBSD serving all that, until me and the rest of FreeBSD community can actually download and use TLS sendfile, Netflix should probably stop saying FreeBSD serves all this TLS traffic. [1]: https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s

marktangotango 8y ago
So the implication is sendfile does not use tls over an https connection for example? That is a shocking revelation to me if true!
- runlevel1 8y ago
  TLS libraries like OpenSSL operate in user space (in-kernel TLS is very a recent addition to Linux). sendfile(2)'s benefit is that it doesn't require transferring the data back and forth between kernel space and user space. You point it to a file descriptor and a socket, and it simply copies the bytes -- all in kernel space.
  - messe 8y ago
    > in-kernel TLS is very a recent addition to Linux Linux has gotten in-kernel TLS? That's slightly horrifying. Seems like it's hard enough to get right without being allowed to overwrite any user page you'd like.
    - philjohn 8y ago
      Doesn't a lot of that come from the very long development history of OpenSSL (so best practices have developed over its lifetime), and the many, many compatibility hacks they still include in the codebase? It also contains a LOT of different things in there, whereas "just" a TLS stream with limited ciphers should be a lot simpler.
      - toast0 8y ago
        Supporting TLS directly in a kernel is horrifying regardless of the legacy of OpenSSL. Having userspace do the session establishment (where all the weird stuff happens) and letting the kernel do the bulk ciphering is reasonable, especially if you're already using sendfile so the kernel can handle the bulk data sending. If something goes sideways, punt the socket back to userspace. Lots of kernels already have bulk ciphering for encrypted disks or ipsec (which is often very similar, userspace negotiates sessions and kernel handles per packet ciphering)
        ronsor 8y ago
        AFAIK that's all Linux's TLS implementation does -- ciphering & encryption, not session setup
    - runlevel1 8y ago
      Yes, but only the symmetric part. You handshake in userspace, then you pass the negotiated cipher keys/IV to the TCP socket with `setsockopt`. Last I looked, the only cipher supported was AES-128-GCM.
      - caf 8y ago
        That's right. And if anything else happens on the connection (like a renegotiation) then it is just punted back to userspace.
        alacombe 8y ago
brobdingnagians 8y ago
To be fair, they do say "It is our intention to upstream any code which we can." and give the 3 reasons they have local commits, which all seem reasonable. I expect they will continue making contributions as they make finish features and generalize the features. Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. So the fact that FreeBSD has made an amazing code base that can and does serve as the foundation of Netflix's distribution network is great; it gives visibility to the OS and also meaningful code contributions from a large player who pushes it to the limits. We may not get all of those _right_now_ because of NDAs and the realities of life, but it is all helping to make it more awesome.
- nordsieck 8y ago
  > Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. There is no "distribution" happening; all of Netflix's use of BSD is internal. Even if they had chosen to modify Linux instead, there would be no requirement to share their changes.
  - empthought 8y ago
    Pretty sure they distribute these appliances to ISPs...
    - nordsieck 8y ago
      > Pretty sure they distribute these appliances to ISPs... They co-locate them at ISP as a form of extremely distributed peering. Netflix, however, owns the boxes. In the context of licenses such as the GPLv2, distribution happens when software transfers from one party to another. This doesn't happen: the software always stays on Netflix owned devices.
kev009 8y ago
The most important part IMHO is here https://github.com/freebsd/freebsd/tree/master/sys/netinet/tcp_stacks https://github.com/freebsd/freebsd/tree/master/sys/netinet/t.... The TLS sendfile piece is a bit specialized, mainly interesting if you are doing the exact same workload i.e. a CDN on UFS disks. I can say on some authority (I had all the relevant code at my last job) NF will give it to you if you ask and have a need/competence. The reason the TLS bits aren't upstream is that they aren't general purpose or "finished" yet. It relies on some third party code outside of src like boringssl and Intel's isa-l.
fefe23 8y ago
I clicked on that video and at the timestamp in the link the guy says they like FreeBSD because it's not GPL. Not that we have anything against GPL or the Free Software Foundation, but the BSD license is more collaborative. I'm paraphrasing here. Is there any actual evidence for that claim? I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Why does FreeBSD then not have your TLS sendfile code? Linux has TLS sendfile. It was contributed by a corporation, as far as I know. Personally, I have always really liked the BSD projects. The people working on them have been nothing but professional and yet courteous, they tend to fix their security issues quickly. I would love for them to get paid properly. With money. Not with "exposure" or "we'll test your code".
- boomboomsubban 8y ago
  >I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Where are you seeing these corporations ripping off FreeBSD? Netflix has contributed code in the past, currently over a thousand commits in FreeBSD, and there's no reason to doubt their intention to continue doing so.
  - alacombe 8y ago
    I worked for 5 years for a company taking from *BSD and not contributing much. Lately they started monetary donation, but no code.
    - protomyth 8y ago
      Weirdly, I actually think that's a feature not a bug. It keep ill intentioned idiots from gumming up the works because they are forced to contribute and really don't have the corporate beliefs or culture that would make them productive and not a distraction.

betaby 8y ago

There is ZERO technical details in this PDF beside 'we track head closely'

_eht 8y ago

Absolute shame flaunting open source as a backbone for DRM delivery.

kgwxd 8y ago

Using Open Source to Deliver Streaming Bytes You Hope Are Just Video

_eht 8y ago
^ better title, please update OP

mikece 8y ago

Why is there such a visceral hatred for non-GPL software? Seems that every time the GPL vs. BSD (or equivalent) issue comes up there is much anger about companies doing proprietary work on top of FreeBSD. I'm curious about the opposite: code taken from BSD (or equivalent) projects and put into GPL code bases and then extended such that those improvements cannot be ported back to the BSD source. GPL proponents say this never happens but in multiple forums I've heard from folks who have had their permissive-licensed projects killed because their work was effectively GPLed and "extended" against their will. To me this boils down to how you see humanity: if you think we're essentially good and willing to help each other, permissive licensing makes more sense. If you think morality has to be enforced by contract because we cannot trust one another then GPL is the only choice. I chose permissive licensing because I don't see my fellow humans as essentially evil.

pjmlp 8y ago
As user of commercial software and ex-Linux zealot, I have my own glass ceiling, so take it with a grain of salt. Had Linux and GNU not been GPL, and we would all be using Aix, HP-UX, Solaris, ..., because none of the BSDs would have gotten back the contributions like it happened to Linux, and BSDs would still be just another UNIX derivative among many. Just look at the optimizations that some embedded OEMs, or even Sony, don't contribute back to LLVM. And speaking of Sony, not everything from PS4 OS has gotten back to BSD.
- IntelMiner 8y ago
  The Playstation 3 and Vita also run/ran NetBSD, as well Sony also helped spawn "ToyBox". A BSD licensed replacement for the GPL "BusyBox" https://en.wikipedia.org/wiki/Toybox https://en.wikipedia.org/wiki/Toybox
trumped 8y ago
The problem might be that companies are considered to be people and it seems easier for companies to be evil.

gigatexal 8y ago

All of the discussion points in this thread are valid but what the slides omit is whether they using ZFS?

dehrmann 8y ago
Is ZFS that interesting for their use case?
- gigatexal 8y ago
  Built in compression (probably not given the uncompressibility of video) but the protection against bit rot would be huge imo
_emacsomancer_ 8y ago
The last I'd heard, they were using UFS.
m11r 8y ago
Pretty sure it's still UFS+J. They've mentioned in the past that the benefits of ZFS don't really apply to the CDN boxes: if a drive dies, it's simply removed from the local cache, and files are re-downloaded/redistributed as necessary across the remaining cache drives the next time the appliance refreshes from its upstream origin. Additionally, ZFS prefetching was actually less performant in some cases from what I remember -- or at least providing no benefit -- because of how it couldn't really anticipate the I/O patterns of video segment requests from users. Additional reference: - https://news.ycombinator.com/item?id=8077594 https://news.ycombinator.com/item?id=8077594 ("…but caching is futile.") - https://www.youtube.com/watch?v=FL5U4wr86L4 https://www.youtube.com/watch?v=FL5U4wr86L4
Cyclenerd 8y ago
UFS. Source: 47:50min @ https://ftp.fau.de/fosdem/2019/Janson/netflix_freebsd.webm https://ftp.fau.de/fosdem/2019/Janson/netflix_freebsd.webm
- gigatexal 8y ago

ksec 8y ago

I was a bit naive when I said I wish Netflix had use Fast.com as their brand of CDN. Turns out their whole CDN are tailored to their special needs of Video and Video only. So it wouldn't work nearly as well if there were many small files or other CDN use cases. What I really wished was I could have a CDN based on BSD. Even though you can't actually use BSD for lots of other reason, you could at last contribute a little by having some infrastructure that is built on top of BSD and you are paying for it. As far as I know only Limestone CDN are on FreeBSD, but unfortunately they only offer business to Enterprise.