13 ms·

I really like the work Netflix has done and open sourced back into FreeBSD [1], But I do take issue with one of the slides, where it says "Using FreeBSD and com

I really like the work Netflix has done and open sourced back into FreeBSD [1], But I do take issue with one of the slides, where it says "Using FreeBSD and commodity parts, we achieve 90 Gb/s serving TLS-encrypted connections". Sorry but that's not FreeBSD, that's a Netflix proprietary fork of FreeBSD serving all that, until me and the rest of FreeBSD community can actually download and use TLS sendfile, Netflix should probably stop saying FreeBSD serves all this TLS traffic. [1]: https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s https://www.youtube.com/watch?v=4sZZN8Szh14&t=834s

marktangotango 8y ago
So the implication is sendfile does not use tls over an https connection for example? That is a shocking revelation to me if true!
- runlevel1 8y ago
  TLS libraries like OpenSSL operate in user space (in-kernel TLS is very a recent addition to Linux). sendfile(2)'s benefit is that it doesn't require transferring the data back and forth between kernel space and user space. You point it to a file descriptor and a socket, and it simply copies the bytes -- all in kernel space.
  - messe 8y ago
    > in-kernel TLS is very a recent addition to Linux Linux has gotten in-kernel TLS? That's slightly horrifying. Seems like it's hard enough to get right without being allowed to overwrite any user page you'd like.
    - philjohn 8y ago
      Doesn't a lot of that come from the very long development history of OpenSSL (so best practices have developed over its lifetime), and the many, many compatibility hacks they still include in the codebase? It also contains a LOT of different things in there, whereas "just" a TLS stream with limited ciphers should be a lot simpler.
      - toast0 8y ago
        Supporting TLS directly in a kernel is horrifying regardless of the legacy of OpenSSL. Having userspace do the session establishment (where all the weird stuff happens) and letting the kernel do the bulk ciphering is reasonable, especially if you're already using sendfile so the kernel can handle the bulk data sending. If something goes sideways, punt the socket back to userspace. Lots of kernels already have bulk ciphering for encrypted disks or ipsec (which is often very similar, userspace negotiates sessions and kernel handles per packet ciphering)
        ronsor 8y ago
        AFAIK that's all Linux's TLS implementation does -- ciphering & encryption, not session setup
    - runlevel1 8y ago
      Yes, but only the symmetric part. You handshake in userspace, then you pass the negotiated cipher keys/IV to the TCP socket with `setsockopt`. Last I looked, the only cipher supported was AES-128-GCM.
      - caf 8y ago
        That's right. And if anything else happens on the connection (like a renegotiation) then it is just punted back to userspace.
        alacombe 8y ago
        Which is the correct way to handle, fast data path in the kernel, slow control path in userspace.
      - messe 8y ago
        Ah, that's a lot more reasonable then.
    - takeda 8y ago
      Don't know about Linux implementation, but sendfile would only need AES, and today's CPUs even come with hardware support. I can't talk about security aspect, but it is not like the openssl is implemented in kernel or hardware, just the symmetric encryption and that only transforms the data.
brobdingnagians 8y ago
To be fair, they do say "It is our intention to upstream any code which we can." and give the 3 reasons they have local commits, which all seem reasonable. I expect they will continue making contributions as they make finish features and generalize the features. Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. So the fact that FreeBSD has made an amazing code base that can and does serve as the foundation of Netflix's distribution network is great; it gives visibility to the OS and also meaningful code contributions from a large player who pushes it to the limits. We may not get all of those _right_now_ because of NDAs and the realities of life, but it is all helping to make it more awesome.
- nordsieck 8y ago
  > Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. There is no "distribution" happening; all of Netflix's use of BSD is internal. Even if they had chosen to modify Linux instead, there would be no requirement to share their changes.
  - empthought 8y ago
    Pretty sure they distribute these appliances to ISPs...
    - nordsieck 8y ago
      > Pretty sure they distribute these appliances to ISPs... They co-locate them at ISP as a form of extremely distributed peering. Netflix, however, owns the boxes. In the context of licenses such as the GPLv2, distribution happens when software transfers from one party to another. This doesn't happen: the software always stays on Netflix owned devices.
kev009 8y ago
The most important part IMHO is here https://github.com/freebsd/freebsd/tree/master/sys/netinet/tcp_stacks https://github.com/freebsd/freebsd/tree/master/sys/netinet/t.... The TLS sendfile piece is a bit specialized, mainly interesting if you are doing the exact same workload i.e. a CDN on UFS disks. I can say on some authority (I had all the relevant code at my last job) NF will give it to you if you ask and have a need/competence. The reason the TLS bits aren't upstream is that they aren't general purpose or "finished" yet. It relies on some third party code outside of src like boringssl and Intel's isa-l.
fefe23 8y ago
I clicked on that video and at the timestamp in the link the guy says they like FreeBSD because it's not GPL. Not that we have anything against GPL or the Free Software Foundation, but the BSD license is more collaborative. I'm paraphrasing here. Is there any actual evidence for that claim? I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Why does FreeBSD then not have your TLS sendfile code? Linux has TLS sendfile. It was contributed by a corporation, as far as I know. Personally, I have always really liked the BSD projects. The people working on them have been nothing but professional and yet courteous, they tend to fix their security issues quickly. I would love for them to get paid properly. With money. Not with "exposure" or "we'll test your code".
- boomboomsubban 8y ago
  >I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Where are you seeing these corporations ripping off FreeBSD? Netflix has contributed code in the past, currently over a thousand commits in FreeBSD, and there's no reason to doubt their intention to continue doing so.
  - alacombe 8y ago
    I worked for 5 years for a company taking from *BSD and not contributing much. Lately they started monetary donation, but no code.
    - protomyth 8y ago
      Weirdly, I actually think that's a feature not a bug. It keep ill intentioned idiots from gumming up the works because they are forced to contribute and really don't have the corporate beliefs or culture that would make them productive and not a distraction.

marktangotango 8y ago

So the implication is sendfile does not use tls over an https connection for example? That is a shocking revelation to me if true!

runlevel1 8y ago
TLS libraries like OpenSSL operate in user space (in-kernel TLS is very a recent addition to Linux). sendfile(2)'s benefit is that it doesn't require transferring the data back and forth between kernel space and user space. You point it to a file descriptor and a socket, and it simply copies the bytes -- all in kernel space.
- messe 8y ago
  > in-kernel TLS is very a recent addition to Linux Linux has gotten in-kernel TLS? That's slightly horrifying. Seems like it's hard enough to get right without being allowed to overwrite any user page you'd like.
  - philjohn 8y ago
    Doesn't a lot of that come from the very long development history of OpenSSL (so best practices have developed over its lifetime), and the many, many compatibility hacks they still include in the codebase? It also contains a LOT of different things in there, whereas "just" a TLS stream with limited ciphers should be a lot simpler.
    - toast0 8y ago
      Supporting TLS directly in a kernel is horrifying regardless of the legacy of OpenSSL. Having userspace do the session establishment (where all the weird stuff happens) and letting the kernel do the bulk ciphering is reasonable, especially if you're already using sendfile so the kernel can handle the bulk data sending. If something goes sideways, punt the socket back to userspace. Lots of kernels already have bulk ciphering for encrypted disks or ipsec (which is often very similar, userspace negotiates sessions and kernel handles per packet ciphering)
      - ronsor 8y ago
        AFAIK that's all Linux's TLS implementation does -- ciphering & encryption, not session setup
  - runlevel1 8y ago
    Yes, but only the symmetric part. You handshake in userspace, then you pass the negotiated cipher keys/IV to the TCP socket with `setsockopt`. Last I looked, the only cipher supported was AES-128-GCM.
    - caf 8y ago
      That's right. And if anything else happens on the connection (like a renegotiation) then it is just punted back to userspace.
      - alacombe 8y ago
        Which is the correct way to handle, fast data path in the kernel, slow control path in userspace.
    - messe 8y ago
      Ah, that's a lot more reasonable then.
  - takeda 8y ago
    Don't know about Linux implementation, but sendfile would only need AES, and today's CPUs even come with hardware support. I can't talk about security aspect, but it is not like the openssl is implemented in kernel or hardware, just the symmetric encryption and that only transforms the data.

brobdingnagians 8y ago

To be fair, they do say "It is our intention to upstream any code which we can." and give the 3 reasons they have local commits, which all seem reasonable. I expect they will continue making contributions as they make finish features and generalize the features. Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. So the fact that FreeBSD has made an amazing code base that can and does serve as the foundation of Netflix's distribution network is great; it gives visibility to the OS and also meaningful code contributions from a large player who pushes it to the limits. We may not get all of those _right_now_ because of NDAs and the realities of life, but it is all helping to make it more awesome.

nordsieck 8y ago
> Part of the allure, uniqueness, and awesomeness of the *BSDs is that they _can_ be modified and forked in this way because of the license. There is no "distribution" happening; all of Netflix's use of BSD is internal. Even if they had chosen to modify Linux instead, there would be no requirement to share their changes.
- empthought 8y ago
  Pretty sure they distribute these appliances to ISPs...
  - nordsieck 8y ago
    > Pretty sure they distribute these appliances to ISPs... They co-locate them at ISP as a form of extremely distributed peering. Netflix, however, owns the boxes. In the context of licenses such as the GPLv2, distribution happens when software transfers from one party to another. This doesn't happen: the software always stays on Netflix owned devices.

kev009 8y ago

The most important part IMHO is here https://github.com/freebsd/freebsd/tree/master/sys/netinet/tcp_stacks https://github.com/freebsd/freebsd/tree/master/sys/netinet/t.... The TLS sendfile piece is a bit specialized, mainly interesting if you are doing the exact same workload i.e. a CDN on UFS disks. I can say on some authority (I had all the relevant code at my last job) NF will give it to you if you ask and have a need/competence. The reason the TLS bits aren't upstream is that they aren't general purpose or "finished" yet. It relies on some third party code outside of src like boringssl and Intel's isa-l.

fefe23 8y ago

I clicked on that video and at the timestamp in the link the guy says they like FreeBSD because it's not GPL. Not that we have anything against GPL or the Free Software Foundation, but the BSD license is more collaborative. I'm paraphrasing here. Is there any actual evidence for that claim? I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Why does FreeBSD then not have your TLS sendfile code? Linux has TLS sendfile. It was contributed by a corporation, as far as I know. Personally, I have always really liked the BSD projects. The people working on them have been nothing but professional and yet courteous, they tend to fix their security issues quickly. I would love for them to get paid properly. With money. Not with "exposure" or "we'll test your code".

boomboomsubban 8y ago
>I'm a Linux guy so my view is probably biased. All I'm seeing is BSD licensed projects being ripped off by corporations left and right. Maybe we mean different things with the word, sharing. Netflix using FreeBSD and their idea of giving back to the project is telling them this will lead to their code being tested really well? That sounds to me like "please create a web page for us for free, it will lead to more exposure for you". Where are you seeing these corporations ripping off FreeBSD? Netflix has contributed code in the past, currently over a thousand commits in FreeBSD, and there's no reason to doubt their intention to continue doing so.
- alacombe 8y ago
  I worked for 5 years for a company taking from *BSD and not contributing much. Lately they started monetary donation, but no code.
  - protomyth 8y ago
    Weirdly, I actually think that's a feature not a bug. It keep ill intentioned idiots from gumming up the works because they are forced to contribute and really don't have the corporate beliefs or culture that would make them productive and not a distraction.