5 ms·
As long as FreeBSD has sustainable interest among cooperate usage it should be fine, but I am not sure if the number of FreeBSD client / cooperate user are grow
by ksec 7y ago
As long as FreeBSD has sustainable interest among cooperate usage it should be fine, but I am not sure if the number of FreeBSD client / cooperate user are growing or shrinking.
WhatsApp moved away from FreeBSD to Linux. And last time there were a few Netflix employees mentioned FreeBSD were used on Open Appliance for historical reason, not technical. So I would not be surprised if someday they move to linux as well. ( Once it offer similar performance )
Mellanox loves FreeBSD, but I am not sure if the same could be said for their new owner Nvidia.
OpenBSD and NetBSD are both an easy sell, one focus on Security and the other on Embedded. I am not sure how to sell FreeBSD, and it needs focus and direction. May be aiming for Network Appliance which is what the majority of its cooperate customer are using it for, where it could offer greater value.
- trasz 7y agoLast time there were Netflix employees claiming Linux can't do more than 60% of FreeBSD's performance in their workloads. Pretty good technical reason if you ask me.
- a012 7y agoNetflix has modified FreeBSD networking stacks to serve 100 Gbps at edge. But it's just their secret juice.
- drewg123 7y agoNothing is secret. AFAIK, everything is upstream in 13-current EDIT: The above applies mostly to the kernel. I've run a vanilla FreeBSD kernel on one of our appliances and gotten good performance (and served at 100Gb/s). In terms of userspace, openssl support for KTLS is only available in OpenSSL's master tree. We'd like to bring them back into the stable OpenSSL that's in FreeBSD's contrib, but we're not sure that's going to happen. And there are some patches to support kTLS with ngninx that are floating around in various forms; I'm pretty sure Mellanox has been giving out a version that should work for FreeBSD as well as Linux ktls.
- trasz 7y agoSame way Google modified Linux to serve whatever they need it to. That’s how Open Source works. (And yet, 60%. Curious.)
- morning_gelato 7y agoCould you please provide a link to those discussions/benchmarks regarding 60% performance? In the presentations by Netflix engineers that I have seen, they've primarily cited licensing as the reason they chose FreeBSD over Linux. It would certainly be interesting to see a performance comparison between Netflix's open connect appliances with Google's global cache servers.
- drewg123 7y agoNetflix employee here: FreeBSD generally scales well for our workload, and where it doesn't, we can improve it and upstream the fixes without a lot of friction. Right now, we're serving 200Gb/s of kTLS encrypted video on a single box (see talk at https://youtu.be/8NSzkYSX5nY https://youtu.be/8NSzkYSX5nY), and looking at 400Gb/s. I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it. In fact, as far as I know, some other big CDN providers that use Linux are just now moving to 100GbE, where as we've been serving at 100Gb/s in production for almost 4 years now on FreeBSD.
- ksec 7y agoYes! I have been following your crazy work pushing the kLTS limit running into Memory bandwidth problems, and things get wonky once it reaches 75% of max bandwidth. I have always assume FreeBSD's performance was sort of a pride within, so I was surprised another Netflix employees being rather dismissive. ( Edit: I just spend 15min searching for it and couldn't find the comment. I should have favourite that. May be it was my false memory ) Which is one reason why I wish there are CDN running on FreeBSD. So you have an ecosystem / business around FreeBSD ( partly to save guard its future ). AFAIK the only CDN running on FreeBSD is Limelight Network, and they dont really do any pay as you go pricing, everything is ask for quote. I think not too long ago I floated the idea that Netflix should open a Video Delivery CDN with the Fast.com branding. ( In hindsight it was a silly idea ) And it was for the same reason, more FreeBSD usage = more vested interest = long term sustainability.
- pdimitar 7y agoWould you recommend a few specific server parts that help you achieve those numbers (in addition to FreeBSD). I am curious about constructing a 100GbE network in my neighbourhood.
- drewg123 7y agoWe use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4 x8). The Mellanox (and Chelsio) NICs are helpful for us because they support RSS assisted TCP LRO. If you're doing just plain packet forwarding, that would not matter.
- laputan_machine 7y agoat FOSDEM the Netflix guy there said they use FreeBSD because of licensing reasons. [1] This is because they've changed certain parts and don't want to release some changes to the community. https://archive.fosdem.org/2019/schedule/event/netflix_freebsd/ https://archive.fosdem.org/2019/schedule/event/netflix_freeb... (first question covers it @ 44m~)
- edgyquant 7y agoThey should only have to release the changes if they publicly release the modified OS.
- drewg123 7y agoIts tricky. For business reasons, Netflix's ISP partners own the Openconnect appliances that are hosted in their data centers. So I think they could request the sources for GPL compliance.
- csdreamer7 7y agoThank you for posting this Drew. It's easy to dismiss companies license concerns over the GPL as ignorance, because, they usually are. But this is a good reason that I have never heard of.
- toast0 7y ago> WhatsApp moved away from FreeBSD to Linux. I'm a former WhatsApp server engineer[1]. WhatsApp primarily moved from bare metal hosting running FreeBSD to Facebook's owned and operating containerized management system which incidentally runs Linux. We did not make a technical choice to abandon FreeBSD in favor or something else, we made an organizational choice to abandon external hosting in favor of owned and operated hosting which required a lot of technical changes, one of which was switching operating systems. About half of the server engineering team pre-acquisition was former Yahoo employees, and we had seen what happens when an acquired company does not assimilate the acquirer's basic tech stack --- every issue with hardware or software first has to go through the triage step of is it broken because you're not running the company's OS and the company's package manager and the company's monitoring service and whatever else. That's not a great position to be in when you're trying to run a reliable service, so we decided it would be better to accept Facebook's foundational tech stack and hope the benefits outweighed the costs. [1] As such, I usually recuse myself from discussions of WhatsApp and Facebook, I'm certainly not authorized to speak on behalf of either; my opinions are my own, etc.