9 ms·

The Magical Mystery Merge Or Why we run FreeBSD-current at Netflix (2023) [pdf]

247 points by ksec 2y ago

P_I_Staker 2y ago
Magicaly mysteries, their stealing from us! Every single moth, they're practicing racing prices, until I've had it. All. You would think some of that would lead to magically making shows or the prices can be lowened. I seen it happen in my country, and pretty repeaterly. I just don't stop. It should start. Lets get the ball rolling. Tonight.
- aidenn0 2y ago
  I'm guessing only a small fraction of the subscription cost goes to delivering the video. Copyright is a government enforced monopoly on content, and money goes to monopolies. As Thiel said, "Competition is for losers."
ptek 2y ago
Maybe I will keep my Netflix account at least I know it's going to improve FreeBSD. (NetBSD fan here).
ogurechny 2y ago
Great, but I think some slide is missing in the end. Something similar to > Making a diff of some kind of text dumps describing current system configuration before and after would be way faster and easier. > Guys, how do we know our systems are built in reproducible manner at all? Last minute edit: My bad, “configured at startup”, not “built”. Building can already be controlled, obviously, starting up is more dynamic and opaque.
mcoliver 2y ago
Hey @drewg. In hindsight would have looking at the power draw chart combined with the CPU utilization have given you a clue that it was running at a lower clock speed and helped you narrow in on a cause rather than having to go the bisect route? Or a system diff of sysctls?
ksec 2y ago
I posted the Serving Netflix Video Traffic at 800Gb/s and Beyond [1] in 2022. For those who are unaware of the context you may want to read the previous PDF and thread. Now we have some update; quote > Important Performance Milestones: 2022: First 800Gb/s CDN server 2x AMD 7713, NIC kTLS offload 2023: First 100Gb/s CDN server consuming only 100W of power, Nvidia Bluefield-3, NIC kTLS offload My immediate question is if the 2x AMD 7713 actually consumes more than 800W of power. i.e More Watts / Gbps. Even if it does, it is based on 7nm Zen 3 and DDR4 came out in 2021. Would a Zen 5 DDR5 outperforms Bluefield in Watts / Gbps? [1] https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881
- papageek 2y ago
  I haven’t read post yet but I used to work in this space @cable. The nice bit with bluefield3 is you can run nginx on the on-board arm cores which have sufficient ram for live hls usecase. You can use liqid pcie fabric and support 20 bluefield nics off a single cpu 1u box. You essentially turn the 1u box into a mid-tier cache for the nics. Doing this I was able to generate 120 gig off each nic off a 1u hp + the pcie fabric/cards. I worked with liqid and hp lab here in Colorado prototyping it. Edit: I ran the cdn edge directly on nic using yocto linix.
- papageek 2y ago
  Another observation, using the host cpu to manage the nvme storage for vod content is also a bottleneck. You can use the liqid 8x nvme pcie cards and address them directly from the bluefield3 arm processes using nvme-of. You are then just limited by pcie 5 switch capacity among the 10-20 shared nvme/bluefield3 cards.
- vitus 2y ago
  Note that your power consumption is more than just the CPU (combined TDP of 2x225W [0]). You also have to consider SSD (16x20W when fully loaded [1]), NIC (4x24W [2]), and the rest of the actual system itself (e.g. cooling, backplane). [0] https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-7713.html https://www.amd.com/en/products/processors/server/epyc/7003-... [1] I couldn't find 14TB enterprise SSDs on Intel's website, so I'm using the numbers from 6.4TB drives: https://ark.intel.com/content/www/us/en/ark/products/202708/intel-ssd-d7-p5600-series-6-4tb-2-5in-pcie-4-0-x4-3d3-tlc.html https://ark.intel.com/content/www/us/en/ark/products/202708/... [2] I'm not sure offhand which model number to use, but both models that support 200GbE on page 93-96 have this maximum wattage: https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapter-cards-for-ocp-spec-3-0-user-manual.pdf https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapte... Or, you can skip all the hand calculations and just fiddle with Dell's website to put together an order for a rack while trying to mirror the specs as closely as possible (I only included two NICs, since it complained that the configuration didn't have enough low-profile PCIe slots for four): https://www.dell.com/en-us/shop/dell-poweredge-servers/poweredge-r7525-rack-server/spd/poweredge-r7525/pe_r7525_tm_vi_vp_sb?configurationid=d6108dd6-0c9c-4dcb-9c13-5eed63f6510e6 https://www.dell.com/en-us/shop/dell-poweredge-servers/power... In this case, I selected a 1100W power supply and it's still yelling at me that it's not enough; 1400W is enough to make that nag go away.
walteweiss 2y ago
Is it part of some talk? Is it available online?
- phoronixrly 2y ago
  Here it is: https://www.youtube.com/watch?v=q4TZxj-Dq7s https://www.youtube.com/watch?v=q4TZxj-Dq7s
jsnell 2y ago
> Had we moved between -stable branches, bisecting 3+ years of changes could have taken weeks Would it really? Going by their number of 4 hours per bisect step, you get 6 bisections per day, which cuts the range to 1/64th of the original. The difference between "three years of changes" and "three weeks of changes" is a factor of 50x. I.e. within one day, they'd already have identified the problematic three week range. After that, the remaining bisection takes exactly as long as this bisection did. Even if they're limited to doing the bisections just doing the working hours in one timezone for some reason, you'd still get those six bisection steps done in just three days. It still would not add weeks.
- iainmerrick 2y ago
  That's a very good point, but merging in three years of changes would also have pulled in a lot of minor performance changes, and possibly some incompatible changes that would require some code updates. That would slow down each bisection step, and also make it harder to pinpoint the problem. If you know that some small set of recent changes caused a 7% regression, you can be fairly confident there's a single cause. If 3 years of updates cause a 6% or 8% regression (say), it's not obvious that there's going to be a single cause. Even if you find a commit that looks bad, it might already have been addressed in a later commit. Edit to clarify: you're technically correct (the best kind of correct!) but I'd still much prefer to merge 3 weeks rather than 3 years, even though their justification isn't quite right.
- wccrawford 2y ago
  I take your point, but that assumes someone working 24 hour days, or constantly handing off the project to at least 2 other people every day. I don't think those are reasonable work scenarios, so it's more like 2 bisects (maybe 3!) per day, rather than 6.
  - jsnell 2y ago
    Not really, it just assumes that the bisection process is automated. But also, I addressed exactly this objection in the second paragraph :)
hnarayanan 2y ago
Going through these slides was like reading a thriller!
- p0seidon 2y ago
  I agree, but: The best thing about this is that the work is actually done by a small number of people instead of an overengineered system with a custom-solution. The thriller is great, but also the efficiency powering all of Netflix CDNs is refreshing.
gigatexal 2y ago
Have they ever talked about why the content isn't stored/read from ZFS just the root pool?
- iv42 2y ago
  Yes. Because sendfile(2) is not zero-copy on ZFS.
- shrubble 2y ago
  Yes, it is because they can do zero-copy sending of content that they can't do under ZFS (yet). Some links to Netflix papers and video talks on this older Reddit thread: https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_overwhelminly_praised_yet_why_does_netflix/ https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_over...
  - gigatexal 2y ago
    bravo thank you for that
- 8fingerlouie 2y ago
  Besides not being able to do zero copy on ZFS (yet), it probably also have to do with them not using RAID for content, and single drive ZFS doesn't make much sense in that scenario. Single drive ZFS brings snapshots and COW, as well as bitrot detection, but for Netflix OCAs, snapshots are not used, and it's mostly read-only content, so not much use for COW, and bitrot is less of a problem with media. Yes, you may get a corrupt pixel every now and then (assuming it's not caught by the codec), but the media will be reseeded again every n days, so the problem will solve itself. I assume they have ample redundancy in the rest of the CDN, so if a drive fails, they can simply redirect traffic to the next nearest mirror, and when a drive is eventually replaced, it will be seeded again by normal content seeding.
  - _zoltan_ 2y ago
    yet? you write "yet" as it's something that would be almost readily available, yet it's been at least 2 years now and it's still not there. am I missing something or that "yet" is more like "maybe sometime if ever"?
phoronixrly 2y ago
Video from the talk at OpenFest 2023: https://www.youtube.com/watch?v=q4TZxj-Dq7s https://www.youtube.com/watch?v=q4TZxj-Dq7s
eatonphil 2y ago
Competition is good (not everyone using Linux I mean), and I've ran FreeBSD on my desktop and server for a few years. But whenever Netflix's use of FreeBSD comes up I never come away with a concrete understanding for: is the cost/performance Netflix gets with FreeBSD really not doable on Linux? I'm trying to understand if it's inertia, or if not, why more cloud companies with similar traffic (all the CDN companies for example) aren't also using FreeBSD somewhere in their stack. If it were just the case that they like FreeBSD that would be fine and I wouldn't ask more. But they mention in the slides FreeBSD is a performance-focused OS which seems to beg the question.
- Thaxll 2y ago
  The truth is that some senior engineer at Netflix chose to use FreeBSD and they stick to that idea since then, FreeBSD is not better it's happen to be the solution they chose. All the benefits they added to FreeBSD could be added the same way in Linux if it was missing. YouTube / Google CDN is much bigger than Netflix and runs 100% on Linux, you can make pretty much everything work on modern solution / languages / framework.
  - inopinatus 2y ago
    Sorry, this is seriously misinformed: > YouTube / Google CDN is much bigger than Netflix Youtube and Netflix are on par. According to Sandvine, Netflix sneaked past Youtube in volume in 2023[1]. I believe their 2024 report shows them still neck-and-neck. > you can make pretty much everything work on modern solution Presenting a false equivalence without evidence is not convincing. "You could write it in SNOBOL and deploy it on TempleOS". Netflix didn't choose something arbitrary or by mistake. They chose one of the world's highest performing and most robustly tested infrastructure operating systems. It's the reason a FreeBSD derivative lies at the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn Darwin and thence macOS, iOS etc. It continues to surprise me how folks in the tech sector routinely fail to notice how much BSD is deployed in the infrastructure around them. > All the benefits they added to FreeBSD could be added the same way in Linux They cannot. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel. Netflix is touting here the value of FreeBSD's unified release, since the bisect process fundamentally relies on a single dimension of change (please ignore the mathematicians muttering about Schur's transform). [1] https://www.sandvine.com/press-releases/sandvines-2023-global-internet-phenomena-report-shows-24-jump-in-video-traffic-with-netflix-volume-overtaking-youtube https://www.sandvine.com/press-releases/sandvines-2023-globa...
Thaxll 2y ago
I'm still surprised they did not get away from Nginx at that point and did something like Cloudflare.
- alberth 2y ago
  When you’re pushing the amount of data Netflix is, you need to work directly with the ISP & Exchanges. At least that’s my guess.
  - yencabulator 2y ago
    Parent is talking purely about software. https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
- toast0 2y ago
  Do you mean moving away from using Nginx, like Cloudflare moved to a custom replacement? [1] I don't think that's as needed for Netflix. My understanding is their CDN nodes are using Nginx to serve static files --- their content management system directs off peak transfers, and routes clients to nodes that it knows has the files. They don't run a traditional caching (reverse) proxy, and they most likely don't hit a lot of the things Cloudflare was hitting, because their use case is so different. (I haven't seen a discussion of how Netflix handles live events, that's clearly a different process than their traditional prerecorded media) [1] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
  - drewg123 2y ago
    Yes, exactly.
  - Thaxll 2y ago
    Yes I'm talking about Nginx running on those bsd boxes, they have such a custom design that writing your own static file process would have made sense.
ay 2y ago
Sometimes it’s still hard to tackle people psychology who are used to “comfort” of the “sta[b]le” branches. So at work I came up with the following observation: if you are a consumer and are afraid of unpredictable events at head of master/main branch - if you use the head of master/main from 21 days ago, you have 3 weeks of completely predictable and modifiable future. Any cherry picks made are made during the build process, so there is no branch divergence - if the fix gets up streamed it is not cherry picked anymore. Thus, unlike with stable branches, by default it converges back to master. “But what if the fix is not upstreamed ?” - then it gets to be there, and depending on the nature of the code it bears a bigger or smaller ongoing cost - which reflects well the technical debt as it is. This has worked pretty well for the past 4-5 years and is now used for quite a few projects.
- bongodongobob 2y ago
  This is how OS updates have worked at every company I've been at. Either have a handful of devices that get them immediately and you scream test, or you simply just wait 3 weeks, then roll them out. (Minus security CVE's of course)
  - ay 2y ago
    Lucky you! :-) The one additional bonus of this scheme that popped up somewhat as a side effect is one can deploy the new feature into the “prod” build even before it hits the master: code it up, add the instruction to cherry-pick the change into the branch of a builder, test it and if it works then merge that instruction into the “master” builder. Then, the feature will remain in prod builds as a custom cherry-pick until it’s merged upstream. And if it isn’t merged - then one has a recurring reminder of their technical debt that they have incurred by diverging. Once adopted, it became a pretty cool way to adopt “right” long term incentives without door-stopping the short term “urgent” deliverables.
kazinator 2y ago
What are they bisecting with? FreeBSD uses CVS and Perforce. These items in the slide don't add up: > Things had worked accidentally for years, due to linkerset alphabetical ordering In other words, the real bug is many years old. Yet, on the last slide: > Since we found the bug within a week or so of it hitting the tree, the developer responsible was incredibly responsive. All the details were fresh in his memory. What? Wasn't what hit the tree the ordering change which exposed the bug? It doesn't seem like that being fresh in anyone's mind is of much help.
- sakjur 2y ago
  FreeBSD has moved to Git as its primary VCS, see https://lists.freebsd.org/pipermail/freebsd-current/2020-December/077858.html https://lists.freebsd.org/pipermail/freebsd-current/2020-Dec...
- drewg123 2y ago
  FreeBSD hasn't use CVS and P4 in decades. FreeBSD uses git internally, and has a github mirror. See https://docs.freebsd.org/en/articles/committers-guide/ https://docs.freebsd.org/en/articles/committers-guide/ The unintentional ordering change being fresh Colin's memory was really helpful, as he quickly realized that he'd actually changed the ordering and sent me a patch to fix it. If it had been 3 years in the past, I suspect he would have been less responsive (I know I would have been).
- pronoiac 2y ago
  There were two sets of bugs: * the new sort handled ties differently. They adjusted that, and they could have stopped there. * the other was that the correct drivers were sensitive to loading order. They handled this and another driver bug, amdtemp. This was masked by the old sort. If they'd found this years later, even investigating the first set would be a slower process - it wouldn't be fresh in minds, and it likely would then have other code relying on it, so adjusting it would be trickier.
kazinator 2y ago
Why would you alphabetically order initializations? Every complex system I've ever worked on that had a large number of initializations was sensitive to orders. Languages with module support like Wirth's Modula-2 ensure that if module A uses B, B's initialization will execute before A's. If there is no circular dependency, that order will never be wrong. The reverse order could work too, but it's a crapshoot then. Module dependency doesn't logically entail initialization order dependency: A's initializations might not require B's initializations to have completed. If you're initializing by explicit calls in a language that doesn't automate the dependencies, the baseline safest thing to do is to call things in a fixed order that is recorded somewhere in the code: array of function addresses, or just an init procedure that calls others directly. If you sort the init calls, it has to be on some property linked to dependency order, otherwise don't do it. Unless you've encoded something related to dependencies into the module names, lexicographic order is not right. In the firmware application I work on now, all modules have a statically allocated signature word that is initially zero and set to a pattern when the module is initialized. The external API functions all assert that the pattern has the correct value, which is strong evidence that the module had been initialized before use. One one occasion I debugged a static array overrun which trashed these signatures, causing the affected modules to assert.
- toast0 2y ago
  Having a consistent ordering avoids differences in results from inconsistent ordering by construction. IIUC, Alpha sort was/is used as a tie breaker after declared dependencies or other ordering information. In this case, two (or more) modules indicate they can handle the same hardware and didn't have information on priority if both were present. Probably this should be detected / raise a fault, but under the previous regime of alpha sort, it was handled nicely because the preferred drivers happened to sort first.
  - 10000truths 2y ago
    A topological sort of the dependency graph is just as consistent as any other sort, so long as you have a deterministic tie breaking mechanism for the case of multiple valid toposorts (which can just be another sort based on some unique property).
andrewstuart 2y ago
>> Most of us are FreeBSD committers or contributors This is why you shouldn’t copy this strategy.
- arp242 2y ago
  Most of us also aren't running Netflix, or anything of the sort. All the big companies that heavily use Linux at scale also employ tons of Linux kernel engineers.
andrewstuart 2y ago
I love FreeBSD and in a different world really it should occupy the place Linux does. But the reality is it’s a Linux world and now Linux has systemd which makes switching to anything else not an option for me. You’d have to pry systemd from my cold, dead hands.

247 points by ksec 2y ago

Magicaly mysteries, their stealing from us! Every single moth, they're practicing racing prices, until I've had it. All. You would think some of that would lead to magically making shows or the prices can be lowened. I seen it happen in my country, and pretty repeaterly. I just don't stop. It should start. Lets get the ball rolling. Tonight.

aidenn0 2y ago
I'm guessing only a small fraction of the subscription cost goes to delivering the video. Copyright is a government enforced monopoly on content, and money goes to monopolies. As Thiel said, "Competition is for losers."

ptek 2y ago

Maybe I will keep my Netflix account at least I know it's going to improve FreeBSD. (NetBSD fan here).

ogurechny 2y ago

Great, but I think some slide is missing in the end. Something similar to > Making a diff of some kind of text dumps describing current system configuration before and after would be way faster and easier. > Guys, how do we know our systems are built in reproducible manner at all? Last minute edit: My bad, “configured at startup”, not “built”. Building can already be controlled, obviously, starting up is more dynamic and opaque.

mcoliver 2y ago

Hey @drewg. In hindsight would have looking at the power draw chart combined with the CPU utilization have given you a clue that it was running at a lower clock speed and helped you narrow in on a cause rather than having to go the bisect route? Or a system diff of sysctls?

ksec 2y ago

I posted the Serving Netflix Video Traffic at 800Gb/s and Beyond [1] in 2022. For those who are unaware of the context you may want to read the previous PDF and thread. Now we have some update; quote > Important Performance Milestones: 2022: First 800Gb/s CDN server 2x AMD 7713, NIC kTLS offload 2023: First 100Gb/s CDN server consuming only 100W of power, Nvidia Bluefield-3, NIC kTLS offload My immediate question is if the 2x AMD 7713 actually consumes more than 800W of power. i.e More Watts / Gbps. Even if it does, it is based on 7nm Zen 3 and DDR4 came out in 2021. Would a Zen 5 DDR5 outperforms Bluefield in Watts / Gbps? [1] https://news.ycombinator.com/item?id=32519881 https://news.ycombinator.com/item?id=32519881

papageek 2y ago
I haven’t read post yet but I used to work in this space @cable. The nice bit with bluefield3 is you can run nginx on the on-board arm cores which have sufficient ram for live hls usecase. You can use liqid pcie fabric and support 20 bluefield nics off a single cpu 1u box. You essentially turn the 1u box into a mid-tier cache for the nics. Doing this I was able to generate 120 gig off each nic off a 1u hp + the pcie fabric/cards. I worked with liqid and hp lab here in Colorado prototyping it. Edit: I ran the cdn edge directly on nic using yocto linix.
papageek 2y ago
Another observation, using the host cpu to manage the nvme storage for vod content is also a bottleneck. You can use the liqid 8x nvme pcie cards and address them directly from the bluefield3 arm processes using nvme-of. You are then just limited by pcie 5 switch capacity among the 10-20 shared nvme/bluefield3 cards.
vitus 2y ago
Note that your power consumption is more than just the CPU (combined TDP of 2x225W [0]). You also have to consider SSD (16x20W when fully loaded [1]), NIC (4x24W [2]), and the rest of the actual system itself (e.g. cooling, backplane). [0] https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-7713.html https://www.amd.com/en/products/processors/server/epyc/7003-... [1] I couldn't find 14TB enterprise SSDs on Intel's website, so I'm using the numbers from 6.4TB drives: https://ark.intel.com/content/www/us/en/ark/products/202708/intel-ssd-d7-p5600-series-6-4tb-2-5in-pcie-4-0-x4-3d3-tlc.html https://ark.intel.com/content/www/us/en/ark/products/202708/... [2] I'm not sure offhand which model number to use, but both models that support 200GbE on page 93-96 have this maximum wattage: https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapter-cards-for-ocp-spec-3-0-user-manual.pdf https://docs.nvidia.com/nvidia-connectx-6-dx-ethernet-adapte... Or, you can skip all the hand calculations and just fiddle with Dell's website to put together an order for a rack while trying to mirror the specs as closely as possible (I only included two NICs, since it complained that the configuration didn't have enough low-profile PCIe slots for four): https://www.dell.com/en-us/shop/dell-poweredge-servers/poweredge-r7525-rack-server/spd/poweredge-r7525/pe_r7525_tm_vi_vp_sb?configurationid=d6108dd6-0c9c-4dcb-9c13-5eed63f6510e6 https://www.dell.com/en-us/shop/dell-poweredge-servers/power... In this case, I selected a 1100W power supply and it's still yelling at me that it's not enough; 1400W is enough to make that nag go away.

walteweiss 2y ago

Is it part of some talk? Is it available online?

phoronixrly 2y ago
Here it is: https://www.youtube.com/watch?v=q4TZxj-Dq7s https://www.youtube.com/watch?v=q4TZxj-Dq7s

jsnell 2y ago

> Had we moved between -stable branches, bisecting 3+ years of changes could have taken weeks Would it really? Going by their number of 4 hours per bisect step, you get 6 bisections per day, which cuts the range to 1/64th of the original. The difference between "three years of changes" and "three weeks of changes" is a factor of 50x. I.e. within one day, they'd already have identified the problematic three week range. After that, the remaining bisection takes exactly as long as this bisection did. Even if they're limited to doing the bisections just doing the working hours in one timezone for some reason, you'd still get those six bisection steps done in just three days. It still would not add weeks.

iainmerrick 2y ago
That's a very good point, but merging in three years of changes would also have pulled in a lot of minor performance changes, and possibly some incompatible changes that would require some code updates. That would slow down each bisection step, and also make it harder to pinpoint the problem. If you know that some small set of recent changes caused a 7% regression, you can be fairly confident there's a single cause. If 3 years of updates cause a 6% or 8% regression (say), it's not obvious that there's going to be a single cause. Even if you find a commit that looks bad, it might already have been addressed in a later commit. Edit to clarify: you're technically correct (the best kind of correct!) but I'd still much prefer to merge 3 weeks rather than 3 years, even though their justification isn't quite right.
wccrawford 2y ago
I take your point, but that assumes someone working 24 hour days, or constantly handing off the project to at least 2 other people every day. I don't think those are reasonable work scenarios, so it's more like 2 bisects (maybe 3!) per day, rather than 6.
- jsnell 2y ago
  Not really, it just assumes that the bisection process is automated. But also, I addressed exactly this objection in the second paragraph :)

hnarayanan 2y ago

Going through these slides was like reading a thriller!

p0seidon 2y ago
I agree, but: The best thing about this is that the work is actually done by a small number of people instead of an overengineered system with a custom-solution. The thriller is great, but also the efficiency powering all of Netflix CDNs is refreshing.

gigatexal 2y ago

Have they ever talked about why the content isn't stored/read from ZFS just the root pool?

iv42 2y ago
Yes. Because sendfile(2) is not zero-copy on ZFS.
shrubble 2y ago
Yes, it is because they can do zero-copy sending of content that they can't do under ZFS (yet). Some links to Netflix papers and video talks on this older Reddit thread: https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_overwhelminly_praised_yet_why_does_netflix/ https://www.reddit.com/r/freebsd/comments/ltjv8m/zfs_is_over...
- gigatexal 2y ago
  bravo thank you for that
8fingerlouie 2y ago
Besides not being able to do zero copy on ZFS (yet), it probably also have to do with them not using RAID for content, and single drive ZFS doesn't make much sense in that scenario. Single drive ZFS brings snapshots and COW, as well as bitrot detection, but for Netflix OCAs, snapshots are not used, and it's mostly read-only content, so not much use for COW, and bitrot is less of a problem with media. Yes, you may get a corrupt pixel every now and then (assuming it's not caught by the codec), but the media will be reseeded again every n days, so the problem will solve itself. I assume they have ample redundancy in the rest of the CDN, so if a drive fails, they can simply redirect traffic to the next nearest mirror, and when a drive is eventually replaced, it will be seeded again by normal content seeding.
- _zoltan_ 2y ago
  yet? you write "yet" as it's something that would be almost readily available, yet it's been at least 2 years now and it's still not there. am I missing something or that "yet" is more like "maybe sometime if ever"?

phoronixrly 2y ago

Video from the talk at OpenFest 2023: https://www.youtube.com/watch?v=q4TZxj-Dq7s https://www.youtube.com/watch?v=q4TZxj-Dq7s

eatonphil 2y ago

Competition is good (not everyone using Linux I mean), and I've ran FreeBSD on my desktop and server for a few years. But whenever Netflix's use of FreeBSD comes up I never come away with a concrete understanding for: is the cost/performance Netflix gets with FreeBSD really not doable on Linux? I'm trying to understand if it's inertia, or if not, why more cloud companies with similar traffic (all the CDN companies for example) aren't also using FreeBSD somewhere in their stack. If it were just the case that they like FreeBSD that would be fine and I wouldn't ask more. But they mention in the slides FreeBSD is a performance-focused OS which seems to beg the question.

Thaxll 2y ago
The truth is that some senior engineer at Netflix chose to use FreeBSD and they stick to that idea since then, FreeBSD is not better it's happen to be the solution they chose. All the benefits they added to FreeBSD could be added the same way in Linux if it was missing. YouTube / Google CDN is much bigger than Netflix and runs 100% on Linux, you can make pretty much everything work on modern solution / languages / framework.
- inopinatus 2y ago
  Sorry, this is seriously misinformed: > YouTube / Google CDN is much bigger than Netflix Youtube and Netflix are on par. According to Sandvine, Netflix sneaked past Youtube in volume in 2023[1]. I believe their 2024 report shows them still neck-and-neck. > you can make pretty much everything work on modern solution Presenting a false equivalence without evidence is not convincing. "You could write it in SNOBOL and deploy it on TempleOS". Netflix didn't choose something arbitrary or by mistake. They chose one of the world's highest performing and most robustly tested infrastructure operating systems. It's the reason a FreeBSD derivative lies at the core of Juniper routers, Isilon and Netapp storage heads, every Playstation 3/4/5, and got mashed into NeXTSTEP to spawn Darwin and thence macOS, iOS etc. It continues to surprise me how folks in the tech sector routinely fail to notice how much BSD is deployed in the infrastructure around them. > All the benefits they added to FreeBSD could be added the same way in Linux They cannot. Case in point, the bisect analysis described in the presentation above doesn't exist for Linux, where userland distributions develop independently from the kernel. Netflix is touting here the value of FreeBSD's unified release, since the bisect process fundamentally relies on a single dimension of change (please ignore the mathematicians muttering about Schur's transform). [1] https://www.sandvine.com/press-releases/sandvines-2023-global-internet-phenomena-report-shows-24-jump-in-video-traffic-with-netflix-volume-overtaking-youtube https://www.sandvine.com/press-releases/sandvines-2023-globa...

Thaxll 2y ago

I'm still surprised they did not get away from Nginx at that point and did something like Cloudflare.

alberth 2y ago
When you’re pushing the amount of data Netflix is, you need to work directly with the ISP & Exchanges. At least that’s my guess.
- yencabulator 2y ago
  Parent is talking purely about software. https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
toast0 2y ago
Do you mean moving away from using Nginx, like Cloudflare moved to a custom replacement? [1] I don't think that's as needed for Netflix. My understanding is their CDN nodes are using Nginx to serve static files --- their content management system directs off peak transfers, and routes clients to nodes that it knows has the files. They don't run a traditional caching (reverse) proxy, and they most likely don't hit a lot of the things Cloudflare was hitting, because their use case is so different. (I haven't seen a discussion of how Netflix handles live events, that's clearly a different process than their traditional prerecorded media) [1] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
- drewg123 2y ago
  Yes, exactly.
- Thaxll 2y ago
  Yes I'm talking about Nginx running on those bsd boxes, they have such a custom design that writing your own static file process would have made sense.

ay 2y ago

Sometimes it’s still hard to tackle people psychology who are used to “comfort” of the “sta[b]le” branches. So at work I came up with the following observation: if you are a consumer and are afraid of unpredictable events at head of master/main branch - if you use the head of master/main from 21 days ago, you have 3 weeks of completely predictable and modifiable future. Any cherry picks made are made during the build process, so there is no branch divergence - if the fix gets up streamed it is not cherry picked anymore. Thus, unlike with stable branches, by default it converges back to master. “But what if the fix is not upstreamed ?” - then it gets to be there, and depending on the nature of the code it bears a bigger or smaller ongoing cost - which reflects well the technical debt as it is. This has worked pretty well for the past 4-5 years and is now used for quite a few projects.

bongodongobob 2y ago
This is how OS updates have worked at every company I've been at. Either have a handful of devices that get them immediately and you scream test, or you simply just wait 3 weeks, then roll them out. (Minus security CVE's of course)
- ay 2y ago
  Lucky you! :-) The one additional bonus of this scheme that popped up somewhat as a side effect is one can deploy the new feature into the “prod” build even before it hits the master: code it up, add the instruction to cherry-pick the change into the branch of a builder, test it and if it works then merge that instruction into the “master” builder. Then, the feature will remain in prod builds as a custom cherry-pick until it’s merged upstream. And if it isn’t merged - then one has a recurring reminder of their technical debt that they have incurred by diverging. Once adopted, it became a pretty cool way to adopt “right” long term incentives without door-stopping the short term “urgent” deliverables.

kazinator 2y ago

What are they bisecting with? FreeBSD uses CVS and Perforce. These items in the slide don't add up: > Things had worked accidentally for years, due to linkerset alphabetical ordering In other words, the real bug is many years old. Yet, on the last slide: > Since we found the bug within a week or so of it hitting the tree, the developer responsible was incredibly responsive. All the details were fresh in his memory. What? Wasn't what hit the tree the ordering change which exposed the bug? It doesn't seem like that being fresh in anyone's mind is of much help.

sakjur 2y ago
FreeBSD has moved to Git as its primary VCS, see https://lists.freebsd.org/pipermail/freebsd-current/2020-December/077858.html https://lists.freebsd.org/pipermail/freebsd-current/2020-Dec...
drewg123 2y ago
FreeBSD hasn't use CVS and P4 in decades. FreeBSD uses git internally, and has a github mirror. See https://docs.freebsd.org/en/articles/committers-guide/ https://docs.freebsd.org/en/articles/committers-guide/ The unintentional ordering change being fresh Colin's memory was really helpful, as he quickly realized that he'd actually changed the ordering and sent me a patch to fix it. If it had been 3 years in the past, I suspect he would have been less responsive (I know I would have been).
pronoiac 2y ago
There were two sets of bugs: * the new sort handled ties differently. They adjusted that, and they could have stopped there. * the other was that the correct drivers were sensitive to loading order. They handled this and another driver bug, amdtemp. This was masked by the old sort. If they'd found this years later, even investigating the first set would be a slower process - it wouldn't be fresh in minds, and it likely would then have other code relying on it, so adjusting it would be trickier.

kazinator 2y ago

Why would you alphabetically order initializations? Every complex system I've ever worked on that had a large number of initializations was sensitive to orders. Languages with module support like Wirth's Modula-2 ensure that if module A uses B, B's initialization will execute before A's. If there is no circular dependency, that order will never be wrong. The reverse order could work too, but it's a crapshoot then. Module dependency doesn't logically entail initialization order dependency: A's initializations might not require B's initializations to have completed. If you're initializing by explicit calls in a language that doesn't automate the dependencies, the baseline safest thing to do is to call things in a fixed order that is recorded somewhere in the code: array of function addresses, or just an init procedure that calls others directly. If you sort the init calls, it has to be on some property linked to dependency order, otherwise don't do it. Unless you've encoded something related to dependencies into the module names, lexicographic order is not right. In the firmware application I work on now, all modules have a statically allocated signature word that is initially zero and set to a pattern when the module is initialized. The external API functions all assert that the pattern has the correct value, which is strong evidence that the module had been initialized before use. One one occasion I debugged a static array overrun which trashed these signatures, causing the affected modules to assert.

toast0 2y ago
Having a consistent ordering avoids differences in results from inconsistent ordering by construction. IIUC, Alpha sort was/is used as a tie breaker after declared dependencies or other ordering information. In this case, two (or more) modules indicate they can handle the same hardware and didn't have information on priority if both were present. Probably this should be detected / raise a fault, but under the previous regime of alpha sort, it was handled nicely because the preferred drivers happened to sort first.
- 10000truths 2y ago
  A topological sort of the dependency graph is just as consistent as any other sort, so long as you have a deterministic tie breaking mechanism for the case of multiple valid toposorts (which can just be another sort based on some unique property).

andrewstuart 2y ago

>> Most of us are FreeBSD committers or contributors This is why you shouldn’t copy this strategy.

arp242 2y ago
Most of us also aren't running Netflix, or anything of the sort. All the big companies that heavily use Linux at scale also employ tons of Linux kernel engineers.

andrewstuart 2y ago

I love FreeBSD and in a different world really it should occupy the place Linux does. But the reality is it’s a Linux world and now Linux has systemd which makes switching to anything else not an option for me. You’d have to pry systemd from my cold, dead hands.