13 ms·
Facebook open-sources new suite of Linux kernel components and tools
- amaccuish 8y agoI didn't quite get this post; half these things are stuff that facebook uses, not that they themselves have created and open-sourced (e.g. BTRFS initially from Oracle)? And yet they've set up they're own site for BTRFS and all (https://facebookmicrosites.github.io/btrfs/ https://facebookmicrosites.github.io/btrfs/)
- traek 8y agoFacebook hired the principal developer of BRTFS, Chris Mason, to work on it full-time. Edit: it looks like a lot of the projects have already been open-sourced but were developed at Facebook (e.g. PSI and cgroups2), and Facebook is now announcing them all together along with some new tools.
- akhilcacharya 8y agoThey hired a bunch of people from Fusion-IO to work on it. https://en.wikipedia.org/wiki/Fusion-io https://en.wikipedia.org/wiki/Fusion-io I met Josef at a FB recruiting event at my school.
- moneil971 8y agoOomd was open sourced before this announcement: https://code.fb.com/production-engineering/oomd/ https://code.fb.com/production-engineering/oomd/ But seems like Facebook is grouping them to show how they can work together.
- h1d 8y agoInteresting they continue to develop btrfs when RedHat severed its tie which sounded like the end of the deal for btrfs when no one but suse uses it as the default file system. But even looking how it never gained mainstream adoption for over 10 years when people always wanted something like zfs, it just feels there's something substantially wrong about its development model. https://news.ycombinator.com/item?id=14907771 https://news.ycombinator.com/item?id=14907771
- chronid 8y agoI feel it's most bad experiences when it was started at first - and on enterprise systems it never really got adoption because of bad RAID. I still remember some bugs with inodes that could not be deleted making me mad and forcing me to format my laptop with the more conventional XFS :) But I guess if (big if?) FB uses it in production (without a lot of internal patches) it may be kinda stable now. Still, I don't see it as being useful for a lot of people nowadays, particularly if you sit on an "expendable" VM in a cloud.
- pfranz 8y agoSynology has rolled out btrfs in par with ext4 a few years ago for models that support it. Changing to a new filesystem is scary so I'm not surprised it has taken this long to reach mainstream. I'm glad there's still a big player like FB supporting it.
- nouseforaname 8y agoI like the fact that there's a little competition. Personally, I tried btrfs as a root filesystem for a couple years. It definitely had weird gotchas like running out of inode space pretty frequently when making use of snapshots. A few years back I also ran into a problem with a bunch of zero length files (race between writing metadata and content?) I switched this year and I found zfs to much more user friendly and reliable, although setting up zfs on a root on ubuntu is still a manual process.
- rachelbythebay 8y agoENOSPC latching. Not just you. It never ends.
- jensv 8y agoFacebook can "show off" and develop a reputation as having interesting, technically innovative, meaty, challenging projects to work on which is useful for recruitment purposes.
- sigi45 8y agoAnd still it's Facebook :D
- romed 8y agoYeah it's really weird. Same for cgroup2 and all this other stuff. This is all established open software.
- dward 8y agoOomd reminds me of userspace OOM handling mechanism proposed by David Reintjes of Google: https://lwn.net/Articles/590960/ https://lwn.net/Articles/590960/
- sofaofthedamned 8y agoDoes this fix the cgroup memory reporting issue? It's caused me a lot of pain with containers containing crap code that assumes the memory is all available.
- macdice 8y agoSee also the wonderfully named SIGDANGER signal on AIX. It seems many people have invented this. I've been wondering out loud recently if PostgreSQL should attempt to support these various impending-doom-notification mechanisms, or whether a system that close to the edge really needs human intervention anyway.
- puzzle 8y agoThe Amiga had a feature in the Exec kernel since version 3.0 that let applications register so-called memory handlers, which the system could call to purge e.g. thumbnail caches owned by the userland process, etc. Of course the Amiga had a single address space and no memory protection or paging, so things were easier.
- egorfine 8y agoiOS has a mechanism like that. The application receives notification on memory pressure and can release some memory in order to not being killed.
- reacharavindh 8y agoI don't understand the point of this... if they are open source Linux Kernel contributions, shouldn't they be upstreamed into the Linux kernel? Is Facebook just throwing out stuff from their private branch of the Linux kernel in hopes that somebody has free time to upstream the changes? Not being snarky, I genuinely don't understand the point of this...
- ISL 8y agoIt is a whole lot easier to peruse and use modifications that someone else has written and debugged than to write them oneself. It is generally a cause for celebration when someone releases their work for everyone else to consider for use.
- pweissbrod 8y agoThe philosophy of the kernel is to work for everyone's need. Its a good question, if these enhancements are of broad benefit and unobtrusive as they seem it's likely better to upstreaming it vs out-of-band patches.
- czardoz 8y agoSome of these are not actually things that should be in the main kernel, which is also why they are being referred to as "components". E.g: Not every laptop user needs to be able to run BPF.
- monocasa 8y ago> Not every laptop user needs to be able to run BPF. True for the moment, but I can see that changing. Having a safe VM at the system call layer is a game changer for so many subsystems. It turns the Linux kernel into a hybrid exokernel.
- DelightOne 8y agoHow would other subsystems use the safety of the BPF program - which I assume is the safe VM you mention? Do they have to assume that the network stack may not halt and have special code for it or sth ?
- deleted 8y ago[deleted]
- ilovecaching 8y agoXDP is the future of high speed networking in the Linux Kernel. What's amazing about XDP is how accessible it is compared to user space kernel bypass. Literally anyone can write an eBPF program, and it even sanity checks it for you! Very excited to see all the amazing work FB puts into eBPF. I've used BCC extensively and it's amazing the granularity you have over resource consumption with eBPF. Also it's really cool how much FB has put into butterFS and cgroups. They're doing very foundational work for the container space, which is very cool.
- zzzzzzzza 8y agoAs a (semi casual) linux user but "kernel outsider" anyone want to break down all the acronyms used here?
- ilovecaching 8y agoFB = Facebook :) XDP = eXpress Data Path, it's an eBPF program that runs before the kernel network stack and allows you to process raw packets as fast as your network card will allow. eBPF = extended Berkley Packet Filter is a BPF program that is compiled and run on a virtual machine in the kernel (allows you to run kernel code from userspace). It can talk to user space through maps and hook in to various parts of the kernel. An important point is that once compiled, an eBPF program is guaranteed to halt and has other verification performed on it, to make it safe to run in the kernel. BCC = BPF Compiler Collection, is a set of tools for working with eBPF, it uses llvm and clang to make it easy to write eBPF programs in Python, Rust, etc. butterFS = btrfs, the filesystem. People often call it butterFS in conversation, even though the btr stands for b-tree. cgroups = Technically the work is on cgroup2, which changes the way processes are laid out in a resource hierarchy from the original cgroups. This is how resource constraints are placed on containers, although many people use it just to monitor processes (without constraining resources).
- jtbayly 8y agoOk, so what does eBGP stand for? (By the way, thanks for the other definitions!)
- ecesena 8y ago> BPF is a highly flexible, efficient code execution engine in the Linux kernel that allows bytecode to run at various hook points, enabling safe and easy modifications of kernel behaviors with custom code. Can someone explain the word "safe" in that paragraph? Genuinely curious, not trying to start a war flame. I'm interested in understanding use cases for BPF.
- ianlevesque 8y agoIt's guaranteed memory safe and halting.
- forgottenpass 8y agoCertain limitations must be checked (e.g. can be proven to halt). It's a bit much for an HN post, see https://lwn.net/Articles/740157/ https://lwn.net/Articles/740157/ subsection "The eBPF in-kernel verifier" for more background.
- ecesena 8y agoThanks for the pointer.
- leshow 8y agoIs btrfs a good choice for a format for a regular Linux install on a dev or user machine now? I've been using ext4 for almost a decade, curious to hear responses.
- stingraycharles 8y agoFor the past few months, I've been trying to make it work (mostly because of the raid + snapshotting functionality), but I have ran into nothing but trouble. Frequent hangs of my system (unrelated to the defrag process that runs daily, which also uses 100% CPU for minutes -- the best I have deduced it towards is either because of using raid+nvme, or because of high (>100GB) RAM)), Docker is very unstable using btrfs [2], etc etc. I moved back to ext4 + mdraid just last week and couldn't be happier. 1 https://serverfault.com/questions/747366/btrfs-write-operations-hang-when-appending-to-files https://serverfault.com/questions/747366/btrfs-write-operati... 2 https://github.com/moby/moby/issues/34501 https://github.com/moby/moby/issues/34501
- jplayer01 8y agoSeems like these are issues mainly related to using older versions of the kernel and/or Docker.
- stingraycharles 8y agoThis is not the case. I've done a lot of trial & error, and tried Ubuntu 18.04 / 18.10, Mint and Arch. The problems persist everywhere. Best to my knowledge, it started when I upgraded from 32GB RAM to 128GB RAM, and/or started using virtual machines more intensively after that. I tried tuning btrfs in various ways, disabling CoW, enabling/disabling autodefrag. The problems start to re-appear and intensify after about one week of using a fresh install. All I know it's definitely not something simple like "use the latest kernel".
- nicolaslem 8y agoEvery year I consider spending some time to get familiar with btrfs and every year I read horror stories like this one and I stop right there.
- shmerl 8y agoBTRFS is good. Surprisingly it's the only common upstream Linux filesystem with transparent compression.
- jolmg 8y agoDo you think the transparent compression is worth it? I imagine most of the data people have in their computers is already compressed (images, audios, videos, etc.), so it would only slow down reading and writing for most files. I can only see it making sense if much of your disk space is used up by uncompressed text files (or other types of uncompressed filetypes).
- shmerl 8y agoIt saves I/O on HDD, which is useful if you have a good multi core CPU. Some things are compressed but not all. BTRFS has some heuristic which switches off compression if it's not saving space. It's not very elaborate now, but they plan to improve it. See https://btrfs.wiki.kernel.org/index.php/Compression#What_happens_to_incompressible_files.3F https://btrfs.wiki.kernel.org/index.php/Compression#What_hap... I'm using zstd compression for my btrfs volumes.
- cmurf 8y agoUsing compress=zstd on system root saves me about 35-40%. And I save about 1/2 the space if I use compress-force=zstd mount option. I do this on computers with SDHC/SDXC cards which have dog slow writes and lots of spare CPU time, and some of the time on HDD. I don't use compression on computers with NVMe, it doesn't appear to help much at all, and also I'm pretty sure the compression is currently a single threaded pipe, so for multi-queue devices it's plausible some workloads could become slower. Another use case, soonish I hope to see support for selectable compression levels for zstd (currently only with zlib). And also zstd supports use of a training file for even higher ratios. Both would work well for archiving and seed/sprout use cases, where taking the heavy front end hit with slow writes will be worth it (faster download times, and of course the sprout can have a different compression level or no compression so subsequent writes can be fast. Latest squashfs support includes zstd and also selectable levels, if you need a read-only image.
- iamgopal 8y agoThe amount of engineering FB us doing, I wish they would make something important that social networks.
- deleted 8y ago[deleted]
- platform 8y agoWould be interesting to learn, if at all possible, how these kinds of contributions are justified to upper management/financial folks. For example, how (and if) they estimated the cost of this new load balancing technology in terms of money saved/user experience/security. BTW, if I am a corporation betting my sales on F5 load balancers, I would be worried (and would probably notify my shareholders of the risk).
- wpietri 8y agoI'd love to see the full breakdown, but I think hints of it are there in the text. E.g., for the load balancing they say, it "helped improve the performance and scalability of network load balancing while drastically reducing inefficiencies." At Facebook's scale, those are pretty easy to translate into dollars by, e.g., looking at how many machines you now didn't have to buy to keep up with growth.
- tjkrusinski 8y agoRelevant: https://code.fb.com/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ https://code.fb.com/production-engineering/introducing-data-...
- Jnr 8y agoI haven't checked what they have released so I could be wrong but my guess is that by open sourcing their tools they might hope to merge it into Linux kernel so that they would not have to maintain their kernel patches. Additionally they could get improvements from others.
- mikekchar 8y agoI'm not quite sure if the OP meant justifying open sourcing their tools or writing them in the first place. However, for open sourcing, I'm pretty sure I could justify that pretty easily just on reducing the cost of recruitment. There was a time that Facebook had (at least in my circles) a terrible reputation as an employer. Working at Facebook would be a laughable job. Now it's one of the most sought after jobs in the industry. Not only that, but people regularly spend time getting familiar with their in house tools before they even send in a resume. For me, it's basically a no-brainer.