6 ms·

> It's a bit early for me to say which is better nowadays on Linux, ZFS or btrfs, but my company is certainly learning the answer by running the same production

> It's a bit early for me to say which is better nowadays on Linux, ZFS or btrfs, but my company is certainly learning the answer by running the same production workload on both. I suspect we'll share findings in a later blog post. I am eager to read this piece! Even though I am afraid to see it confirm that btrfs still struggles to catch up… The 2016 bcachefs benchmarks[0] are a mixed bag. [0]: https://evilpiepirate.org/~kent/benchmark-full-results-2016-04-19/terse https://evilpiepirate.org/~kent/benchmark-full-results-2016-...

OllyMason 9y ago
Also eager to read this piece
jsiepkes 9y ago
Is BTRFS really an option now that RedHat has decided to pull the plug on their BTRFS development? That basically leaves Oracle and Suse I think? As far as I can tell the future of BTRFS doesn't look good. Facebook using it doesn't mean anything since they are probably using it for distributed applications. Meaning the entire box (including BTRFS) can just die and the cluster won't be impacted. I really can't imagine they are using BTRFS on every node in their cluster.
- geofft 9y ago
  > Facebook using it doesn't mean anything since they are probably using it for distributed applications. Meaning the entire box (including BTRFS) can just die and the cluster won't be impacted. I don't think that follows for lots of reasons: - If enough of your boxes die that you lose quorum (whether from filesystem instability or from unrelated causes like hardware glitches), your cluster is impacted. So, at the least, if you expect your boxes to die at an abnormally high rate, you have to have an abnormally high number of them to maintain service. - Filesystem instability is (I think) much less random than hardware glitches. If a workload causes your filesystem to crash on one machine, recovering and retrying it on the next machine will probably also make it crash. So you may not even be able to save your service by throwing more nodes at the problem. A bad filesystem will probably actually break your service. - Crashes cause a performance impact, because you have to replay the request and you have fewer machines in the cluster until your crashed node reboots. It would take an extraordinarily fast filesystem to be a net performance win if it's even somewhat crashy. - Most importantly, distributed systems generally only help you if you get clean crashes, as in power failure, network disconnects, etc. If you have silent data corruption, or some amount of data corruption leading up to a crash later, or a filesystem that can't fsck properly, your average distributed system is going to deal very poorly. See Ganesan et al., "Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to Single Errors and Corruptions", https://www.usenix.org/system/files/conference/fast17/fast17-ganesan.pdf https://www.usenix.org/system/files/conference/fast17/fast17... So it's very doubtful that Facebook has decided that it's okay that btrfs is crashy because they're running it in distributed systems only.
  - CrystalGamma 9y ago
    Given how many times btrfs has failed to read data or to mount (with an error), I would imagine this is why btrfs is used by Facebook: because it isn't afraid to 'just let it crash' (cleanly), to use Erlang rethoric.
    - geofft 9y ago
      Yeah, it's definitely true that you want a filesystem with data and metadata checksums if you want high reliability. (I think btrfs and ZFS are the only Linux-or-other-UNIX filesystems with data checksums?) But I think the inference to make is that Facebook trusts btrfs to increase reliability, not that Facebook trusts their distributed systems to cover for btrfs decreasing reliability to gain performance (or features).
  - jsiepkes 9y ago
    This article https://www.linux.com/news/learn/intro-to-linux/how-facebook-uses-linux-and-btrfs-interview-chris-mason https://www.linux.com/news/learn/intro-to-linux/how-facebook... explains somewhat what Facebook does with BTRFS. "Mason: The easiest way to describe the infrastructure at Facebook is that it's pretty much all Linux. The places we're targeting for Btrfs are really management tasks around distributing the operating system, distributing updates quickly using the snapshotting features of Btrfs, using the checksumming features of Btrfs and so on. We also have a number of machines running Gluster, using both XFS and Btrfs. The target there is primary data storage. One of the reasons why they like Btrfs for the Gluster use case is because the data CRCs (cyclic redundancy checks) and the metadata CRCs give us the ability to detect problems in the hardware such as silent data corruption in the hardware. We have actually found a few major hardware bugs with Btrfs so it’s been very beneficial to Btrfs." The sentence: "We also have a number of machines running Gluster, using both XFS and Btrfs." seems to imply Facebook is not using it heavily for actual data storage. What I distill from this (which is obviously my personal interpretation) is that Facebook mostly uses it for the OS and not for actual precious data.
    - geofft 9y ago
      I'm reading that as quite the opposite: they're saying that Gluster, a networked file storage system, is being backed with btrfs as the local filesystem, so all data stored in Gluster is ultimately stored on btrfs volumes. (They're also using it for OS snapshotting, yes, but insofar as the data stored in Gluster is important, they're storing important data on btrfs.) See also https://code.facebook.com/posts/938078729581886/improving-the-linux-kernel-with-upstream-contributions/ https://code.facebook.com/posts/938078729581886/improving-th... "We have been working toward deploying Btrfs slowly throughout the fleet, and we have been using large gluster storage clusters to help stabilize Btrfs. The gluster workloads are extremely demanding, and this half we gained a lot more confidence running Btrfs in production. More than 50 changes went into the stabilization effort, and Btrfs was able to protect production data from hardware bugs other filesystems would have missed."
- vbernat 9y ago
  Redhat was never a big contributor to BTRFS. This still means less users, but not less devs.
  - acdha 9y ago
    For a filesystem I'd worry about what that implies for testing, especially for enterprise hardware and workloads. RHEL has significantly more users and it seems likely that their users would have more diversity than just Oracle shops.
    - kuschku 9y ago
      Are you completely forgetting SUSE? They’re a thing, too, after all.
      - acdha 9y ago
        Not forgetting them or Oracle but RHEL has something like twice the marketshare of the two combined. I'm not sure I've ever seen a support matrix where either of them is listed but RHEL is not, whereas the reverse is not uncommon.
- josefbacik 9y ago
  Facebook is using it for lots of shit. Chroots for containers because we can easily snapshot them and use them. Use it for build testing, so snapshot base repo, checkout commit, build, throw away. Gluster which takes huge backups and then other random workloads, it's used in a few places. Just because Facebook is fault tolerant doesn't mean we don't care about failures. We actively run down any issues we hit, so while it doesn't have much of impact in the short term, we don't just ignore issues. And also Red Hat hasn't contributed much to btrfs in years, and Oracle has one developer working on it. We are constantly working on it, and now that my priorities have shifted back to btrfs I hope that we will start to close out some of these long term projects.
  - user5994461 9y ago
    Just because Facebook is working on btrfs in their OS, doesn't mean they will publish their code and patches to the public. Just because some code is added to the main kernel, assuming it is, doesn't mean it will be propagated to the main LTS distributions anytime soon.
    - josefbacik 9y ago
      Lol what? We don't ship code that isn't upstream. It sucks to maintain and rebase if you have a shitton of proprietary patches. Everything we do with the kernel is open source, you wouldn't be able to hire kernel developers if you didn't do open source. Nice comment tho, I legitimately lol'ed.

OllyMason 9y ago

Also eager to read this piece

jsiepkes 9y ago

Is BTRFS really an option now that RedHat has decided to pull the plug on their BTRFS development? That basically leaves Oracle and Suse I think? As far as I can tell the future of BTRFS doesn't look good. Facebook using it doesn't mean anything since they are probably using it for distributed applications. Meaning the entire box (including BTRFS) can just die and the cluster won't be impacted. I really can't imagine they are using BTRFS on every node in their cluster.

geofft 9y ago
> Facebook using it doesn't mean anything since they are probably using it for distributed applications. Meaning the entire box (including BTRFS) can just die and the cluster won't be impacted. I don't think that follows for lots of reasons: - If enough of your boxes die that you lose quorum (whether from filesystem instability or from unrelated causes like hardware glitches), your cluster is impacted. So, at the least, if you expect your boxes to die at an abnormally high rate, you have to have an abnormally high number of them to maintain service. - Filesystem instability is (I think) much less random than hardware glitches. If a workload causes your filesystem to crash on one machine, recovering and retrying it on the next machine will probably also make it crash. So you may not even be able to save your service by throwing more nodes at the problem. A bad filesystem will probably actually break your service. - Crashes cause a performance impact, because you have to replay the request and you have fewer machines in the cluster until your crashed node reboots. It would take an extraordinarily fast filesystem to be a net performance win if it's even somewhat crashy. - Most importantly, distributed systems generally only help you if you get clean crashes, as in power failure, network disconnects, etc. If you have silent data corruption, or some amount of data corruption leading up to a crash later, or a filesystem that can't fsck properly, your average distributed system is going to deal very poorly. See Ganesan et al., "Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to Single Errors and Corruptions", https://www.usenix.org/system/files/conference/fast17/fast17-ganesan.pdf https://www.usenix.org/system/files/conference/fast17/fast17... So it's very doubtful that Facebook has decided that it's okay that btrfs is crashy because they're running it in distributed systems only.
- CrystalGamma 9y ago
  Given how many times btrfs has failed to read data or to mount (with an error), I would imagine this is why btrfs is used by Facebook: because it isn't afraid to 'just let it crash' (cleanly), to use Erlang rethoric.
  - geofft 9y ago
    Yeah, it's definitely true that you want a filesystem with data and metadata checksums if you want high reliability. (I think btrfs and ZFS are the only Linux-or-other-UNIX filesystems with data checksums?) But I think the inference to make is that Facebook trusts btrfs to increase reliability, not that Facebook trusts their distributed systems to cover for btrfs decreasing reliability to gain performance (or features).
- jsiepkes 9y ago
  This article https://www.linux.com/news/learn/intro-to-linux/how-facebook-uses-linux-and-btrfs-interview-chris-mason https://www.linux.com/news/learn/intro-to-linux/how-facebook... explains somewhat what Facebook does with BTRFS. "Mason: The easiest way to describe the infrastructure at Facebook is that it's pretty much all Linux. The places we're targeting for Btrfs are really management tasks around distributing the operating system, distributing updates quickly using the snapshotting features of Btrfs, using the checksumming features of Btrfs and so on. We also have a number of machines running Gluster, using both XFS and Btrfs. The target there is primary data storage. One of the reasons why they like Btrfs for the Gluster use case is because the data CRCs (cyclic redundancy checks) and the metadata CRCs give us the ability to detect problems in the hardware such as silent data corruption in the hardware. We have actually found a few major hardware bugs with Btrfs so it’s been very beneficial to Btrfs." The sentence: "We also have a number of machines running Gluster, using both XFS and Btrfs." seems to imply Facebook is not using it heavily for actual data storage. What I distill from this (which is obviously my personal interpretation) is that Facebook mostly uses it for the OS and not for actual precious data.
  - geofft 9y ago
    I'm reading that as quite the opposite: they're saying that Gluster, a networked file storage system, is being backed with btrfs as the local filesystem, so all data stored in Gluster is ultimately stored on btrfs volumes. (They're also using it for OS snapshotting, yes, but insofar as the data stored in Gluster is important, they're storing important data on btrfs.) See also https://code.facebook.com/posts/938078729581886/improving-the-linux-kernel-with-upstream-contributions/ https://code.facebook.com/posts/938078729581886/improving-th... "We have been working toward deploying Btrfs slowly throughout the fleet, and we have been using large gluster storage clusters to help stabilize Btrfs. The gluster workloads are extremely demanding, and this half we gained a lot more confidence running Btrfs in production. More than 50 changes went into the stabilization effort, and Btrfs was able to protect production data from hardware bugs other filesystems would have missed."
vbernat 9y ago
Redhat was never a big contributor to BTRFS. This still means less users, but not less devs.
- acdha 9y ago
  For a filesystem I'd worry about what that implies for testing, especially for enterprise hardware and workloads. RHEL has significantly more users and it seems likely that their users would have more diversity than just Oracle shops.
  - kuschku 9y ago
    Are you completely forgetting SUSE? They’re a thing, too, after all.
    - acdha 9y ago
      Not forgetting them or Oracle but RHEL has something like twice the marketshare of the two combined. I'm not sure I've ever seen a support matrix where either of them is listed but RHEL is not, whereas the reverse is not uncommon.
josefbacik 9y ago
Facebook is using it for lots of shit. Chroots for containers because we can easily snapshot them and use them. Use it for build testing, so snapshot base repo, checkout commit, build, throw away. Gluster which takes huge backups and then other random workloads, it's used in a few places. Just because Facebook is fault tolerant doesn't mean we don't care about failures. We actively run down any issues we hit, so while it doesn't have much of impact in the short term, we don't just ignore issues. And also Red Hat hasn't contributed much to btrfs in years, and Oracle has one developer working on it. We are constantly working on it, and now that my priorities have shifted back to btrfs I hope that we will start to close out some of these long term projects.
- user5994461 9y ago
  Just because Facebook is working on btrfs in their OS, doesn't mean they will publish their code and patches to the public. Just because some code is added to the main kernel, assuming it is, doesn't mean it will be propagated to the main LTS distributions anytime soon.
  - josefbacik 9y ago
    Lol what? We don't ship code that isn't upstream. It sucks to maintain and rebase if you have a shitton of proprietary patches. Everything we do with the kernel is open source, you wouldn't be able to hire kernel developers if you didn't do open source. Nice comment tho, I legitimately lol'ed.