9 ms·

Migrating Away from Bcachefs

83 points by todsacerdoti 2y ago

Darkskiez 2y ago
I need to write up my experience. But I'm trying it out. Linux needs something like this. I've had issues, posted traces and had them fixed in seconds. Pretty damn amazing. I'd love to see a bigger team involved though.
- 7e 2y ago
  A change to a filesystem should never be made in seconds.
  - hinkley 2y ago
    Confidence intervals don’t have precise timelines associated with them. Sometimes you know exactly what the problem is when you hear the symptoms. We always balance new work versus cleanup. I always have a laundry list of beefs with my own work. You often have a sneaking suspicion that this piece of code is wrong in a manner you can’t quite put your finger on. Or you do but a new customer is trumping a hypothetical error. Or ten people are blocked waiting for you to merge some other feature. And then someone gives you a test case and you know exactly what is wrong with it. Sometimes that’s fixing a null check, an off by one error, or sorting results. And sometimes the repro case basically writes itself.
- gkmcd 2y ago
  My experience also. Kent is obviously very committed to the project.
kstrauser 2y ago
What's all that about the bcachefs author's complaints with Rust and Debian? I'm far out of the loop on this stuff.
- ajb 2y ago
  His side is that basically it doesn't make sense to package the tools for an experimental FS in an OS that's going to get very far behind, as debian stable will do, since the tools have to iterate rapidly to fix problems with releases of the FS. Debian-stable had an old set of some rust libraries and the packager relaxed the dependency constraints in order to package it, which doesn't make a lot of sense on something you are hoping to fix your filesystem. Basically it shouldn't be packaged for LTS-class OS releases until it's not experimental.
  - kstrauser 2y ago
    Ah, I see. So "the Rust team" here means the team of Debian maintainers who package Rust for it, not the Rust developers? That makes sense.
    - ajb 2y ago
      Yeah I assume that's what "the Rust team" meant
  - knowitnone 2y ago
    This makes perfect sense to me. Why would Debian-stable include something obviously not stable. They have an experimental branch for a reason
    - 6SixTy 2y ago
      To my knowledge, Debian changed deps in bcachefs-tools to synchronize with Debian's Rust repo, breaking it. It's one part that it's a green fs, and the other with clashing expectations of how to treat dependencies between Rust and Debian.
  - bhelkey 2y ago
    > [bcachefs auhor's] side is that basically it doesn't make sense to package the tools for an experimental FS in an OS that's going to get very far behind If the bcachefs author's believe the tool is too unstable for Debian stable and Debian developer's believe bcachefs is too unstable for Debian stable [1], it sounds as if they agree. [1] https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/ https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...
ajb 2y ago
Bcachefs is experimental, and last I heard, the authors hope to be able to declare it not experimental in ~6 months. To me, that's 'try on a cache server/build server' territory, not on anything where you even think about backing it up. Kent Overstreet does have a problem working with people; I can well believe that interactions around bugs were painful. He likely he should try to hire someone else to deal with users and distributions and use his limited patience to deal with the kernel maintainers. But it sounds like the OP was a bit naive about what it means to run an experimental FS.
- AceJohnny2 2y ago
  > last I heard, the authors hope to be able to declare it not experimental in ~6 months How long have they been saying that? I feel like that's been the case for years.
  - ajb 2y ago
    It's only been in the upstream kernel 6 months or so? I know the FS has actually been around longer, but I've not heard claims about it's stability before that. It's only now that's it's getting a lot more testing
    - pixelesque 2y ago
      It was merged in 6.7, 12 months ago...
    - aragilar 2y ago
      Searching bcachefs on LWN (https://lwn.net/Search/DoTextSearch https://lwn.net/Search/DoTextSearch) does make for a somewhat interesting read e.g. https://lwn.net/Articles/717379/ https://lwn.net/Articles/717379/ (from 2017) which suggests the disc format would have one last change (c.f. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2622f2904170 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... which is for the upcoming 6.14 release, which notes the format will have at least one more changes after the one included), and the numerous articles on its upstreaming (e.g. https://lwn.net/Articles/755276/ https://lwn.net/Articles/755276/ from 2018), and the stability claims do seem a bit, well, optimistic?
agartner 2y ago
I've been patiently waiting to convert my ZFS array to bcachefs. I'm very excited about better SSD promotion logic. But I'm not willing to spend any time on an experimental filesystem on my main systems. > But you can expect to get flamed about running Debian, or perhaps more accurately, not being willing to spearhead Kent's crusade against Debian's Rust packaging policies. It is quite unfortunate that Kent couldn't have just said "Debian isn't supported, we will revisit this when bcachefs is more stable" and stopped talking after that. Debian and experimental software just don't work well together.
- ThatPlayer 2y ago
  I've been running bcachefs on my spare dedicated SteamOS gaming machine for fun. Especially for the SSD promotion logic. It's a spare computer with an old 128GB SSD and 3TB HDD that I've got as a single filesystem. I love not having to manage games between the SSD/HDD. Too bad it's a mITX build with no space for more old drives I could stick in.
- koverstreet 2y ago
  Oh, the author's completely misrepresenting what happened here. We had a major snafu with Debian, where a maintainer volunteered to package bcachefs-tools (this was not something I ever asked for!), and I explained that Debian policy on Rust dependencies would cause problems, and asked him not to do that. But he did debundle, and then down the road he broke the build (by debundled bindgen and ignoring the minimum version we'd specified), and then _sat on it for months_, without even reporting the issue. So Debian users weren't getting updates, and that meant they didn't get a critical bugfix that fixed passing of mount options. Then a bunch of Debian users weren't able to mount in degraded mode when a drive died. And guess who was fielding bug reports? After that is when I insisted that if bcachefs-tools is packaged for debian, dependencies don't get debundled. If you're going to create a mess and drop it in my lap, and it results in users not able to access their filesystem, don't go around bitching about being asked to not repeat that.
  - agartner 2y ago
    Yeah just typical Debian stuff. jwz has been ranting about this for years. It's not worth spending any time on it. Some suggestions: - Only "supporting" the latest mainline kernel and latest tools. I prefer to point to CI system configurations to show exactly what it "supported" - Make this clear via your website and a pinned issue on Github. - Force users to report the versions they use via an issue template: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository https://docs.github.com/en/communities/using-templates-to-en.... Immediately close any issues not meeting your version/system requirements without further discussion or thought.
snvzz 2y ago
I stick to OpenZFS, it's rock solid. Filesystems are the one thing I don't want to experiment with. Too much potential trouble.
- MaKey 2y ago
  Can it still be called "rock solid" with a data corruption bug in its younger history? https://news.ycombinator.com/item?id=38770168 https://news.ycombinator.com/item?id=38770168
  - helf 2y ago
    Yes.
  - hulitu 2y ago
    > Can it still be called "rock solid" with a data corruption bug in its younger history? https://news.ycombinator.com/item?id=38770168 https://news.ycombinator.com/item?id=38770168 Ok.It is sand solid. There were a lot of voices on HN trying to downplay the issue, but, a filesystem cannot be named stable, if it corrupts data, in 2020.
  - snvzz 2y ago
    Absolutely. We can trust it because it has already done through all this. It is battle-proven. Unlike some new filesystem created just yesterday.
  - koverstreet 2y ago
    Every filesystem has had data corruption bugs. Unfortunately, that's just how it is with the complexity of modern filesystems and the tools we have. (We're still coding in bloody _C_). What you really don't want is to be losing entire filesytems. A single file here and there getting corrupted is one thing, and it's usually highly workload dependent so it'll probably only hit one application, but losing an entire filesystem is much more impactful. ext 2/3/4 has the best track record here (and note, it's also had data corruption bugs), with the relatively simple on disk format and e2fsck being quite robust. XFS is probably next up, I've seen reports of XFS filesystems being lost but only one or two - a tiny, tiny fraction of the btrfs reports. Can't speak as much to ZFS.
honestSysAdmin 2y ago
I don't expect ZFS to be dethroned this decade, nor the next one.
- nicman23 2y ago
  I used to run zfs with bcache blocks lol
  - honestSysAdmin 2y ago
    Some of the zpool configurations I have set up seem "odd" to others, but they do have a purpose and are well thought out. What motivated you to combine ZFS and bcache? What did that configuration look like? What was the thinking behind it? I would prefer to inquire further rather than make and present wrong assumptions as to why the configuration is wrong.
pronoiac 2y ago
Is XFS well-regarded by others? I was benchmarking filesystems by generating a billion empty files on them, and while ext2, ext4, and btrfs could finish in a day or two, xfs hit a wall in the first 4 million files, and was on track to take weeks. After hitting ctrl-c, it took 20+ minutes to get a responsive shell, and unmounting took a few minutes. This surprised me, because it's been around for decades, and I expected scalability, performance, and robustness. A one-user accidental DOS was a surprising rake to trip over.
- ak217 2y ago
  I don't know what caused your experience, but I've had the opposite experience using XFS with many small files. It's my filesystem of choice for simple (single device) use cases. The main reason is that XFS automatically allocates inode space. With ext4, I would quickly run out of the default inode quota while the volume was nowhere near full, and then manually tune the inode to block ratio to accommodate more files. XFS took care of that automatically. Performance was otherwise identical, and I've never seen any data loss bugs or even crashes from either one.
  - GuestFAUniverse 2y ago
    How many decades ago was that? Sounds more like a partition converted from ext3. No ext4 partition I've seen in the last 15y, didn't have a ridiculous amount of inodes. I do support for several hundred Linux systems.
    - starspangled 2y ago
      From this paper https://arxiv.org/html/2408.01805v1 https://arxiv.org/html/2408.01805v1 (2024) > EXT4 is the general-purpose filesystem for a majority of Linux distributions and gets installed as the default. However, it is unable to accommodate one billion files by default due to a small number of inodes available, typically set at 240 million files or folders, created during installation. Which is interesting. I knew EXT2/3/4 had inode bitmaps, but I haven't been paying them much attention for the past decade. Slightly surprised they haven't added an option for dynamic allocation, OTOH inodes are small compared with storage and most people don't need billions of files.
arp242 2y ago
Another thing is that I'm half expecting bcachefs to be pulled from mainline. That's my prediction for 2025, I guess? Not that I want that to happen, but Kent's behaviour has been ... difficult, and Linus already hinted this is a possibility. I appreciate that Kent means well, but as an outsider reading some of his communications ... yeah, I'd struggle working with that too.
nicman23 2y ago
User uses experimental fs. It breaks. Surprised Pikachu face
- stavros 2y ago
  If that's what you got from the article, I recommend you read it again.
- aragilar 2y ago
  Yes and no—said user ran this as an experiment (relevant quote: "But with a filesystem this young, you inevitably have to accept some rough edges in return for some fun."), but eventually stopped it partially due to the people involved (relevant quotes: "But here's the main issue for me: Reporting bugs to bcachefs is not a pleasant experience."; "And at some point, the fun just stopped for me."), not just because there were bugs.
  - nicman23 2y ago
    fun is not something i want to describe my fs as

83 points by todsacerdoti 2y ago

I need to write up my experience. But I'm trying it out. Linux needs something like this. I've had issues, posted traces and had them fixed in seconds. Pretty damn amazing. I'd love to see a bigger team involved though.

7e 2y ago
A change to a filesystem should never be made in seconds.
- hinkley 2y ago
  Confidence intervals don’t have precise timelines associated with them. Sometimes you know exactly what the problem is when you hear the symptoms. We always balance new work versus cleanup. I always have a laundry list of beefs with my own work. You often have a sneaking suspicion that this piece of code is wrong in a manner you can’t quite put your finger on. Or you do but a new customer is trumping a hypothetical error. Or ten people are blocked waiting for you to merge some other feature. And then someone gives you a test case and you know exactly what is wrong with it. Sometimes that’s fixing a null check, an off by one error, or sorting results. And sometimes the repro case basically writes itself.
gkmcd 2y ago
My experience also. Kent is obviously very committed to the project.

kstrauser 2y ago

What's all that about the bcachefs author's complaints with Rust and Debian? I'm far out of the loop on this stuff.

ajb 2y ago
His side is that basically it doesn't make sense to package the tools for an experimental FS in an OS that's going to get very far behind, as debian stable will do, since the tools have to iterate rapidly to fix problems with releases of the FS. Debian-stable had an old set of some rust libraries and the packager relaxed the dependency constraints in order to package it, which doesn't make a lot of sense on something you are hoping to fix your filesystem. Basically it shouldn't be packaged for LTS-class OS releases until it's not experimental.
- kstrauser 2y ago
  Ah, I see. So "the Rust team" here means the team of Debian maintainers who package Rust for it, not the Rust developers? That makes sense.
  - ajb 2y ago
    Yeah I assume that's what "the Rust team" meant
- knowitnone 2y ago
  This makes perfect sense to me. Why would Debian-stable include something obviously not stable. They have an experimental branch for a reason
  - 6SixTy 2y ago
    To my knowledge, Debian changed deps in bcachefs-tools to synchronize with Debian's Rust repo, breaking it. It's one part that it's a green fs, and the other with clashing expectations of how to treat dependencies between Rust and Debian.
- bhelkey 2y ago
  > [bcachefs auhor's] side is that basically it doesn't make sense to package the tools for an experimental FS in an OS that's going to get very far behind If the bcachefs author's believe the tool is too unstable for Debian stable and Debian developer's believe bcachefs is too unstable for Debian stable [1], it sounds as if they agree. [1] https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/ https://jonathancarter.org/2024/08/29/orphaning-bcachefs-too...

ajb 2y ago

Bcachefs is experimental, and last I heard, the authors hope to be able to declare it not experimental in ~6 months. To me, that's 'try on a cache server/build server' territory, not on anything where you even think about backing it up. Kent Overstreet does have a problem working with people; I can well believe that interactions around bugs were painful. He likely he should try to hire someone else to deal with users and distributions and use his limited patience to deal with the kernel maintainers. But it sounds like the OP was a bit naive about what it means to run an experimental FS.

AceJohnny2 2y ago
> last I heard, the authors hope to be able to declare it not experimental in ~6 months How long have they been saying that? I feel like that's been the case for years.
- ajb 2y ago
  It's only been in the upstream kernel 6 months or so? I know the FS has actually been around longer, but I've not heard claims about it's stability before that. It's only now that's it's getting a lot more testing
  - pixelesque 2y ago
    It was merged in 6.7, 12 months ago...
  - aragilar 2y ago
    Searching bcachefs on LWN (https://lwn.net/Search/DoTextSearch https://lwn.net/Search/DoTextSearch) does make for a somewhat interesting read e.g. https://lwn.net/Articles/717379/ https://lwn.net/Articles/717379/ (from 2017) which suggests the disc format would have one last change (c.f. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2622f2904170 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... which is for the upcoming 6.14 release, which notes the format will have at least one more changes after the one included), and the numerous articles on its upstreaming (e.g. https://lwn.net/Articles/755276/ https://lwn.net/Articles/755276/ from 2018), and the stability claims do seem a bit, well, optimistic?

agartner 2y ago

I've been patiently waiting to convert my ZFS array to bcachefs. I'm very excited about better SSD promotion logic. But I'm not willing to spend any time on an experimental filesystem on my main systems. > But you can expect to get flamed about running Debian, or perhaps more accurately, not being willing to spearhead Kent's crusade against Debian's Rust packaging policies. It is quite unfortunate that Kent couldn't have just said "Debian isn't supported, we will revisit this when bcachefs is more stable" and stopped talking after that. Debian and experimental software just don't work well together.

ThatPlayer 2y ago
I've been running bcachefs on my spare dedicated SteamOS gaming machine for fun. Especially for the SSD promotion logic. It's a spare computer with an old 128GB SSD and 3TB HDD that I've got as a single filesystem. I love not having to manage games between the SSD/HDD. Too bad it's a mITX build with no space for more old drives I could stick in.
koverstreet 2y ago
Oh, the author's completely misrepresenting what happened here. We had a major snafu with Debian, where a maintainer volunteered to package bcachefs-tools (this was not something I ever asked for!), and I explained that Debian policy on Rust dependencies would cause problems, and asked him not to do that. But he did debundle, and then down the road he broke the build (by debundled bindgen and ignoring the minimum version we'd specified), and then _sat on it for months_, without even reporting the issue. So Debian users weren't getting updates, and that meant they didn't get a critical bugfix that fixed passing of mount options. Then a bunch of Debian users weren't able to mount in degraded mode when a drive died. And guess who was fielding bug reports? After that is when I insisted that if bcachefs-tools is packaged for debian, dependencies don't get debundled. If you're going to create a mess and drop it in my lap, and it results in users not able to access their filesystem, don't go around bitching about being asked to not repeat that.
- agartner 2y ago
  Yeah just typical Debian stuff. jwz has been ranting about this for years. It's not worth spending any time on it. Some suggestions: - Only "supporting" the latest mainline kernel and latest tools. I prefer to point to CI system configurations to show exactly what it "supported" - Make this clear via your website and a pinned issue on Github. - Force users to report the versions they use via an issue template: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository https://docs.github.com/en/communities/using-templates-to-en.... Immediately close any issues not meeting your version/system requirements without further discussion or thought.

snvzz 2y ago

I stick to OpenZFS, it's rock solid. Filesystems are the one thing I don't want to experiment with. Too much potential trouble.

MaKey 2y ago
Can it still be called "rock solid" with a data corruption bug in its younger history? https://news.ycombinator.com/item?id=38770168 https://news.ycombinator.com/item?id=38770168
- helf 2y ago
  Yes.
- hulitu 2y ago
  > Can it still be called "rock solid" with a data corruption bug in its younger history? https://news.ycombinator.com/item?id=38770168 https://news.ycombinator.com/item?id=38770168 Ok.It is sand solid. There were a lot of voices on HN trying to downplay the issue, but, a filesystem cannot be named stable, if it corrupts data, in 2020.
- snvzz 2y ago
  Absolutely. We can trust it because it has already done through all this. It is battle-proven. Unlike some new filesystem created just yesterday.
- koverstreet 2y ago
  Every filesystem has had data corruption bugs. Unfortunately, that's just how it is with the complexity of modern filesystems and the tools we have. (We're still coding in bloody _C_). What you really don't want is to be losing entire filesytems. A single file here and there getting corrupted is one thing, and it's usually highly workload dependent so it'll probably only hit one application, but losing an entire filesystem is much more impactful. ext 2/3/4 has the best track record here (and note, it's also had data corruption bugs), with the relatively simple on disk format and e2fsck being quite robust. XFS is probably next up, I've seen reports of XFS filesystems being lost but only one or two - a tiny, tiny fraction of the btrfs reports. Can't speak as much to ZFS.

honestSysAdmin 2y ago

I don't expect ZFS to be dethroned this decade, nor the next one.

nicman23 2y ago
I used to run zfs with bcache blocks lol
- honestSysAdmin 2y ago
  Some of the zpool configurations I have set up seem "odd" to others, but they do have a purpose and are well thought out. What motivated you to combine ZFS and bcache? What did that configuration look like? What was the thinking behind it? I would prefer to inquire further rather than make and present wrong assumptions as to why the configuration is wrong.

pronoiac 2y ago

Is XFS well-regarded by others? I was benchmarking filesystems by generating a billion empty files on them, and while ext2, ext4, and btrfs could finish in a day or two, xfs hit a wall in the first 4 million files, and was on track to take weeks. After hitting ctrl-c, it took 20+ minutes to get a responsive shell, and unmounting took a few minutes. This surprised me, because it's been around for decades, and I expected scalability, performance, and robustness. A one-user accidental DOS was a surprising rake to trip over.

ak217 2y ago
I don't know what caused your experience, but I've had the opposite experience using XFS with many small files. It's my filesystem of choice for simple (single device) use cases. The main reason is that XFS automatically allocates inode space. With ext4, I would quickly run out of the default inode quota while the volume was nowhere near full, and then manually tune the inode to block ratio to accommodate more files. XFS took care of that automatically. Performance was otherwise identical, and I've never seen any data loss bugs or even crashes from either one.
- GuestFAUniverse 2y ago
  How many decades ago was that? Sounds more like a partition converted from ext3. No ext4 partition I've seen in the last 15y, didn't have a ridiculous amount of inodes. I do support for several hundred Linux systems.
  - starspangled 2y ago
    From this paper https://arxiv.org/html/2408.01805v1 https://arxiv.org/html/2408.01805v1 (2024) > EXT4 is the general-purpose filesystem for a majority of Linux distributions and gets installed as the default. However, it is unable to accommodate one billion files by default due to a small number of inodes available, typically set at 240 million files or folders, created during installation. Which is interesting. I knew EXT2/3/4 had inode bitmaps, but I haven't been paying them much attention for the past decade. Slightly surprised they haven't added an option for dynamic allocation, OTOH inodes are small compared with storage and most people don't need billions of files.

arp242 2y ago

Another thing is that I'm half expecting bcachefs to be pulled from mainline. That's my prediction for 2025, I guess? Not that I want that to happen, but Kent's behaviour has been ... difficult, and Linus already hinted this is a possibility. I appreciate that Kent means well, but as an outsider reading some of his communications ... yeah, I'd struggle working with that too.

nicman23 2y ago

User uses experimental fs. It breaks. Surprised Pikachu face

stavros 2y ago
If that's what you got from the article, I recommend you read it again.
aragilar 2y ago
Yes and no—said user ran this as an experiment (relevant quote: "But with a filesystem this young, you inevitably have to accept some rough edges in return for some fun."), but eventually stopped it partially due to the people involved (relevant quotes: "But here's the main issue for me: Reporting bugs to bcachefs is not a pleasant experience."; "And at some point, the fun just stopped for me."), not just because there were bugs.
- nicman23 2y ago
  fun is not something i want to describe my fs as