13 ms·

Just recently I had a worst-case nightmare scenario with a single BTRFS partition 5TB disk (no RAID or anything crazy). It was a single BTRFS disk, single par

Just recently I had a worst-case nightmare scenario with a single BTRFS partition 5TB disk (no RAID or anything crazy). It was a single BTRFS disk, single partition external USB disk running under Linux Mint. At some point after I restarted the computer and it wouldn't mount the partition saying that the path for mounting was already being used (even after restarting??), but the disk was not mounted. Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1]. At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. Which I did... and it supposedly changed successfully. Except that after trying to mount the partition again, it will tell me that there was an error in the partition: the tree had mismatching UUIDs :-/. I tried updating the UUID several times, and the command ended in success, but the error remained. My BTRFS disk was officially broken... After spending several hours trying to fix the issue following the BTRFS documentation (pretty shitty TBH), in the end, I ended up having to do `btrfs restore -iv ...` to extract the data from the disk into some other external disk (formatted NTFS this time!!). The command is still going, about 2 weeks later and I am slowly recovering my 5TB of data. But the one thing clear to me is that I don't trust BTRFS or it's admin commands AT ALL after this experience. Once I finish recovering my data, I'll nuke the partition, format the disk in NTFS and forget about this sour experience. [1] https://unix.stackexchange.com/questions/603528/why-am-i-getting-this-file-exists-error-trying-to-mount-unmounted-filesystem https://unix.stackexchange.com/questions/603528/why-am-i-get...

bravetraveler 3y ago
I can echo/support this claim - I wandered down the same rabbit hole. It doesn't even get much better with many drives and avoiding the docs/tooling BTRFS arrays will consistently corrupt with my reset button -- RAID10 on gen4 NVMe drives... while LVM/dm-raid + traditional file systems are absolutely fine
- xtracto 3y ago
  >BTRFS arrays will consistently corrupt with my reset button YES!!! Sorry to beat a dead horse, but it has been so frustrating for me. I thought I did something wrong, like, how could the partition break for doing nothing! Where did I fucked up? I cannot imagine the experience with a RAIDed BTRFS array shudders
  - phire 3y ago
    I had a BTRFS raid corrupt because one of my HDD had a firmware bug and was silently discarding writes. Sure... HDDs shouldn't do that, but it's literally the job of a BTRFS raid to protect me from such issues. The data was all still intact on the other drive. I could see it with dd, but BTRFS refused to read it.
- curt15 3y ago
  Isn't copy-on-write supposed to be inherently resilient to crashes[1]? Why would btrfs be more susceptible to resets than traditional filesystems? [1]: https://unix.stackexchange.com/questions/634050/in-what-ways-is-the-cow-filesystem-an-improvement-over-the-journaling-filesystem https://unix.stackexchange.com/questions/634050/in-what-ways...
  - bravetraveler 3y ago
    I doubt that because of the experience I just outlined In theory perhaps, but absolutely not in practice. Making things atomic is tricky.
  - londons_explore 3y ago
    I think most of the bugs are in the drives themselves - doing things like claiming data is flushed when it is not. However, btrfs is a very fragile filesystem and any inconsistencies caused by the drive bugs tend to lead to it breaking and refusing to mount, whereas on other filesystems you'll just get a warning in dmesg. It doesn't help that btrfs developers refuse to work around drive bugs.
    - phire 3y ago
      > It doesn't help that btrfs developers refuse to work around drive bugs. The entire selling point of the filesystem is that it's resilient to data corruption. Drive bugs are just another class of data corruption.
- yrro 3y ago
  > BTRFS arrays will consistently corrupt with my reset button I know you don't want to get asked this, and I wouldn't if you'd just said "I tried this once a while ago and it happened", but if it's consistent... have you filed a bug?
  - bravetraveler 3y ago
    It's a fair ask! I don't mind. It's occurred to me, but I expect a fair bit of push back; 'you abuse the array and expect it to... what?' I'm not really interested in debating that, and it's remarkably easy to get hung up on. I've floated this issue a few times unofficially and it's a constant stickler I know expecting coherency in this situation is a little silly, but BTRFS is notably less reliable/robust than the alternatives All in all... I'm avoiding a situation that may or may not occur, by not participating. Not saying it's a good thing, but it's easier for me to just use What Works
    - yrro 3y ago
      With properly functioning hardware (i.e., that doesn't lie to you when it says "that has been committed to durable storage") then metadata or data corruption following power loss is definitely a bug! It didn't even occur to me that you might get a different response. I'd be kind of annoyed if I went to the trouble of reporting a bug and got that response. But I'd also not assume it would end that way...
      - bravetraveler 3y ago
        With where we're at with SSDs chasing the performance dragon... I suspect some deception is at play! The DRAM caches are pretty hefty compared to metadata, so I wouldn't be surprised - though that's about my extent of understanding. Even then, though - other filesystems are certainly more durable, dealing with whatever these drives are doing. FWIW, this was with four Sabrent Rocket 4.0 Plus drives in RAID10 with 4K sectors. I'm not sure if these are particularly well known one way or another for deception/trickery They hold up just fine with either dm-raid or native LVM raid10 under more traditional filesystems (I tested EXT4/XFS, using the latter consistently)
Arnavion 3y ago
>Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1]. It's not a bug. It's just a fact that every BTRFS filesystem visible to the kernel (mounted or not) should have a unique ID. Otherwise operations on one filesystem (including mounting the partition that contains it) can end up applying to the other filesystem. >At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. The solution would've been to find out why you have two partitions with the same filesystem ID and remove one of them. One example is that if you have the BTRFS filesystem on an LVM partition and then you clone the LVM partition, you end up with two BTRFS filesystems with the same ID. I accidentally encountered it while converting an unencrypted BTRFS parition into an encrypted one by dd'ing it into a new LUKS device - when I tried to mount the encrypted partition it ended mounting the unencrypted one. Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.
- londons_explore 3y ago
  Sounds like a kernel bug ... If it encounters another filesystem with the same ID, it should warn in dmesg and assign a null ID to one of them.
  - _flux 3y ago
    If it was this simple, they'd have done it, right? The idea is that filesystems have have the same UUID are actually part of the same filesystem. It's a long-lasting hm feature of Btrfs. I think the actual dynamic filesystem discovery is in user space and kernel just gets a list of devices, so it should be even easier to change that.
    - thrtythreeforty 3y ago
      That's fine but we can have a "device UUID" and "filesystem UUID" as separate constructs and then the complaint remains valid. It's okay for it to fail if you dd and try to mount (suboptimal, but okay), but it should never silently start breaking things while appearing to work.
MaKey 3y ago
I'm not sure you can fault BTRFS for the issue you've experienced. It sounds like your system had a problem which lead to you breaking your (likely perfectly fine) BTRFS file system.
- deleted 3y ago
  [deleted]
- omniglottal 3y ago
  A filesystem breaks by the user pushing a button on their computer and you blame the user...?!
  - MaKey 3y ago
    He had trouble mounting it on his system because of a duplicate UUID - this doesn't mean that the filesystem was broken. As Arnavion wrote: > Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.
_flux 3y ago
Surely this incident highlights the importance of backups, right? 5 TB is even a manageable amount of data. I also used to run btrfs in btrfs-RAID10 configuration until apparently a flapping SATA link and fsck attempts were able to break the fs completely. Full system backups were great that day. I run https://kopia.io/ https://kopia.io/ nowadays every three hours during day time and I've been quite happy with it. Nowadays I run bcachefs.. Backups are still handy :). I suppose the reason why you chose NTFS was to be able to access the data from Windows, at least in case of emergency? Because there are a lot of filesystems that are presumably more mature than NTFS is for Linux.
- xtracto 3y ago
  The finny thing is that , THAT is my backup drive, mostly. It has lots of crap, including GB synced from Google Drive (photos being the mos important). Other than that, it's stuff I could re-download from elsewhere, but 3TB is a lot to re-download. Aah yeah and it also has the TimeHift backups. I thought of going EXT4 , but as you said, NTFS is more widely supported.
generalizations 3y ago
That doesn't even sound like a BTRFS problem; I recently (couple weekends ago) had the same UUID cache problem with an XFS partition.

bravetraveler 3y ago

I can echo/support this claim - I wandered down the same rabbit hole. It doesn't even get much better with many drives and avoiding the docs/tooling BTRFS arrays will consistently corrupt with my reset button -- RAID10 on gen4 NVMe drives... while LVM/dm-raid + traditional file systems are absolutely fine

xtracto 3y ago
>BTRFS arrays will consistently corrupt with my reset button YES!!! Sorry to beat a dead horse, but it has been so frustrating for me. I thought I did something wrong, like, how could the partition break for doing nothing! Where did I fucked up? I cannot imagine the experience with a RAIDed BTRFS array shudders
- phire 3y ago
  I had a BTRFS raid corrupt because one of my HDD had a firmware bug and was silently discarding writes. Sure... HDDs shouldn't do that, but it's literally the job of a BTRFS raid to protect me from such issues. The data was all still intact on the other drive. I could see it with dd, but BTRFS refused to read it.
curt15 3y ago
Isn't copy-on-write supposed to be inherently resilient to crashes[1]? Why would btrfs be more susceptible to resets than traditional filesystems? [1]: https://unix.stackexchange.com/questions/634050/in-what-ways-is-the-cow-filesystem-an-improvement-over-the-journaling-filesystem https://unix.stackexchange.com/questions/634050/in-what-ways...
- bravetraveler 3y ago
  I doubt that because of the experience I just outlined In theory perhaps, but absolutely not in practice. Making things atomic is tricky.
- londons_explore 3y ago
  I think most of the bugs are in the drives themselves - doing things like claiming data is flushed when it is not. However, btrfs is a very fragile filesystem and any inconsistencies caused by the drive bugs tend to lead to it breaking and refusing to mount, whereas on other filesystems you'll just get a warning in dmesg. It doesn't help that btrfs developers refuse to work around drive bugs.
  - phire 3y ago
    > It doesn't help that btrfs developers refuse to work around drive bugs. The entire selling point of the filesystem is that it's resilient to data corruption. Drive bugs are just another class of data corruption.
yrro 3y ago
> BTRFS arrays will consistently corrupt with my reset button I know you don't want to get asked this, and I wouldn't if you'd just said "I tried this once a while ago and it happened", but if it's consistent... have you filed a bug?
- bravetraveler 3y ago
  It's a fair ask! I don't mind. It's occurred to me, but I expect a fair bit of push back; 'you abuse the array and expect it to... what?' I'm not really interested in debating that, and it's remarkably easy to get hung up on. I've floated this issue a few times unofficially and it's a constant stickler I know expecting coherency in this situation is a little silly, but BTRFS is notably less reliable/robust than the alternatives All in all... I'm avoiding a situation that may or may not occur, by not participating. Not saying it's a good thing, but it's easier for me to just use What Works
  - yrro 3y ago
    With properly functioning hardware (i.e., that doesn't lie to you when it says "that has been committed to durable storage") then metadata or data corruption following power loss is definitely a bug! It didn't even occur to me that you might get a different response. I'd be kind of annoyed if I went to the trouble of reporting a bug and got that response. But I'd also not assume it would end that way...
    - bravetraveler 3y ago
      With where we're at with SSDs chasing the performance dragon... I suspect some deception is at play! The DRAM caches are pretty hefty compared to metadata, so I wouldn't be surprised - though that's about my extent of understanding. Even then, though - other filesystems are certainly more durable, dealing with whatever these drives are doing. FWIW, this was with four Sabrent Rocket 4.0 Plus drives in RAID10 with 4K sectors. I'm not sure if these are particularly well known one way or another for deception/trickery They hold up just fine with either dm-raid or native LVM raid10 under more traditional filesystems (I tested EXT4/XFS, using the latter consistently)

Arnavion 3y ago

>Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1]. It's not a bug. It's just a fact that every BTRFS filesystem visible to the kernel (mounted or not) should have a unique ID. Otherwise operations on one filesystem (including mounting the partition that contains it) can end up applying to the other filesystem. >At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. The solution would've been to find out why you have two partitions with the same filesystem ID and remove one of them. One example is that if you have the BTRFS filesystem on an LVM partition and then you clone the LVM partition, you end up with two BTRFS filesystems with the same ID. I accidentally encountered it while converting an unencrypted BTRFS parition into an encrypted one by dd'ing it into a new LUKS device - when I tried to mount the encrypted partition it ended mounting the unencrypted one. Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.

londons_explore 3y ago
Sounds like a kernel bug ... If it encounters another filesystem with the same ID, it should warn in dmesg and assign a null ID to one of them.
- _flux 3y ago
  If it was this simple, they'd have done it, right? The idea is that filesystems have have the same UUID are actually part of the same filesystem. It's a long-lasting hm feature of Btrfs. I think the actual dynamic filesystem discovery is in user space and kernel just gets a list of devices, so it should be even easier to change that.
  - thrtythreeforty 3y ago
    That's fine but we can have a "device UUID" and "filesystem UUID" as separate constructs and then the complaint remains valid. It's okay for it to fail if you dd and try to mount (suboptimal, but okay), but it should never silently start breaking things while appearing to work.

MaKey 3y ago

I'm not sure you can fault BTRFS for the issue you've experienced. It sounds like your system had a problem which lead to you breaking your (likely perfectly fine) BTRFS file system.

deleted 3y ago
[deleted]
omniglottal 3y ago
A filesystem breaks by the user pushing a button on their computer and you blame the user...?!
- MaKey 3y ago
  He had trouble mounting it on his system because of a duplicate UUID - this doesn't mean that the filesystem was broken. As Arnavion wrote: > Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.

_flux 3y ago

Surely this incident highlights the importance of backups, right? 5 TB is even a manageable amount of data. I also used to run btrfs in btrfs-RAID10 configuration until apparently a flapping SATA link and fsck attempts were able to break the fs completely. Full system backups were great that day. I run https://kopia.io/ https://kopia.io/ nowadays every three hours during day time and I've been quite happy with it. Nowadays I run bcachefs.. Backups are still handy :). I suppose the reason why you chose NTFS was to be able to access the data from Windows, at least in case of emergency? Because there are a lot of filesystems that are presumably more mature than NTFS is for Linux.

xtracto 3y ago
The finny thing is that , THAT is my backup drive, mostly. It has lots of crap, including GB synced from Google Drive (photos being the mos important). Other than that, it's stuff I could re-download from elsewhere, but 3TB is a lot to re-download. Aah yeah and it also has the TimeHift backups. I thought of going EXT4 , but as you said, NTFS is more widely supported.

generalizations 3y ago

That doesn't even sound like a BTRFS problem; I recently (couple weekends ago) had the same UUID cache problem with an XFS partition.