6 ms·

XFS online filesystem check and repair

50 points by psxuaw 3y ago

colechristensen 3y ago
Back in the day XFS was great and worked excellently and was quite resilient, performant, etc. etc... until it wasn't. If it got too full or somehow corrupted beyond its capacity to recover then you were just screwed and it was very difficult to actually get your data back. Like it had a much higher threshold for being shitty and was great before you hit that threshold but after you hit it it was much worse than ext2/3 or whatever else I was using. Something around 80% full and it started to get awful and you risked losing your entire filesystem. I'd have real reservations about knowingly using it again these days because of bad experiences, though this bias is based on really old experiences which are quite possibly no longer relevant.
- butlerm 3y ago
  That sounds like the experience a lot of people had with BTRFS (unfortunately).
  - bravetraveler 3y ago
    Yep, ENOSPC and BTRFS are old friends
- linsomniac 3y ago
  I have two bad XFS experiences that stick out in my memory. I like XFS, don't get me wrong. I ran XFS on our Linux distribution/package mirror. We ended up having some sort of error and the filesystem wouldn't mount, so I tried running the fsck. After around a day, it bombed out with out of memory. I scrounged around, took down another server or two to max out the memory on the mirror server and eventually was able to get it to complete the fsck in a couple more days. For a while I ran XFS on my laptop. This was some Thinkpad that had fancy-pants graphics, and the graphics drivers made the system quite unstable It would oops every 2-4 hours of normal use. XFS had this bad habit at the time (long since fixed, IIRC) of reverting files to HOURS old data after the reboot. A peer of this comment mentioned btrfs. Around the same time I played around with btrfs. For around a year I ran a couple of my company laptops on btrfs with zero problems. Then, after one set of updates, both of them completely corrupted their filesystems. I stopped using it, hoping that btrfs would fairly quickly stabilize, but have never ended up going back to it. I love the idea of btrfs, but am a bit gun-shy. I used Reiser extensively for map tile file-systems, largely because their idea of a "small file" was 1-2 orders of magnitude smaller than most other filesystems. This particular use case was full world tiles, and a lot of them were solid blue (oceans, you know), I want to say 37 bytes. My current gig puts tiles on XFS and it works great for that. ZFS is a filesystem that has never let me down. Performance can often be spotty, especially if you think you can use dedup, but I've had some pretty bad things happen to ZFS volumes and have never lost data on one of them. It's really the only place I'll put archive data. I've run a lot of backup servers using ZFS, even back in the days of zfsfuse, and it's just been a workhorse. Even in early days when zfsfuse had some pretty obscure bugs and would choke about every month or two. Kudos to the zfs+fuse developers, they quickly tracked down every issue I ran into once I had a stress test use case going. EXT has been a real workhorse too. I've lost data, rarely, but considering it's 99% of the filesystems I've run over the decades... Pretty good track record. Veritas vx on HP-UX was like a space age filesystem back in the mid '90s, but also was kind of painful. My memories of it are just vague and bad. I'd love to have HAMMER under Linux. :-) Ok, discuss!
  - SubjectToChange 3y ago
    I’ve been running btrfs on my laptop and workstation since 2015 and haven’t had any issues. I get all the advanced FS functionality I want without needing to worry about kernel versions or any of the other hassle of setting up ZFS. In many ways ZFS feels like the last hurrah of “traditional” file systems. The fact that ZFS finds itself without genuine alternatives isn’t because it’s just that good, it’s because industry players have moved on. If all you need is a monster file server or a local data archive, sure, use ZFS.
    - abwizz 3y ago
      > file server or a local data archive thats all you want from a fs, isn't it? everything else is temporary data, living in memory.
  - mrktf 3y ago
    With btrfs, interestingly enough never lost data, but got volume stuck to read only mode (fsck didn't help) multiple times over the years. On sad note qgroup qouta (it is fancy qouta - you can limit disk usage per subvolume basis https://btrfs.readthedocs.io/en/latest/btrfs-quota.html#performance-implications https://btrfs.readthedocs.io/en/latest/btrfs-quota.html#perf... sounds much more positive than real impact :/ although didn't test with latest kernel ) starts struggle when there are load using 5 users (using docker containers with jupyterhub) and when load goes near 10 docker instances, new docker container creation literally crawls - start taking minutes
- pbh101 3y ago
  In my experience enabling the finobt (free inode btree) setting made for a better experience under consistent capacity pressure. Context: CentOS 7, 7 TB RAID-10 SSD with ~100million files and ~4% churn rate per day, in production above 80% full for most of 5 years. Twice we experienced an XFS issue but xfs_repair got the FS back online and intact.
- renewiltord 3y ago
  Back in the early 2000s I had XFS zero out some files on power outage. Well known bug at the time. Fixed soon but retained the branding. What a time. People would recommend ReiserFS back then. Big fan club.
  - adrian_b 3y ago
    Yes, it was very annoying and it happened even to files that were open only for reading, not for writing. For me, /etc/fstab was zeroed a couple of times, which lead to a bricked system (before 2000). Fortunately, this bug was eventually solved and I have been using XFS for more than twenty years on a great variety of hardware, with excellent results and no data loss whatsoever.
  - anthk 3y ago
    Reiser and XFS were totally opposite in design. Reiser was born for being fast with tons of small files, suitable for your KDE desktop with lots of software/kparts/plugins installed. XFS was created to manage big multimedia files, such as uncut/unedited RAW photos and videos.
hestefisk 3y ago
Is there any case where one would choose this over ext4 or even zfs?
- postmodest 3y ago
  On spinning rust, it had guaranteed bandwidth and multiple I/O. On NVME? Not really.
  - jeffbee 3y ago
    On Linux, XFS never offered guaranteed-rate I/O. It was really an IRIX feature that also required hardware support.
    - T3OU-736 3y ago
      Fun story. GRIO (Guaranteed Rate IO) XFS support was added to IRIX due to a request by a US Gov't agency. This agency was doing electronic signal collection with IRIX on SGI HW, and wanted to a way to guarantee that whatever their sensors captured would be written to the storage media, no matter what else is going on at the OS level.
- phsau 3y ago
  It's workload dependent; XFS performs better for our database needs.
  - SkyMarshal 3y ago
    Compared to Ext4 or ZFS?
    - abwizz 3y ago
      i'd rekon cow fs performs somewhat poorly with db workload, so, both
- brucethemoose2 3y ago
  zfs and xfs is apples to oranges. One is simple and low overhead, the other is stuffed with features and options. ext4 gives me questionable performance with writing a huge amount (like 50k+) of small files, but other than that its great.
jeffbee 3y ago
I find the whole concept slightly scary. Some reasons for an online filesystem consistency check to fail are that the hardware has lied to the operating system about the success of some past operation, the filesystem has gone down a bad road and written garbage to a storage device, or the checker and the filesystem are inconsistent, with uncertainty about which side is correct. Under all those conditions I think I want my system to halt and log. Continuing seems like a real bad idea. There are fault-tolerant ways of coding operating systems and filesystem but it does not seem like those ways are practiced by Linux.

50 points by psxuaw 3y ago

Back in the day XFS was great and worked excellently and was quite resilient, performant, etc. etc... until it wasn't. If it got too full or somehow corrupted beyond its capacity to recover then you were just screwed and it was very difficult to actually get your data back. Like it had a much higher threshold for being shitty and was great before you hit that threshold but after you hit it it was much worse than ext2/3 or whatever else I was using. Something around 80% full and it started to get awful and you risked losing your entire filesystem. I'd have real reservations about knowingly using it again these days because of bad experiences, though this bias is based on really old experiences which are quite possibly no longer relevant.

butlerm 3y ago
That sounds like the experience a lot of people had with BTRFS (unfortunately).
- bravetraveler 3y ago
  Yep, ENOSPC and BTRFS are old friends
linsomniac 3y ago
I have two bad XFS experiences that stick out in my memory. I like XFS, don't get me wrong. I ran XFS on our Linux distribution/package mirror. We ended up having some sort of error and the filesystem wouldn't mount, so I tried running the fsck. After around a day, it bombed out with out of memory. I scrounged around, took down another server or two to max out the memory on the mirror server and eventually was able to get it to complete the fsck in a couple more days. For a while I ran XFS on my laptop. This was some Thinkpad that had fancy-pants graphics, and the graphics drivers made the system quite unstable It would oops every 2-4 hours of normal use. XFS had this bad habit at the time (long since fixed, IIRC) of reverting files to HOURS old data after the reboot. A peer of this comment mentioned btrfs. Around the same time I played around with btrfs. For around a year I ran a couple of my company laptops on btrfs with zero problems. Then, after one set of updates, both of them completely corrupted their filesystems. I stopped using it, hoping that btrfs would fairly quickly stabilize, but have never ended up going back to it. I love the idea of btrfs, but am a bit gun-shy. I used Reiser extensively for map tile file-systems, largely because their idea of a "small file" was 1-2 orders of magnitude smaller than most other filesystems. This particular use case was full world tiles, and a lot of them were solid blue (oceans, you know), I want to say 37 bytes. My current gig puts tiles on XFS and it works great for that. ZFS is a filesystem that has never let me down. Performance can often be spotty, especially if you think you can use dedup, but I've had some pretty bad things happen to ZFS volumes and have never lost data on one of them. It's really the only place I'll put archive data. I've run a lot of backup servers using ZFS, even back in the days of zfsfuse, and it's just been a workhorse. Even in early days when zfsfuse had some pretty obscure bugs and would choke about every month or two. Kudos to the zfs+fuse developers, they quickly tracked down every issue I ran into once I had a stress test use case going. EXT has been a real workhorse too. I've lost data, rarely, but considering it's 99% of the filesystems I've run over the decades... Pretty good track record. Veritas vx on HP-UX was like a space age filesystem back in the mid '90s, but also was kind of painful. My memories of it are just vague and bad. I'd love to have HAMMER under Linux. :-) Ok, discuss!
- SubjectToChange 3y ago
  I’ve been running btrfs on my laptop and workstation since 2015 and haven’t had any issues. I get all the advanced FS functionality I want without needing to worry about kernel versions or any of the other hassle of setting up ZFS. In many ways ZFS feels like the last hurrah of “traditional” file systems. The fact that ZFS finds itself without genuine alternatives isn’t because it’s just that good, it’s because industry players have moved on. If all you need is a monster file server or a local data archive, sure, use ZFS.
  - abwizz 3y ago
    > file server or a local data archive thats all you want from a fs, isn't it? everything else is temporary data, living in memory.
- mrktf 3y ago
  With btrfs, interestingly enough never lost data, but got volume stuck to read only mode (fsck didn't help) multiple times over the years. On sad note qgroup qouta (it is fancy qouta - you can limit disk usage per subvolume basis https://btrfs.readthedocs.io/en/latest/btrfs-quota.html#performance-implications https://btrfs.readthedocs.io/en/latest/btrfs-quota.html#perf... sounds much more positive than real impact :/ although didn't test with latest kernel ) starts struggle when there are load using 5 users (using docker containers with jupyterhub) and when load goes near 10 docker instances, new docker container creation literally crawls - start taking minutes
pbh101 3y ago
In my experience enabling the finobt (free inode btree) setting made for a better experience under consistent capacity pressure. Context: CentOS 7, 7 TB RAID-10 SSD with ~100million files and ~4% churn rate per day, in production above 80% full for most of 5 years. Twice we experienced an XFS issue but xfs_repair got the FS back online and intact.
renewiltord 3y ago
Back in the early 2000s I had XFS zero out some files on power outage. Well known bug at the time. Fixed soon but retained the branding. What a time. People would recommend ReiserFS back then. Big fan club.
- adrian_b 3y ago
  Yes, it was very annoying and it happened even to files that were open only for reading, not for writing. For me, /etc/fstab was zeroed a couple of times, which lead to a bricked system (before 2000). Fortunately, this bug was eventually solved and I have been using XFS for more than twenty years on a great variety of hardware, with excellent results and no data loss whatsoever.
- anthk 3y ago
  Reiser and XFS were totally opposite in design. Reiser was born for being fast with tons of small files, suitable for your KDE desktop with lots of software/kparts/plugins installed. XFS was created to manage big multimedia files, such as uncut/unedited RAW photos and videos.

hestefisk 3y ago

Is there any case where one would choose this over ext4 or even zfs?

postmodest 3y ago
On spinning rust, it had guaranteed bandwidth and multiple I/O. On NVME? Not really.
- jeffbee 3y ago
  On Linux, XFS never offered guaranteed-rate I/O. It was really an IRIX feature that also required hardware support.
  - T3OU-736 3y ago
    Fun story. GRIO (Guaranteed Rate IO) XFS support was added to IRIX due to a request by a US Gov't agency. This agency was doing electronic signal collection with IRIX on SGI HW, and wanted to a way to guarantee that whatever their sensors captured would be written to the storage media, no matter what else is going on at the OS level.
phsau 3y ago
It's workload dependent; XFS performs better for our database needs.
- SkyMarshal 3y ago
  Compared to Ext4 or ZFS?
  - abwizz 3y ago
    i'd rekon cow fs performs somewhat poorly with db workload, so, both
brucethemoose2 3y ago
zfs and xfs is apples to oranges. One is simple and low overhead, the other is stuffed with features and options. ext4 gives me questionable performance with writing a huge amount (like 50k+) of small files, but other than that its great.

jeffbee 3y ago

I find the whole concept slightly scary. Some reasons for an online filesystem consistency check to fail are that the hardware has lied to the operating system about the success of some past operation, the filesystem has gone down a bad road and written garbage to a storage device, or the checker and the filesystem are inconsistent, with uncertainty about which side is correct. Under all those conditions I think I want my system to halt and log. Continuing seems like a real bad idea. There are fault-tolerant ways of coding operating systems and filesystem but it does not seem like those ways are practiced by Linux.