7 ms·

XFS: the filesystem of the future?

70 points by dmm 15y ago

colanderman 15y ago
I want to throw in another XFS data-eating anecdote. When I used it, I found it had a tendency to zero out files which were open for writing during a system crash. I've been using JFS now for 6 years or so and have never had files go missing on me. JFS routinely takes second place to XFS in various benchmarks but still far outperforms ext3 and the like.
- RexRollman 15y ago
  I always thought that was interesting because on my computer, JFS feels faster than XFS does (especially when using Pacman, on Arch Linux, which creates lots of small files). Personally, I have never lost data with XFS or JFS.
- gcp 15y ago
  When I used it, I found it had a tendency to zero out files which were open for writing during a system crash. This was pretty much a FAQ and very much by design. See for example: http://madduck.net/blog/2006.08.11:xfs-zeroes/ http://madduck.net/blog/2006.08.11:xfs-zeroes/
  - deleted 15y ago
    [deleted]
  - colanderman 15y ago
    Interesting. I don't buy the security argument though. If, after a crash, the file size is larger than the actual extents, why not just truncate the file size to match the extents?
    - gcp 15y ago
      Because there's no guarantees the extents actually point to the file data.
- Zak 15y ago
  I had this happen once too. As I recall, the machine lost power after a cached write and before the data was actually written to disk. The file was zeroed and all its former contents lost. My friend, who had borrowed the computer to write a paper and lost hours of work was very upset. I switched all my XFS filesystems to something else that day.
owenmarshall 15y ago
It's somewhat amusing that the filesystem of the future was developed by SGI in 1994 ;) I've had little to complain about when I've used XFS. It seems like a very good file system, and has handled the large datasets I've thrown at it without any problems.
- sespindola 15y ago
  As mentioned on the article's comments, when I used XFS about 8 years ago, I found exceptionally prone to data corruption. It remains the only filesystem that I couldn't at least partially recover. For a good mix of speed, reliability and huge data capacity on Linux, I stick to ReiserFS, at least until btrfs becomes more stable.
  - asabil 15y ago
    I had the same experience concerning XFS, I experienced repeated data loss on power failure about 4 years ago. These days I just stick with ext4
  - simcop2387 15y ago
    I've actually been moving off of ReiserFS myself. Mostly for some of the known bugs and unfixable problems with it relating to storing another ReiserFS image on it and then trying to recover data from a crash. I've run into that more than once and that made me move to something that was a little more sane during recovery (ext3 at the time, now ext4).
  - owenmarshall 15y ago
    8 years ago would make you an extremely early adopter of the Linux port. XFS has seen significant development since then. I don't think experiences based on eight year old software hold much water. XFS has adopted some nasty stigma of being a filesystem that eats your data. But it seems that for every user that complains about data loss, another does not.
    - Terretta 15y ago
      > But it seems that for every user that complains about data loss, another does not. So, a coin toss then? You're not helping!
waffle_ss 15y ago
I've had good experiences with XFS on my home file server. The only things to keep in mind are that once you cannot shrink the filesystem once it's created, and that fsck doesn't run when booted (you can write an init script to run xfs_repair if you like). Also, my fragmentation often gets quite high - over 90% - but it doesn't seem to really affect performance.
- RexRollman 15y ago
  XFS filesystems can be defragmented. The tool is called xfs_fsr http://linux.die.net/man/8/xfs_fsr http://linux.die.net/man/8/xfs_fsr.
  - waffle_ss 15y ago
    Yes, I have a cron job that does that.
    - RexRollman 15y ago
      And yet you are getting 90% defragmentation? I wonder why that is happening to you.
kijin 15y ago
> For I/O-heavy workloads with a lot of metadata changes - unpacking a tarball was given as an example - Dave said that ext4 could be 20-50 times faster than XFS. A while ago, I read somewhere that XFS is good for a small number of large files, whereas ReiserFS is good for a large number of small files. I don't know whether that's true anymore if it ever was; but perhaps extracting a tarball with thousands of small files in it is not the best way to bring out XFS's strength? By the way, when are we expecting the "experimental" label to be taken off of btrfs?
- noahdesu 15y ago
  > By the way, when are we expecting the "experimental" label to be taken off of btrfs? I believe distributions are waiting on an fsck for btrfs.
  - sneak 15y ago
    Hahaha. That's pretty much /thread, innit?
    - eru 15y ago
      What are you trying to say?
  - waitwhat 15y ago
    "It looks like this will finally be released by the middle of February." -- http://www.phoronix.com/vr.php?view=MTA0Njk http://www.phoronix.com/vr.php?view=MTA0Njk
- Tuna-Fish 15y ago
  > A while ago, I read somewhere that XFS is good for a small number of large files, whereas ReiserFS is good for a large number of small files. I don't know whether that's true anymore if it ever was It used to be true. XFS has for a long time been the very fastest real filesystem available when it comes to moving data, and one of the slowest ones in real-world use when it comes to changing metadata. Untarring a kernel tarball a second time (on top of the first) is pretty much the canonical benchmark for metadata operations, and the one where XFS used to be at it's absolute worst. XFS is in the news right now exactly because they have fixed that deficiency. It's still faster than ext*, zfs and btrfs in pretty much everything else, but now it's comparable to them when modifying metadata. So this is exactly the right benchmark to use, because it's the one where there has been recent change. Video of Dave Chinner presenting this at linux.conf.au: https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw
yusufg 15y ago
Is XFS available in stock RHEL 6.2 ? or does it require an additional purchase of the Scalable File System Add-On http://www.redhat.com/products/enterprise-linux-add-ons/file-systems/ http://www.redhat.com/products/enterprise-linux-add-ons/file... Redhat doesn't list prices of its add-ons and its generally onerus if one has to call sales to find the price of a filesystem Reminds me of Veritas VxFS
- antoncohen 15y ago
  I think it's in the base RHEL 6, try 'yum list xfsprogs', if it's there you have XFS. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/xfsmain.html http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6...
calloc 15y ago
How does XFS stack up with ZFS when it comes to speed, reliability and protecting my data?
tytso 15y ago
All open source developers want users. Users means more bug reports and often more developers to contribute to the prokect. In addition, the greater relevance also means it's more likely they will be able to get companies to be able to pay for them to do what they do, conferences to invite them to talk about their projects, etc. (It's for this reason that when the IMHO over-aggressive enforcement of busybox license forcing people who get sued to release code on code unrelated to busybox has caused FUD amongst Linux embedded companies, the emergency of a replacement toybox project which avoids these issues has caused so much dismay amongst the busybox development community.) Given that Dave Chinner is one of the primary developers of XFS, it's not surprising that he wants to promote XFS. And to be fair, XFS has not gotten as much attention as it has perhaps deserved based on technical considerations alone (as has other perfectly capable filesystems at the their time, such as JFS) and that's no doubt frustrated him. In addition, the work he has done to improve XFS removes one of the significant performance bottlenecks often seen by desktop users and developers, and he should be saluted for that. That being said, it's also true that in many cases the file system is not the bottleneck, and so other issues that aren't tightly focused on performance (i.e., the quality of the userspace tools such as e2fsck and debugfs for ext4, and their equivalent or lack thereof in other file systems), familiarity by sysadmins, data recovery services, etc., ease of upgrading of existing large, production file systems, and so on. In addition, it's dangerous to draw conclusions from a single microbenchmark such as fs_mark alone. It's not common that you have workloads which create thousands and thousands of small (< 64k) files in parallel across lots of CPU cores at the same time, on the same file system. So using this this benchmark alone to say that file system X is more scalable than file system Y is just not going to tell the whole story. Personally, I like to use microbenchmarks as a tool for improving a file system, and not as an argument to try to get people to switch from one file system to another. Unless someone's use case is exactly mirrored by the microbenchmark, I personally find this approach to be a little dishonest. I will say that at the moment, many of the developers who have been working on ext4 are employed by companies who are using ext4 as part of a cloud data storage stack. This is why there has been changes such as no journal mode (which is great when you have consistency guarantees being provided by a cluster file system above you, since they have provide the file even if an entire server's power supply has exploded), and good performance when under severe memory pressure (funny how most benchmarks are done when the only thing running on the server is the benchmark, so there is no competition for the CPU and for system memory --- XFS in particular have proven to be a memory hog, and others have noted severe performance degredations and in some cases stability problems, under memory pressure; not a problem on a stand-alone file server, but not so great if you are also trying to run VM's or other applications using the file system on the same machine). Arguably some of these improvements don't mean as much for desktop users, although I believe some of the performance enhancements we've made have also trickled down to help the desktop. XFS, in contrast, has been focusing a lot of attention on the desktop use case, and they've traditionally owned the big stream writes, HPC workloads using huge servers, huge memory, and huge RAID arrays. It's good that XFS has made these improvements, and I salute them for it. But to state that these two workloads are the only ones which are important, and therefore they are the file system of the future, may be overstating matters --- with all due respect to Dave and his many years of experience working on XFS.
yason 15y ago
A file system for generic purpose has only one primary feature: "Don't lose data. Ever." If I pop the SATA cable off my drive while writing, the filesystem should later remount in 100% uncorrupted condition with whatever data it had time to write to the disk. I don't want to run fsck or use debugfs to recover from a hairy state. Backups or RAID-1 take care of physical failures of the disk. Beyond avoiding data loss, anything else is ultimately secondary. Speed is nice but near-average filesystem performance is all right for a HDD. SSD gives you more speed. Totally abysmal speed might be a reason to switch filesystems—though, I'd still take the abysmal speed if it saved me my data and the faster filesystem wouldn't. Due to this conservative approach, switching filesystems is really hard. I've been using ReiserFS since 2000 or so because it hasn't failed me once. I've had HDDs going slowly broken and lost some individual files until I cloned the old disk to a new one. But I've never had to fsck, defrag, recover, or anything my ReiserFS partitions. Never. It's getting harder and harder to switch. A conservative alternative would be ext3 but ext3 has lost me data on another computer. I have some interest in btrfs but I probably won't switch until I have to. XFS would be very interesting but not just worth the risk because ReiserFS is good enough.
- cbsmith 15y ago
  It must seem very strange to you that most filesystems don't do data journaling and that they have options which increase the risk of data integrity problems.
- gcb 15y ago
  you are lucky because if you ever need to use data rescue, well, there's nothing that works well with reiserfs. that being said, i'm a reiserFS fan too. been using it for almost a decade. two months ago, installed debian sid on my box, and ext4 ate my data after the third boot! kid you not. tried to find the cause to submit a report, but run out of time, just reinstalled with ext3 for now.
aidenn0 15y ago
I'll have to give XFS a look again. We use subversion at my work and an "svn up" took 4-8x longer on XFS than on reiser3.6; it was insanely faster for just about everything else though. It definitely is not great for sudden power-failure, but really if you're not using a laptop, buy a UPS, it's not that expensive.
jamesfmilne 15y ago
At our company, we have been shipping XFS as the filesystem for our products for over 8 years. We've shipped many petabytes of storage in that time, all of which is being heavily hammered every day in feature film & TV post-production. We've found XFS to be a very robust filesystem, and any problems have generally been traceable to modifications we've made to tools like xfs_fsr. In general it's been considerably more frequent for people to end up in a panic after making a mistake replacing a disk in a 3ware array and deleting a RAID unit! Luckily our resident RAID ninja has been able to get the majority of people out of that hole. :-)

70 points by dmm 15y ago

I want to throw in another XFS data-eating anecdote. When I used it, I found it had a tendency to zero out files which were open for writing during a system crash. I've been using JFS now for 6 years or so and have never had files go missing on me. JFS routinely takes second place to XFS in various benchmarks but still far outperforms ext3 and the like.

RexRollman 15y ago
I always thought that was interesting because on my computer, JFS feels faster than XFS does (especially when using Pacman, on Arch Linux, which creates lots of small files). Personally, I have never lost data with XFS or JFS.
gcp 15y ago
When I used it, I found it had a tendency to zero out files which were open for writing during a system crash. This was pretty much a FAQ and very much by design. See for example: http://madduck.net/blog/2006.08.11:xfs-zeroes/ http://madduck.net/blog/2006.08.11:xfs-zeroes/
- deleted 15y ago
  [deleted]
- colanderman 15y ago
  Interesting. I don't buy the security argument though. If, after a crash, the file size is larger than the actual extents, why not just truncate the file size to match the extents?
  - gcp 15y ago
    Because there's no guarantees the extents actually point to the file data.
Zak 15y ago
I had this happen once too. As I recall, the machine lost power after a cached write and before the data was actually written to disk. The file was zeroed and all its former contents lost. My friend, who had borrowed the computer to write a paper and lost hours of work was very upset. I switched all my XFS filesystems to something else that day.

owenmarshall 15y ago

It's somewhat amusing that the filesystem of the future was developed by SGI in 1994 ;) I've had little to complain about when I've used XFS. It seems like a very good file system, and has handled the large datasets I've thrown at it without any problems.

sespindola 15y ago
As mentioned on the article's comments, when I used XFS about 8 years ago, I found exceptionally prone to data corruption. It remains the only filesystem that I couldn't at least partially recover. For a good mix of speed, reliability and huge data capacity on Linux, I stick to ReiserFS, at least until btrfs becomes more stable.
- asabil 15y ago
  I had the same experience concerning XFS, I experienced repeated data loss on power failure about 4 years ago. These days I just stick with ext4
- simcop2387 15y ago
  I've actually been moving off of ReiserFS myself. Mostly for some of the known bugs and unfixable problems with it relating to storing another ReiserFS image on it and then trying to recover data from a crash. I've run into that more than once and that made me move to something that was a little more sane during recovery (ext3 at the time, now ext4).
- owenmarshall 15y ago
  8 years ago would make you an extremely early adopter of the Linux port. XFS has seen significant development since then. I don't think experiences based on eight year old software hold much water. XFS has adopted some nasty stigma of being a filesystem that eats your data. But it seems that for every user that complains about data loss, another does not.
  - Terretta 15y ago
    > But it seems that for every user that complains about data loss, another does not. So, a coin toss then? You're not helping!

waffle_ss 15y ago

I've had good experiences with XFS on my home file server. The only things to keep in mind are that once you cannot shrink the filesystem once it's created, and that fsck doesn't run when booted (you can write an init script to run xfs_repair if you like). Also, my fragmentation often gets quite high - over 90% - but it doesn't seem to really affect performance.

RexRollman 15y ago
XFS filesystems can be defragmented. The tool is called xfs_fsr http://linux.die.net/man/8/xfs_fsr http://linux.die.net/man/8/xfs_fsr.
- waffle_ss 15y ago
  Yes, I have a cron job that does that.
  - RexRollman 15y ago
    And yet you are getting 90% defragmentation? I wonder why that is happening to you.

kijin 15y ago

> For I/O-heavy workloads with a lot of metadata changes - unpacking a tarball was given as an example - Dave said that ext4 could be 20-50 times faster than XFS. A while ago, I read somewhere that XFS is good for a small number of large files, whereas ReiserFS is good for a large number of small files. I don't know whether that's true anymore if it ever was; but perhaps extracting a tarball with thousands of small files in it is not the best way to bring out XFS's strength? By the way, when are we expecting the "experimental" label to be taken off of btrfs?

noahdesu 15y ago
> By the way, when are we expecting the "experimental" label to be taken off of btrfs? I believe distributions are waiting on an fsck for btrfs.
- sneak 15y ago
  Hahaha. That's pretty much /thread, innit?
  - eru 15y ago
    What are you trying to say?
- waitwhat 15y ago
  "It looks like this will finally be released by the middle of February." -- http://www.phoronix.com/vr.php?view=MTA0Njk http://www.phoronix.com/vr.php?view=MTA0Njk
Tuna-Fish 15y ago
> A while ago, I read somewhere that XFS is good for a small number of large files, whereas ReiserFS is good for a large number of small files. I don't know whether that's true anymore if it ever was It used to be true. XFS has for a long time been the very fastest real filesystem available when it comes to moving data, and one of the slowest ones in real-world use when it comes to changing metadata. Untarring a kernel tarball a second time (on top of the first) is pretty much the canonical benchmark for metadata operations, and the one where XFS used to be at it's absolute worst. XFS is in the news right now exactly because they have fixed that deficiency. It's still faster than ext*, zfs and btrfs in pretty much everything else, but now it's comparable to them when modifying metadata. So this is exactly the right benchmark to use, because it's the one where there has been recent change. Video of Dave Chinner presenting this at linux.conf.au: https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw

yusufg 15y ago

Is XFS available in stock RHEL 6.2 ? or does it require an additional purchase of the Scalable File System Add-On http://www.redhat.com/products/enterprise-linux-add-ons/file-systems/ http://www.redhat.com/products/enterprise-linux-add-ons/file... Redhat doesn't list prices of its add-ons and its generally onerus if one has to call sales to find the price of a filesystem Reminds me of Veritas VxFS

antoncohen 15y ago
I think it's in the base RHEL 6, try 'yum list xfsprogs', if it's there you have XFS. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/xfsmain.html http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6...

calloc 15y ago

How does XFS stack up with ZFS when it comes to speed, reliability and protecting my data?

tytso 15y ago

All open source developers want users. Users means more bug reports and often more developers to contribute to the prokect. In addition, the greater relevance also means it's more likely they will be able to get companies to be able to pay for them to do what they do, conferences to invite them to talk about their projects, etc. (It's for this reason that when the IMHO over-aggressive enforcement of busybox license forcing people who get sued to release code on code unrelated to busybox has caused FUD amongst Linux embedded companies, the emergency of a replacement toybox project which avoids these issues has caused so much dismay amongst the busybox development community.) Given that Dave Chinner is one of the primary developers of XFS, it's not surprising that he wants to promote XFS. And to be fair, XFS has not gotten as much attention as it has perhaps deserved based on technical considerations alone (as has other perfectly capable filesystems at the their time, such as JFS) and that's no doubt frustrated him. In addition, the work he has done to improve XFS removes one of the significant performance bottlenecks often seen by desktop users and developers, and he should be saluted for that. That being said, it's also true that in many cases the file system is not the bottleneck, and so other issues that aren't tightly focused on performance (i.e., the quality of the userspace tools such as e2fsck and debugfs for ext4, and their equivalent or lack thereof in other file systems), familiarity by sysadmins, data recovery services, etc., ease of upgrading of existing large, production file systems, and so on. In addition, it's dangerous to draw conclusions from a single microbenchmark such as fs_mark alone. It's not common that you have workloads which create thousands and thousands of small (< 64k) files in parallel across lots of CPU cores at the same time, on the same file system. So using this this benchmark alone to say that file system X is more scalable than file system Y is just not going to tell the whole story. Personally, I like to use microbenchmarks as a tool for improving a file system, and not as an argument to try to get people to switch from one file system to another. Unless someone's use case is exactly mirrored by the microbenchmark, I personally find this approach to be a little dishonest. I will say that at the moment, many of the developers who have been working on ext4 are employed by companies who are using ext4 as part of a cloud data storage stack. This is why there has been changes such as no journal mode (which is great when you have consistency guarantees being provided by a cluster file system above you, since they have provide the file even if an entire server's power supply has exploded), and good performance when under severe memory pressure (funny how most benchmarks are done when the only thing running on the server is the benchmark, so there is no competition for the CPU and for system memory --- XFS in particular have proven to be a memory hog, and others have noted severe performance degredations and in some cases stability problems, under memory pressure; not a problem on a stand-alone file server, but not so great if you are also trying to run VM's or other applications using the file system on the same machine). Arguably some of these improvements don't mean as much for desktop users, although I believe some of the performance enhancements we've made have also trickled down to help the desktop. XFS, in contrast, has been focusing a lot of attention on the desktop use case, and they've traditionally owned the big stream writes, HPC workloads using huge servers, huge memory, and huge RAID arrays. It's good that XFS has made these improvements, and I salute them for it. But to state that these two workloads are the only ones which are important, and therefore they are the file system of the future, may be overstating matters --- with all due respect to Dave and his many years of experience working on XFS.

yason 15y ago

A file system for generic purpose has only one primary feature: "Don't lose data. Ever." If I pop the SATA cable off my drive while writing, the filesystem should later remount in 100% uncorrupted condition with whatever data it had time to write to the disk. I don't want to run fsck or use debugfs to recover from a hairy state. Backups or RAID-1 take care of physical failures of the disk. Beyond avoiding data loss, anything else is ultimately secondary. Speed is nice but near-average filesystem performance is all right for a HDD. SSD gives you more speed. Totally abysmal speed might be a reason to switch filesystems—though, I'd still take the abysmal speed if it saved me my data and the faster filesystem wouldn't. Due to this conservative approach, switching filesystems is really hard. I've been using ReiserFS since 2000 or so because it hasn't failed me once. I've had HDDs going slowly broken and lost some individual files until I cloned the old disk to a new one. But I've never had to fsck, defrag, recover, or anything my ReiserFS partitions. Never. It's getting harder and harder to switch. A conservative alternative would be ext3 but ext3 has lost me data on another computer. I have some interest in btrfs but I probably won't switch until I have to. XFS would be very interesting but not just worth the risk because ReiserFS is good enough.

cbsmith 15y ago
It must seem very strange to you that most filesystems don't do data journaling and that they have options which increase the risk of data integrity problems.
gcb 15y ago
you are lucky because if you ever need to use data rescue, well, there's nothing that works well with reiserfs. that being said, i'm a reiserFS fan too. been using it for almost a decade. two months ago, installed debian sid on my box, and ext4 ate my data after the third boot! kid you not. tried to find the cause to submit a report, but run out of time, just reinstalled with ext3 for now.

aidenn0 15y ago

I'll have to give XFS a look again. We use subversion at my work and an "svn up" took 4-8x longer on XFS than on reiser3.6; it was insanely faster for just about everything else though. It definitely is not great for sudden power-failure, but really if you're not using a laptop, buy a UPS, it's not that expensive.

jamesfmilne 15y ago

At our company, we have been shipping XFS as the filesystem for our products for over 8 years. We've shipped many petabytes of storage in that time, all of which is being heavily hammered every day in feature film & TV post-production. We've found XFS to be a very robust filesystem, and any problems have generally been traceable to modifications we've made to tools like xfs_fsr. In general it's been considerably more frequent for people to end up in a panic after making a mistake replacing a disk in a 3ware array and deleting a RAID unit! Luckily our resident RAID ninja has been able to get the majority of people out of that hole. :-)