6 ms·

The XFS-as-default change is surprising. Does anyone know what drove this? I don't keep up with XFS development on Linux, so I don't understand why it's a bette

by gnoway 12y ago

The XFS-as-default change is surprising. Does anyone know what drove this? I don't keep up with XFS development on Linux, so I don't understand why it's a better choice vs. EXT4 for RHEL7 than it might have been for RHEL6. edit: cleanup wording.

ramLlama 12y ago
My understanding is that XFS is now default because it has far better support than ext4 for large (hundreds of TB to PB) volumes. In addition, RedHat employs many core XFS members. With the quashing of the slow metadata performance (http://lwn.net/Articles/476263/ http://lwn.net/Articles/476263/), XFS seems to be all-round just as good as ext4 but with more future-proofiness for large volumes. Keep in mind that RHEL releases are supported for around a decade.
- JohnTHaller 12y ago
  The new release increases ext4 from 16TB to 50TB and xfs from 100TB to 500TB.
  - awda 12y ago
    Sure, but these are mostly just arbitrary caps. xfs performance at 50TB is supposed to be better than ext4 at 50TB. I don't know what xfs actually does over ext4, but I do know a little bit about ext4; on-disk it looks very similar to the ancient Unix FFS. xfs may use more scalable on-disk structures. Looks like XFS supports concurrency better (both on the request side, and on the kernel<->backing store side).
    - colechristensen 12y ago
      Back in the day, we used xfs instead of ext3 so that when something happened and there was a bad shutdown, our samba servers weren't stuck in fsck for an hour (or a day).
gtirloni 12y ago
It looks like the enterprise-y features and agreement with partners mostly drove it, but it's unclear. Red Hat doesn't focus on EXT4 bugs but on selling XFS features. This presentation should be interesting to answer your question: http://rhsummit.files.wordpress.com/2014/04/rwheeler_thursday_0945_rhel7_beta_file_systems.pdf http://rhsummit.files.wordpress.com/2014/04/rwheeler_thursda...
nailer 12y ago
XFS has been stable on Linux for over a decade (since SGI Linux, which was RHEL with XFS, in the early 2000s). btrfs while having some design advantages over XFS and ZFS is still too new for most of Red Hat's customers.
- mbreese 12y ago
  I'm one of those people who had to deal with an XFS corruption about ten years ago, and the recovery process was messy - to say the least. At least with ext4, the recovery is easier to manage. I'm still scarred by it. This actually makes me happier with my decision to use FreeBSD and ZFS for a recent file server. I'll still use RHEL/CentOS for client nodes, but they'll all be using the ZFS export for storage.
  - 4ad 12y ago
    10 years ago XFS on Linux was nowhere near as reliable as XFS on IRIX, plus it was missing tools that were available in IRIX. I haven't used it since, but I would expect RedHat to have invested in correcting these problems over the years (I hope I'm not expecting too much). Of course your point about ZFS still stands.
  - Florin_Andrei 12y ago
    I had the same experience over 10 years ago - and I was actually at SGI back then. But that was then. This is now. I'm curious how things have changed meanwhile. I'm going to give this thing a spin. I also remember the massive performance and effortless delivery of XFS under concurrent access, while using large files. That was fantastic. I would expect that's still the case now.
  - rodgerd 12y ago
    Eight years ago Joyent suffered high-profile, irretrievable data corruption with ZFS. By your own metric I'm surprised you're using that.
    - mbreese 12y ago
      Of course I'm going to have a bad taste in my mouth for XFS. Because it happened to me. At the time is was particularly frustrating because XFS was claimed to be very stable and should be used over ext3(? I don't remember if ext4 was out yet). Our group ended up moving all of those machines over to ext3/4 and we were quite happy with the results. I remember when Joyent had their problem, and it made me hedge on ZFS for quite a while as well. Some time working with both Solaris and FreeBSD implementations in production was enough to change my mind. I'm sure that XFS now-a-days is much more stable - Redhat wouldn't make it the default filesystem if it wasn't. Just because I had a bad experience with it many years ago doesn't mean that I won't use it, but I will evaluate if I need to. But, I honestly wonder how much longer it will be the default with Btrfs coming along quickly.
conception 12y ago
-Disclaimer: A Guess- Since BTRFS isn't up, they employ XFS people and they use XFS in their Gluster implementation, it's easier for them to test/support it and push Redhat Enterprise Storage Server out to folks. I wouldn't be surprised if we see XFS for the default on Ceph soon as well.
rwmj 12y ago
The lifecycle of RHEL is minimum 10 years (longer for some customers). In 10 years time, disks will be considerably larger than what ext4 is comfortable with. I know that ext4 claims to support large disks, but only XFS is actually tested at these levels and known to work properly.
- sliverstorm 12y ago
  Not only tested and known to work, but tested and known to work for many years now! I first learned of & deployed XFS for volumes of hundreds of TB back in 2009? 2010?
  - mixmastamyk 12y ago
    We used XFS at a VFX studio, circa '97.
avar 12y ago
EXT4 cannot be resized while mounted. This is the main reason we use XFS instead of EXT4 where I work, we currently mostly use RHEL 6. This matters a lot for e.g. database masters.
- rwmj 12y ago
  ext4 can be resized while mounted (using resize2fs). It can only be extended, not shrunk however.
- pilif 12y ago
  This is surprising to me. I have resized many ext4 partitions while mounted (as / even). You can't shrink them. Yes. But you can grow them. I don't think shrinking partitions is a common operation and IMHO that alone wouldn't warrant a switch of the default file system.
  - vacri 12y ago
    Unless it's been changed in the past couple of years, XFS can't be shrunk at all.
otterley 12y ago
ext4 has some pretty sharp edge cases in terms of performance degradation. For example, readdir() performance is unacceptable in directories that contain lots of dentries (on the order of 100,000s) -- or that ever once did. (Directory read performance was just as bad even with 5 dentries after deleting the rest.) We had to switch to XFS for just that reason.
- mitchty 12y ago
  Actually that is a libc issue. It only reads in 32k of directory entries at a time. If you compile your own lame ls implementation and read in say a 1 meg buffer vs 1024 bytes at a time, you can print out directories with over 4 million entries in about a second versus 20ish hours with 1024 byte buffers (yeah I timed it). I'm not saying ext4 doesn't have issues, just that its not as simple as it seems on the surface.
  - otterley 12y ago
    If it were a libc issue, the problem would also occur with XFS. The reason we know it's a filesystem issue is that on ext4, readdir() still takes an eternity even after we delete all the files in the subdirectory.
    - jfindley 12y ago
      Yeah, because ext4 never cleans up the dentry table after it's been made. This is a pain. Unfortunately XFS also sucks for handling larger numbers of files - at the hundreds of millions or larger scale, you're largely metadata bound - and XFS is very slow at this for a number of reasons. RH did an interesting study a while back [0] on storing 1 billion files, which came out in favor of ext4. If you're able to work around the lack of dentry cleanup, I still think ext4 is probably the best choice here (ideally with the journal on a small SSD) - assuming you can't store them in a database or something more appropriate like mogileFS [1] [0]: http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf http://www.redhat.com/summit/2011/presentations/summit/decod... [1]: https://code.google.com/p/mogilefs/wiki/Start https://code.google.com/p/mogilefs/wiki/Start
jlawer 12y ago
Back at the RHEL6 launch (working for Red Hat Support) scuttlebutt around Red Hat was that they weren't confident they had the resources to provide full support for all clients for XFS. At that time it was offered as an add-on for a substantial amount of money. Considering the kind of case work that was being dealt with I don't think this was unreasonable. There was at least 1 time I needed to pull the first few mb of the disk for investigate a corruption issue. I believe the XFS devs (or at least the few I spoke to from the Australian team) all believed XFS was more then ready to replace EXT and should have replaced XFS. I suspect now that EXT4 is showing its age with massive filesystem sizes, that Red Hat is faced with 2 complex file systems. XFS is proven, and reliable. BtrFS is feature rich, evolving, but lacking a proven history with the diversity of systems that Red Hat need to support. Given that, I think going XFS was the expected move by Red Hat... conservative and incremental.
uulbiy 12y ago
Although you got some pretty good answers there is one point that is not covered: parallel I/O. This is especially important for virtualization hosts as it improves performance significantly.

by gnoway 12y ago

ramLlama 12y ago

My understanding is that XFS is now default because it has far better support than ext4 for large (hundreds of TB to PB) volumes. In addition, RedHat employs many core XFS members. With the quashing of the slow metadata performance (http://lwn.net/Articles/476263/ http://lwn.net/Articles/476263/), XFS seems to be all-round just as good as ext4 but with more future-proofiness for large volumes. Keep in mind that RHEL releases are supported for around a decade.

JohnTHaller 12y ago
The new release increases ext4 from 16TB to 50TB and xfs from 100TB to 500TB.
- awda 12y ago
  Sure, but these are mostly just arbitrary caps. xfs performance at 50TB is supposed to be better than ext4 at 50TB. I don't know what xfs actually does over ext4, but I do know a little bit about ext4; on-disk it looks very similar to the ancient Unix FFS. xfs may use more scalable on-disk structures. Looks like XFS supports concurrency better (both on the request side, and on the kernel<->backing store side).
  - colechristensen 12y ago
    Back in the day, we used xfs instead of ext3 so that when something happened and there was a bad shutdown, our samba servers weren't stuck in fsck for an hour (or a day).

gtirloni 12y ago

It looks like the enterprise-y features and agreement with partners mostly drove it, but it's unclear. Red Hat doesn't focus on EXT4 bugs but on selling XFS features. This presentation should be interesting to answer your question: http://rhsummit.files.wordpress.com/2014/04/rwheeler_thursday_0945_rhel7_beta_file_systems.pdf http://rhsummit.files.wordpress.com/2014/04/rwheeler_thursda...

nailer 12y ago

XFS has been stable on Linux for over a decade (since SGI Linux, which was RHEL with XFS, in the early 2000s). btrfs while having some design advantages over XFS and ZFS is still too new for most of Red Hat's customers.

mbreese 12y ago
I'm one of those people who had to deal with an XFS corruption about ten years ago, and the recovery process was messy - to say the least. At least with ext4, the recovery is easier to manage. I'm still scarred by it. This actually makes me happier with my decision to use FreeBSD and ZFS for a recent file server. I'll still use RHEL/CentOS for client nodes, but they'll all be using the ZFS export for storage.
- 4ad 12y ago
  10 years ago XFS on Linux was nowhere near as reliable as XFS on IRIX, plus it was missing tools that were available in IRIX. I haven't used it since, but I would expect RedHat to have invested in correcting these problems over the years (I hope I'm not expecting too much). Of course your point about ZFS still stands.
- Florin_Andrei 12y ago
  I had the same experience over 10 years ago - and I was actually at SGI back then. But that was then. This is now. I'm curious how things have changed meanwhile. I'm going to give this thing a spin. I also remember the massive performance and effortless delivery of XFS under concurrent access, while using large files. That was fantastic. I would expect that's still the case now.
- rodgerd 12y ago
  Eight years ago Joyent suffered high-profile, irretrievable data corruption with ZFS. By your own metric I'm surprised you're using that.
  - mbreese 12y ago
    Of course I'm going to have a bad taste in my mouth for XFS. Because it happened to me. At the time is was particularly frustrating because XFS was claimed to be very stable and should be used over ext3(? I don't remember if ext4 was out yet). Our group ended up moving all of those machines over to ext3/4 and we were quite happy with the results. I remember when Joyent had their problem, and it made me hedge on ZFS for quite a while as well. Some time working with both Solaris and FreeBSD implementations in production was enough to change my mind. I'm sure that XFS now-a-days is much more stable - Redhat wouldn't make it the default filesystem if it wasn't. Just because I had a bad experience with it many years ago doesn't mean that I won't use it, but I will evaluate if I need to. But, I honestly wonder how much longer it will be the default with Btrfs coming along quickly.

conception 12y ago

-Disclaimer: A Guess- Since BTRFS isn't up, they employ XFS people and they use XFS in their Gluster implementation, it's easier for them to test/support it and push Redhat Enterprise Storage Server out to folks. I wouldn't be surprised if we see XFS for the default on Ceph soon as well.

rwmj 12y ago

The lifecycle of RHEL is minimum 10 years (longer for some customers). In 10 years time, disks will be considerably larger than what ext4 is comfortable with. I know that ext4 claims to support large disks, but only XFS is actually tested at these levels and known to work properly.

sliverstorm 12y ago
Not only tested and known to work, but tested and known to work for many years now! I first learned of & deployed XFS for volumes of hundreds of TB back in 2009? 2010?
- mixmastamyk 12y ago
  We used XFS at a VFX studio, circa '97.

avar 12y ago

EXT4 cannot be resized while mounted. This is the main reason we use XFS instead of EXT4 where I work, we currently mostly use RHEL 6. This matters a lot for e.g. database masters.

rwmj 12y ago
ext4 can be resized while mounted (using resize2fs). It can only be extended, not shrunk however.
pilif 12y ago
This is surprising to me. I have resized many ext4 partitions while mounted (as / even). You can't shrink them. Yes. But you can grow them. I don't think shrinking partitions is a common operation and IMHO that alone wouldn't warrant a switch of the default file system.
- vacri 12y ago
  Unless it's been changed in the past couple of years, XFS can't be shrunk at all.

otterley 12y ago

ext4 has some pretty sharp edge cases in terms of performance degradation. For example, readdir() performance is unacceptable in directories that contain lots of dentries (on the order of 100,000s) -- or that ever once did. (Directory read performance was just as bad even with 5 dentries after deleting the rest.) We had to switch to XFS for just that reason.

mitchty 12y ago
Actually that is a libc issue. It only reads in 32k of directory entries at a time. If you compile your own lame ls implementation and read in say a 1 meg buffer vs 1024 bytes at a time, you can print out directories with over 4 million entries in about a second versus 20ish hours with 1024 byte buffers (yeah I timed it). I'm not saying ext4 doesn't have issues, just that its not as simple as it seems on the surface.
- otterley 12y ago
  If it were a libc issue, the problem would also occur with XFS. The reason we know it's a filesystem issue is that on ext4, readdir() still takes an eternity even after we delete all the files in the subdirectory.
  - jfindley 12y ago
    Yeah, because ext4 never cleans up the dentry table after it's been made. This is a pain. Unfortunately XFS also sucks for handling larger numbers of files - at the hundreds of millions or larger scale, you're largely metadata bound - and XFS is very slow at this for a number of reasons. RH did an interesting study a while back [0] on storing 1 billion files, which came out in favor of ext4. If you're able to work around the lack of dentry cleanup, I still think ext4 is probably the best choice here (ideally with the journal on a small SSD) - assuming you can't store them in a database or something more appropriate like mogileFS [1] [0]: http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf http://www.redhat.com/summit/2011/presentations/summit/decod... [1]: https://code.google.com/p/mogilefs/wiki/Start https://code.google.com/p/mogilefs/wiki/Start

jlawer 12y ago

Back at the RHEL6 launch (working for Red Hat Support) scuttlebutt around Red Hat was that they weren't confident they had the resources to provide full support for all clients for XFS. At that time it was offered as an add-on for a substantial amount of money. Considering the kind of case work that was being dealt with I don't think this was unreasonable. There was at least 1 time I needed to pull the first few mb of the disk for investigate a corruption issue. I believe the XFS devs (or at least the few I spoke to from the Australian team) all believed XFS was more then ready to replace EXT and should have replaced XFS. I suspect now that EXT4 is showing its age with massive filesystem sizes, that Red Hat is faced with 2 complex file systems. XFS is proven, and reliable. BtrFS is feature rich, evolving, but lacking a proven history with the diversity of systems that Red Hat need to support. Given that, I think going XFS was the expected move by Red Hat... conservative and incremental.

uulbiy 12y ago

Although you got some pretty good answers there is one point that is not covered: parallel I/O. This is especially important for virtualization hosts as it improves performance significantly.