5 ms·

XFS, ext and per-inode mutexes

43 points by Garbage 15y ago

alexgartrell 15y ago
I'm starting at Facebook in about a week and a half and a big reason I chose Facebook over Google as an infrastructure guy was (somewhat ironically) their openness (in infrastructure/engineering practices). I much prefer the quick Facebook note to the occasional well-edited white paper.
- RexRollman 15y ago
  Good luck to you!!
asb 15y ago
Does anybody know the ext4 behaviour?
- rg3 15y ago
  Yes, ext4 uses inode mutexes by default like ext3. They're needed if you want to use a journal, for example. Open http://www.kernel.org/doc/Documentation/filesystems/ext4.txt http://www.kernel.org/doc/Documentation/filesystems/ext4.txt and look for dioread_lock for more information.
- viraptor 15y ago
  Also, does anyone know what does btrfs do for writes? Its .txt is considerably shorter than ext4's unfortunately. (I could only find "Btree lock contention: The btree locks, especially on the root block can be very hot. We need to improve this, especially in read mostly workloads." in the unclaimed projects which seems a bit related)
RexRollman 15y ago
I like both XFS and JFS filesystems. I have used both on and off with Arch Linux and I've never had a bit of problem with either. Some people don't fancy them because they are in maintenance mode now but, frankly, I would rather use a finished and maintained filesystem over something that is in constant development (unless the new filesystem in development provides something truly unique). I think the one thing I miss from BeOS (my all-time favorite OS along with Nextstep) was BeFS. It was nice to be able to create arbitrary file metadata and then have it indexed and searchable in real time. Does any Linux FS have this capability? I believe that XFS supports extended metadata but from what understand it is not something one can search on.
- danudey 15y ago
  You can't inherently search on it (i.e the filesystem doesn't manage its own index) but it would be trivial (with something like inotify or its successors) to write an app to monitor the filesystem for metadata changes and index them itself. This is how Spotlight works on OS X, with HFS+'s arbitrary metadata - the spotlight daemon watches for FS events and updates its index when a change occurs. I strongly suspect that this is actually what was happening in BeOS as well, at least on some level. It would be surprising to me if the filesystem itself was also what maintained the index; it seems like this would reduce write throughput if it had to update the indexes all the time as well.
- wmf 15y ago
  I think GNOME Tracker and Zeitgeist are going in this direction.
- wazoox 15y ago
  I wouldn't say XFS is in maintenance mode, it received important enhancements in the recent past, and significant performance gains.
uriel 15y ago
I used XFS for years, until I got tired of it semi-randomly overriding my files with zeros and then found out that apparently this was 'by design', after that I stick with ext3/4 that might not be as fast in some corner cases, but which are just as fast or more with my usage patterns, and which so far has been good at not corrupting any of my data.
- moe 15y ago
  semi-randomly overriding my files with zeros and then found out that apparently this was 'by design Can you elaborate on this? I'm using XFS in a few fairly large deployments and so far without any problems. Should I worry?
  - jwatzman 15y ago
    Someone please correct me if I'm wrong, but my understanding is that it's something approximately like this: In the event of an error and journal replay, the filesystem metadata can be recovered, but not necessarily the data itself. ext3 tries to keep your data intact, or at least as much of it as it can piece back together. XFS (and I think JFS), if they don't know for sure that all the data is intact, just zero the file. This is by design, on both sides. So in normal usage everything is fine; it's just a difference in error handling.
  - mycatverbs 15y ago
    I believe not, unless you are in the habit of losing power without getting a chance to unmount your filesystems cleanly.
  - wazoox 15y ago
    No, because XFS doesn't eat your data if you're using decent hardware with decent applications. See http://flamingspork.com/talks/2007/06/eat_my_data.odp http://flamingspork.com/talks/2007/06/eat_my_data.odp It's probably not the best choice for a basic desktop without UPS, though, or a laptop.
  - kl4m 15y ago
    XFS Zeroes the block instead of keeping the random/sensitive data that was already on the disk if the metadata necessary for that write was not commited before the failure. Ext leaves the data as is. If you want full protection against those situations instead of praying that it's still the right data or that it doesn't happen at all, you need a copy-on-write FS, or enable data=journal on ext3, at a massive performance cost.
gojomo 15y ago
Anyone know if btrfs would be more like XFS or ext3/4 in this regard?
spudlyo 15y ago
Mark didn't mention this in his article, but writes to the inode are only serialized in jfs/ext(2|3|4) if you are using direct I/O (i.e O_DIRECT) which for performance reasons MySQL ibdata files often are. This is why this problem does not often show up on filesystem benchmarks.

43 points by Garbage 15y ago

I'm starting at Facebook in about a week and a half and a big reason I chose Facebook over Google as an infrastructure guy was (somewhat ironically) their openness (in infrastructure/engineering practices). I much prefer the quick Facebook note to the occasional well-edited white paper.

RexRollman 15y ago
Good luck to you!!

asb 15y ago

Does anybody know the ext4 behaviour?

rg3 15y ago
Yes, ext4 uses inode mutexes by default like ext3. They're needed if you want to use a journal, for example. Open http://www.kernel.org/doc/Documentation/filesystems/ext4.txt http://www.kernel.org/doc/Documentation/filesystems/ext4.txt and look for dioread_lock for more information.
viraptor 15y ago
Also, does anyone know what does btrfs do for writes? Its .txt is considerably shorter than ext4's unfortunately. (I could only find "Btree lock contention: The btree locks, especially on the root block can be very hot. We need to improve this, especially in read mostly workloads." in the unclaimed projects which seems a bit related)

RexRollman 15y ago

I like both XFS and JFS filesystems. I have used both on and off with Arch Linux and I've never had a bit of problem with either. Some people don't fancy them because they are in maintenance mode now but, frankly, I would rather use a finished and maintained filesystem over something that is in constant development (unless the new filesystem in development provides something truly unique). I think the one thing I miss from BeOS (my all-time favorite OS along with Nextstep) was BeFS. It was nice to be able to create arbitrary file metadata and then have it indexed and searchable in real time. Does any Linux FS have this capability? I believe that XFS supports extended metadata but from what understand it is not something one can search on.

danudey 15y ago
You can't inherently search on it (i.e the filesystem doesn't manage its own index) but it would be trivial (with something like inotify or its successors) to write an app to monitor the filesystem for metadata changes and index them itself. This is how Spotlight works on OS X, with HFS+'s arbitrary metadata - the spotlight daemon watches for FS events and updates its index when a change occurs. I strongly suspect that this is actually what was happening in BeOS as well, at least on some level. It would be surprising to me if the filesystem itself was also what maintained the index; it seems like this would reduce write throughput if it had to update the indexes all the time as well.
wmf 15y ago
I think GNOME Tracker and Zeitgeist are going in this direction.
wazoox 15y ago
I wouldn't say XFS is in maintenance mode, it received important enhancements in the recent past, and significant performance gains.

uriel 15y ago

I used XFS for years, until I got tired of it semi-randomly overriding my files with zeros and then found out that apparently this was 'by design', after that I stick with ext3/4 that might not be as fast in some corner cases, but which are just as fast or more with my usage patterns, and which so far has been good at not corrupting any of my data.

moe 15y ago
semi-randomly overriding my files with zeros and then found out that apparently this was 'by design Can you elaborate on this? I'm using XFS in a few fairly large deployments and so far without any problems. Should I worry?
- jwatzman 15y ago
  Someone please correct me if I'm wrong, but my understanding is that it's something approximately like this: In the event of an error and journal replay, the filesystem metadata can be recovered, but not necessarily the data itself. ext3 tries to keep your data intact, or at least as much of it as it can piece back together. XFS (and I think JFS), if they don't know for sure that all the data is intact, just zero the file. This is by design, on both sides. So in normal usage everything is fine; it's just a difference in error handling.
- mycatverbs 15y ago
  I believe not, unless you are in the habit of losing power without getting a chance to unmount your filesystems cleanly.
- wazoox 15y ago
  No, because XFS doesn't eat your data if you're using decent hardware with decent applications. See http://flamingspork.com/talks/2007/06/eat_my_data.odp http://flamingspork.com/talks/2007/06/eat_my_data.odp It's probably not the best choice for a basic desktop without UPS, though, or a laptop.
- kl4m 15y ago
  XFS Zeroes the block instead of keeping the random/sensitive data that was already on the disk if the metadata necessary for that write was not commited before the failure. Ext leaves the data as is. If you want full protection against those situations instead of praying that it's still the right data or that it doesn't happen at all, you need a copy-on-write FS, or enable data=journal on ext3, at a massive performance cost.

gojomo 15y ago

Anyone know if btrfs would be more like XFS or ext3/4 in this regard?

spudlyo 15y ago

Mark didn't mention this in his article, but writes to the inode are only serialized in jfs/ext(2|3|4) if you are using direct I/O (i.e O_DIRECT) which for performance reasons MySQL ibdata files often are. This is why this problem does not often show up on filesystem benchmarks.