7 ms·

Fast Commits for Ext4

164 points by lukastyrychtr 6y ago

sroussey 6y ago
Curious how this change set would affect MySQL or postgresql?
- cbhl 6y ago
  Do folks typically turn on journaling at the filesystem layer when running a database? The database itself contains journaling, so one might choose to run with data=writeback or even directly against the block device if they were concerned about performance.
  - comboy 6y ago
    I don't think that those who read the manual do: https://www.postgresql.org/docs/13/wal-intro.html https://www.postgresql.org/docs/13/wal-intro.html (unless they care about quick crash recovery)
    - kureikain 6y ago
      Is this why many database yield better performance with XFS vx Ext4.
  - jabberwcky 6y ago
    You definitely need both, these are two completely different kinds of journalling: - Filesystem journalling is making robust changes to the data structures describing directories, files, and where files live, in units of atomic filesystem operations. For example, the filesystem journal may record "CREATE FILE", which translates to "update directory entry 1234 in directory block 5678, then allocate and initialize extent descriptor 9999, then write an inode at array entry 74234" - Database journalling is making robust changes to the data structures describing the actual file contents, in units of atomic logical application operations. For example, a DB journal may record "INSERT ROW", which translates to "update block 123 of this index file, and 234 of this data file", application-specific relationships like that cannot be captured by the filesystem on UNIX. (Note: NTFS is transactional on Windows. It's entirely possible to correlate independent writes and make them atomic, so on Windows at least, in theory a DB could exist without a separate journal. I don't know if this is used in practice). Even if it were in use, it places severe limits on the kinds of concurrency optimizations a database system could otherwise perform, because all of that stuff moves behind the curtain of the OS interfaces.
    - the8472 6y ago
      data=writeback does not disable the journal completely. It only removes ordering of the data writes relative to the metadata journaling. The metadata journaling itself remains active. You can create the file, preallocate space, fsync the inode and the directory to ensure that it will be visible after a crash and then begin using the allocated space as a journal. Then you only have to fdatasync or sync_file_range whatever part of your journal needs to be persisted and those syncs can now be unordered relative to the filesystem's metadata journal without risk of data loss. So data=writeback can be used safely, but you have to be very very careful about getting the syscall sequence right. Most applications implemented with sufficient paranoia and so are better served by stricter ordering modes and auto_da_alloc.
  - drran 6y ago
    After power loss without FS journal, your database may not have files to work with.
  - fulafel 6y ago
    It's on by default in ext4.
- jldugger 6y ago
  Probably not much -- the main win is not having to fsync files unrelated to your work, which is great for desktops which run multiple unrelated tasks (browser, terminals, rss readers, email clients, steam/game updates, package downloads). But I have to imagine SQL databases are typically run on systems dedicated to that singular task.
  - danudey 6y ago
    On the other hand, DBs call fsync() very often, which would presumably trigger this frequent "stop the world, write the data" behaviour vastly more than otherwise, which could make your I/O very stuttery. My solution has always been to just add a RAID controller with a battery-backed write cache, and then to disable barriers on ext4 and switch to journal=writeback. The data gets written to the cache "instantly", meaning less risk of data corruption (either at DB or filesystem level) from crashes or power outages. Saves having to worry about any of this sort of stuff.
    - jldugger 6y ago
      > On the other hand, DBs call fsync() very often, which would presumably trigger this frequent "stop the world, write the data" behaviour vastly more than otherwise, which could make your I/O very stuttery. But that's because they want to stop the world and write the data. Or more specically write the data they wrote. The issue is that more data is getting written than POSIX deems required, and that's the perf hit. Depending on your usecase, libeatmydata or battery backed RAID can help, but I'd feel a bit skeptical of 'we want to lie about fsync() to our DB' for many workloads.
The_rationalist 6y ago
I wonder if this interact with https://github.com/clearlinux-pkgs/linux/blob/master/0102-increase-the-ext4-default-commit-age.patch https://github.com/clearlinux-pkgs/linux/blob/master/0102-in...
quotemstr 6y ago
> One of the things that I did discuss with Harshad was using some hueristics, where if there are two "unrelated" applications (e.g., different session id, or process group leader, or different uid, etc. --- details to be determined layer), we would not entangele writes to unrelated files via fsync(2), while forcing files written by the same application to share fate with one another even if only file is fsync'ed. Ugh. This is why we can't have nice things. I really don't want the kernel's filesystem performance to depend on the number of different UIDs writing to the filesystem. That is insanity! Ted Ts'o is just wrong here: performance should take priority over preserving the behavior of applications that rely on non-contractual implementation details of the Linux kernel. fsync should sync only the indicated file, and that's that. We can add a mount option to let users opt into the older, safer behavior, but we shouldn't suffer for essentially an eternity because somewhere, someone might have written an application that depends on an ext4 implementation detail.
- anarazel 6y ago
  I think you have to see that in the light of all the complaints about data loss when people moved from ext3 to ext4. Ted IIRC pushed back against that with arguments like yours, but eventually gave up (leading to a heuristic for detecting such cases, cf the auto_da_alloc mount option).
  - quotemstr 6y ago
    Back then, people could point to specific examples of applications that depended on the sync-on-rename feature. What specific examples is he able to cite now? The entire discussion seems like a discussion of hypotheticals.
- nfoz 6y ago
  I'm not seeing your quoted text in the article. Am I missing something?
ape4 6y ago
ext4 is getting better just as Fedora has moved to Btrfs
- throwawaygulf 6y ago
  >Fedora has moved to Btrfs (Sigh)... Another reason to move away from Fedora.
  - LanternLight83 6y ago
    Funnily enough, BTRFS and ZRAM out-of-the-box are the reasons I moved to Fedora for all but my main Gentoo/ZFS system. To each their own c: Would you mind sharing what other features you've found pushing you away from Fedora and why? For my part, I'll admit, BTRFS and ZRAM both work against hibernation out-of-the-box, which is my most major gripe.
    - cmurf 6y ago
      Neither Btrfs nor zram-based swap, inhibit hibernation. There is a kernel lockdown policy that inhibits hibernation when UEFI Secure Boot is enabled. Since the hibernation image is neither encrypted nor signed, it's a potential attack vector to subvert the UEFI Secure Boot mechanism. Because UEFI Secure Boot systems are common (Windows hardware certification requires it exist and be enabled by default), it was decided to, by default, save the space on disk for disk-based swap and use zram-based swap instead. It is straight forward to create a disk-based partition during installation, and it is configured to support hibernation, while still remaining subject to the lockdown policy.
      - Dylan16807 6y ago
        Well why isn't it encrypted or signed?
        cmurf 6y ago
        The work is incomplete. https://lkml.org/lkml/2019/7/10/601 https://lkml.org/lkml/2019/7/10/601
severino 6y ago
I don't know if I got this correctly: > When we look at the fsync() and fdatasync() man pages, we see that those system calls only guarantee to write data linked to the given file descriptor. With ext4, as a side effect of the filesystem structure, all pending data and metadata for all file descriptors will be flushed instead. This creates a lot of I/O traffic that is unneeded to satisfy any given fsync() or fdatasync() call Does it mean that, under ext4, a call to fsync is essentially the same as a call to sync(2)?
- AdamJacobMuller 6y ago
  No, it might sync some metadata for other files, but it won't sync their data.
  - danudey 6y ago
    Are you certain? The quote actually says: > With ext4, as a side effect of the filesystem structure, *all pending data and metadata for all file descriptors* will be flushed instead. Is the article mistaken?
    - jradd 6y ago
      manpage suggests the article is not mistaken. `max_batch_time=0` disables batch processing synchronous write operations. the default is 15ms. It seems like a good idea for client filesystem to mount synchronously for data integrity. I'm not aware of i/o being a factor here. Of course this doesn't prevent applications mishandling of file objects.
- jbritton 6y ago
  Last year I had a problem where I had a function that was logging and a sync() call in another process was blocking the logging for greater than 10 seconds and this caused a timeout aborting the operation my code was performing. I have now moved all logging into a queue that logs on its own thread. This was a gotcha I never anticipated.
  - 6y ago
gautamcgoel 6y ago
Realistically, when can we expect this on fresh installs of Debian or Fedora? Will desktop users notice a perceptible performance gain when performing operations on large files?
- LeoPanthera 6y ago
  Fedora has switched to btrfs, but you can still choose your own preferred filesystem at install, just as you have always been able to.
  - cozzyd 6y ago
    i mean, my Fedora system will be on ext4 for a while since I don't wipe anew with every upgrade...
  - throw7 6y ago
    That's just the desktop "spin", albeit that is the default download. Personally, it's gonna be some time before I'd switch to btrfs if ever.
    - gautamcgoel 6y ago
      Why is that?
      - throw7 6y ago
        For me personally, I am waiting for ENOSPC to be 100% solved and understood. It may or may not be (maybe it's theoretically impossible to solve who knows), but until then I'm content with what I have now.
        the8472 6y ago
        > I am waiting for ENOSPC to be 100% solved and understood It's understood, there's no black magic. When there's not enough contiguous space to perform a metadata copy (it's a copy on write filesystem, no in-place overwrites!) then it returns an error, even if there's still some free but fragmented space or some free space in the data block groups. A rebalance when metadata blockgroup total size diverges significantly from actual use should solve this. For small filesystems there also is the option to choose mixed block groups, then there won't be the situation where there's still free space in the data groups while the metadata groups are full.
dis-sys 6y ago
maybe it is too early to ask, but is there any realistic benchmarks on this new feature?

164 points by lukastyrychtr 6y ago

sroussey 6y ago

Curious how this change set would affect MySQL or postgresql?

cbhl 6y ago
Do folks typically turn on journaling at the filesystem layer when running a database? The database itself contains journaling, so one might choose to run with data=writeback or even directly against the block device if they were concerned about performance.
- comboy 6y ago
  I don't think that those who read the manual do: https://www.postgresql.org/docs/13/wal-intro.html https://www.postgresql.org/docs/13/wal-intro.html (unless they care about quick crash recovery)
  - kureikain 6y ago
    Is this why many database yield better performance with XFS vx Ext4.
- jabberwcky 6y ago
  You definitely need both, these are two completely different kinds of journalling: - Filesystem journalling is making robust changes to the data structures describing directories, files, and where files live, in units of atomic filesystem operations. For example, the filesystem journal may record "CREATE FILE", which translates to "update directory entry 1234 in directory block 5678, then allocate and initialize extent descriptor 9999, then write an inode at array entry 74234" - Database journalling is making robust changes to the data structures describing the actual file contents, in units of atomic logical application operations. For example, a DB journal may record "INSERT ROW", which translates to "update block 123 of this index file, and 234 of this data file", application-specific relationships like that cannot be captured by the filesystem on UNIX. (Note: NTFS is transactional on Windows. It's entirely possible to correlate independent writes and make them atomic, so on Windows at least, in theory a DB could exist without a separate journal. I don't know if this is used in practice). Even if it were in use, it places severe limits on the kinds of concurrency optimizations a database system could otherwise perform, because all of that stuff moves behind the curtain of the OS interfaces.
  - the8472 6y ago
    data=writeback does not disable the journal completely. It only removes ordering of the data writes relative to the metadata journaling. The metadata journaling itself remains active. You can create the file, preallocate space, fsync the inode and the directory to ensure that it will be visible after a crash and then begin using the allocated space as a journal. Then you only have to fdatasync or sync_file_range whatever part of your journal needs to be persisted and those syncs can now be unordered relative to the filesystem's metadata journal without risk of data loss. So data=writeback can be used safely, but you have to be very very careful about getting the syscall sequence right. Most applications implemented with sufficient paranoia and so are better served by stricter ordering modes and auto_da_alloc.
- drran 6y ago
  After power loss without FS journal, your database may not have files to work with.
- fulafel 6y ago
  It's on by default in ext4.
jldugger 6y ago
Probably not much -- the main win is not having to fsync files unrelated to your work, which is great for desktops which run multiple unrelated tasks (browser, terminals, rss readers, email clients, steam/game updates, package downloads). But I have to imagine SQL databases are typically run on systems dedicated to that singular task.
- danudey 6y ago
  On the other hand, DBs call fsync() very often, which would presumably trigger this frequent "stop the world, write the data" behaviour vastly more than otherwise, which could make your I/O very stuttery. My solution has always been to just add a RAID controller with a battery-backed write cache, and then to disable barriers on ext4 and switch to journal=writeback. The data gets written to the cache "instantly", meaning less risk of data corruption (either at DB or filesystem level) from crashes or power outages. Saves having to worry about any of this sort of stuff.
  - jldugger 6y ago
    > On the other hand, DBs call fsync() very often, which would presumably trigger this frequent "stop the world, write the data" behaviour vastly more than otherwise, which could make your I/O very stuttery. But that's because they want to stop the world and write the data. Or more specically write the data they wrote. The issue is that more data is getting written than POSIX deems required, and that's the perf hit. Depending on your usecase, libeatmydata or battery backed RAID can help, but I'd feel a bit skeptical of 'we want to lie about fsync() to our DB' for many workloads.

The_rationalist 6y ago

I wonder if this interact with https://github.com/clearlinux-pkgs/linux/blob/master/0102-increase-the-ext4-default-commit-age.patch https://github.com/clearlinux-pkgs/linux/blob/master/0102-in...

quotemstr 6y ago

> One of the things that I did discuss with Harshad was using some hueristics, where if there are two "unrelated" applications (e.g., different session id, or process group leader, or different uid, etc. --- details to be determined layer), we would not entangele writes to unrelated files via fsync(2), while forcing files written by the same application to share fate with one another even if only file is fsync'ed. Ugh. This is why we can't have nice things. I really don't want the kernel's filesystem performance to depend on the number of different UIDs writing to the filesystem. That is insanity! Ted Ts'o is just wrong here: performance should take priority over preserving the behavior of applications that rely on non-contractual implementation details of the Linux kernel. fsync should sync only the indicated file, and that's that. We can add a mount option to let users opt into the older, safer behavior, but we shouldn't suffer for essentially an eternity because somewhere, someone might have written an application that depends on an ext4 implementation detail.

anarazel 6y ago
I think you have to see that in the light of all the complaints about data loss when people moved from ext3 to ext4. Ted IIRC pushed back against that with arguments like yours, but eventually gave up (leading to a heuristic for detecting such cases, cf the auto_da_alloc mount option).
- quotemstr 6y ago
  Back then, people could point to specific examples of applications that depended on the sync-on-rename feature. What specific examples is he able to cite now? The entire discussion seems like a discussion of hypotheticals.
nfoz 6y ago
I'm not seeing your quoted text in the article. Am I missing something?

ape4 6y ago

ext4 is getting better just as Fedora has moved to Btrfs

throwawaygulf 6y ago
>Fedora has moved to Btrfs (Sigh)... Another reason to move away from Fedora.
- LanternLight83 6y ago
  Funnily enough, BTRFS and ZRAM out-of-the-box are the reasons I moved to Fedora for all but my main Gentoo/ZFS system. To each their own c: Would you mind sharing what other features you've found pushing you away from Fedora and why? For my part, I'll admit, BTRFS and ZRAM both work against hibernation out-of-the-box, which is my most major gripe.
  - cmurf 6y ago
    Neither Btrfs nor zram-based swap, inhibit hibernation. There is a kernel lockdown policy that inhibits hibernation when UEFI Secure Boot is enabled. Since the hibernation image is neither encrypted nor signed, it's a potential attack vector to subvert the UEFI Secure Boot mechanism. Because UEFI Secure Boot systems are common (Windows hardware certification requires it exist and be enabled by default), it was decided to, by default, save the space on disk for disk-based swap and use zram-based swap instead. It is straight forward to create a disk-based partition during installation, and it is configured to support hibernation, while still remaining subject to the lockdown policy.
    - Dylan16807 6y ago
      Well why isn't it encrypted or signed?
      - cmurf 6y ago
        The work is incomplete. https://lkml.org/lkml/2019/7/10/601 https://lkml.org/lkml/2019/7/10/601

severino 6y ago

I don't know if I got this correctly: > When we look at the fsync() and fdatasync() man pages, we see that those system calls only guarantee to write data linked to the given file descriptor. With ext4, as a side effect of the filesystem structure, all pending data and metadata for all file descriptors will be flushed instead. This creates a lot of I/O traffic that is unneeded to satisfy any given fsync() or fdatasync() call Does it mean that, under ext4, a call to fsync is essentially the same as a call to sync(2)?

AdamJacobMuller 6y ago
No, it might sync some metadata for other files, but it won't sync their data.
- danudey 6y ago
  Are you certain? The quote actually says: > With ext4, as a side effect of the filesystem structure, *all pending data and metadata for all file descriptors* will be flushed instead. Is the article mistaken?
  - jradd 6y ago
    manpage suggests the article is not mistaken. `max_batch_time=0` disables batch processing synchronous write operations. the default is 15ms. It seems like a good idea for client filesystem to mount synchronously for data integrity. I'm not aware of i/o being a factor here. Of course this doesn't prevent applications mishandling of file objects.
jbritton 6y ago
Last year I had a problem where I had a function that was logging and a sync() call in another process was blocking the logging for greater than 10 seconds and this caused a timeout aborting the operation my code was performing. I have now moved all logging into a queue that logs on its own thread. This was a gotcha I never anticipated.
- 6y ago

gautamcgoel 6y ago

Realistically, when can we expect this on fresh installs of Debian or Fedora? Will desktop users notice a perceptible performance gain when performing operations on large files?

LeoPanthera 6y ago
Fedora has switched to btrfs, but you can still choose your own preferred filesystem at install, just as you have always been able to.
- cozzyd 6y ago
  i mean, my Fedora system will be on ext4 for a while since I don't wipe anew with every upgrade...
- throw7 6y ago
  That's just the desktop "spin", albeit that is the default download. Personally, it's gonna be some time before I'd switch to btrfs if ever.
  - gautamcgoel 6y ago
    Why is that?
    - throw7 6y ago
      For me personally, I am waiting for ENOSPC to be 100% solved and understood. It may or may not be (maybe it's theoretically impossible to solve who knows), but until then I'm content with what I have now.
      - the8472 6y ago
        > I am waiting for ENOSPC to be 100% solved and understood It's understood, there's no black magic. When there's not enough contiguous space to perform a metadata copy (it's a copy on write filesystem, no in-place overwrites!) then it returns an error, even if there's still some free but fragmented space or some free space in the data block groups. A rebalance when metadata blockgroup total size diverges significantly from actual use should solve this. For small filesystems there also is the option to choose mixed block groups, then there won't be the situation where there's still free space in the data groups while the metadata groups are full.

dis-sys 6y ago

maybe it is too early to ask, but is there any realistic benchmarks on this new feature?