4 ms·
The current default is data=ordered, which should prevent this problem if the hardware doesn't lie. The data doesn't go in the journal, but it has to be writte
by Dylan16807 3y ago
The current default is data=ordered, which should prevent this problem if the hardware doesn't lie. The data doesn't go in the journal, but it has to be written before the journal is committed.
There was a point where ext3 defaulted to data=writeback, which can definitely give you files full of null bytes.
And data=journal exists but is overkill for this situation.
- charleshn 3y agoIt's likely because of delayed allocations (delalloc): https://issuetracker.google.com/issues/172227346#comment6 https://issuetracker.google.com/issues/172227346#comment6 because the only guarantee which data=ordered provides is the security guarantee that stale data won't be revealed. Yes, it's bad and breaks prefix append consistency, and does not match the documentation...
- fulafel 3y agoFor more context, that's a comment from one of the ext4 main authors, Ted Ts'o. the other subsequent comment from him spells out the case more but sadly no spelled out NUL byte origin story I spotted from skimming.
- charleshn 3y agoThe original report [0] shows the corruption due to NUL bytes at the end of the file (see the hexdump). This comment [1] from Ted Ts'o details the exact chain of events leading to it. [0] https://issuetracker.google.com/issues/172227346 https://issuetracker.google.com/issues/172227346 [1] https://issuetracker.google.com/issues/172227346#comment8 https://issuetracker.google.com/issues/172227346#comment8
- altfredd 3y agoThe "data" setting of ext filesystems isn't replacement for fsync().
- Dylan16807 3y agoIt's not a replacement but it can give you some guarantees. Also fsync is a terrible API that should be replaced, but that's mostly a different topic.
- the8472 3y agoAt least on linux you can use io_uring to make fsync asynchronous. And you can initiate some preparatory flushing with sync_file_range and only do the final commit with fsync to cut down the latency.
- consp 3y agoMy only n=1 observation is that null values in logs occur on nvme, ssd and spinning rust. All ext4 with defaults. I do have the idea it occurs more on nvme drives though. But maybe my systems settings are just booked.
- matja 3y ago> which should prevent this problem if the hardware doesn't lie. Or, one can take the ZFS approach and assume the hardware often lies :)
- worthless-trash 3y agoI do not know how zfs will overcome hardware lying. If its going to fetch data that is in the drives cache, how will it overcome the persistence problem ?
- danarmak 3y agoIt will at the very least notice that the read data does not match the stored checksum and not return the garbage data to the application. In redundant (raidz) setups it will then read the data from another disk, and update the faulty disk. In a non-redundant setup (or if enough disks are corrupted) it will signal an IO error. An error is preferred to silently returning garbage data!
- gumby 3y agoIt can’t tell if the drive is lying to it.
- worthless-trash 3y agoThis was my theory too, its not going to help in -these- situations. I can't see how.
- nolist_policy 3y agoThe "zeroed-out file" problem is not about firmware lying though, it is about applications using fsync() wrongly or not at all. Look up the O_PONIES controversy. Sure, due to their COW nature zfs and btrfs provide better behavior despite broken applications. But you can't solve persistence in the face of lying firmware. Even thought zfs has some enhancements to not corrupt itself on such drives, if you run for example a database on top, all guarantees around commit go out the window.