7 ms·

On Btrfs, you can mark a folder/file/subvolume to have nocow, which has the effect of only doing a COW operation when you are creating snapshots.

mustache_kimono 4y ago
And that may work for btrfs, but again at some cost: "When you enable nocow on your files, Btrfs cannot compute checksums, meaning the integrity against bitrot and other corruptions cannot be guaranteed (i.e. in nocow mode, Btrfs drops to similar data consistency guarantees as other popular filesystems, like ext4, XFS, ...). In RAID modes, Btrfs cannot determine which mirror has the good copy if there is corruption on one of them."[0] [0]: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs#Test_2:_Using_nocow_option https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs...
- lazide 4y ago
  Yup. It’s a pretty fundamental thing. COW and data checksums (and usually automatic/inline compression) co-exist that way because it’s otherwise too expensive performance wise, and potentially dangerous corruption wise. For instance, if you modify a single byte in a large file, you need to update the data on disk as well as the checksum in the block header, and other related data. Chances are, these are in different sectors, and also require re-reading in all the other data in the block to compute the checksum. Anywhere in that process is a chance for corruption of the original data and the update. If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation. You could end up with the original data and update both invalid. Writing out a new COW block is done all at once, and if it fails, the write failed atomically, with the original data still intact.
  - tjoff 4y ago
    > Chances are, these are in different sectors, and also require re-reading in all the other data in the block to compute the checksum. Anywhere in that process is a chance for corruption of the original data and the update. Not much different than any interrupted write though. And a COW needs to reread just as much. > If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation. Something that you must always pay in a COW filesystem anyway? Is handled by other non-COW filesystems anyway. Just because a filesystem isn't COW doesn't mean every change needs to be in place either. Of course, a filesystem that is primarily COW might not want to maintain compression for non-COW edge-cases and that is quite reasonable.
    - lazide 4y ago
      Literally none of what you are saying is true. A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values). Additionally, a non-COW filesystem needs to update all metadata checksums/values for existing blocks. It's a much more pathological case (interrupted write wise) than a COW filesystem, because if it writes data, but hasn't written the checksum yet - the block is now corrupt. If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere. Which has the exact same overhead or worse as COW. And the data in a FS block is usually many large multiples of the sector size, which makes writes pretty hard to do in any sort of atomic way. Journaling helps, but not with performance here! Since you'd need to store the prior values + the new values, or you're still guaranteed to lose data. Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized. But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere.
      - tjoff 4y ago
        >A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values). No, a COW will re-read the entire block. Make the change and update the checksum and then write it back (to a new location, obviously). Way more than the newly written bytes - but way less than the entire file of course. Just as a non-COW fs will. >If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere. Which is exactly what the journal is for - and you already have a journal. Or you don't update it in-place. Just because it isn't COW doesn't mean you always have to update in place. >Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized. You can do exactly the same for a non-COW filesystem. >But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere. COW has the same issue. COW always need to "reshuffle"(?) data somewhere.
- Arnavion 4y ago
  While filesystem-integrated RAID makes sense since the filesystem can do filesystem-specific RAID placements (eg zfs), for now the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem.
  - mustache_kimono 4y ago
    > the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem. I suppose I don't understand this. Why would this be the case?
    - Arnavion 4y ago
      dm-integrity solves the problem of identifying which replica is good and which is bad. mdadm solves the problem of reading from the replica identified as good and fixing / reporting the replica identified as bad. The filesystem doesn't notice or care.
      - mustache_kimono 4y ago
        Ahh, so you intend, "If you can't use ZFS/btrfs, use dm-integrity"?
        Arnavion 4y ago
        No. I don't use ZFS since it's not licensed correctly, so I have no opinion on it. And BTRFS raid is not safe enough for use. So I'm saying "Use filesystem on mdadm on dm-integrity".
        nix23 4y ago
        >I don't use ZFS since it's not licensed correctly, Oh look a hobby-lawyer!! Please Linus, license your code "correctly" it's called ISC not GPL.

mustache_kimono 4y ago

And that may work for btrfs, but again at some cost: "When you enable nocow on your files, Btrfs cannot compute checksums, meaning the integrity against bitrot and other corruptions cannot be guaranteed (i.e. in nocow mode, Btrfs drops to similar data consistency guarantees as other popular filesystems, like ext4, XFS, ...). In RAID modes, Btrfs cannot determine which mirror has the good copy if there is corruption on one of them."[0] [0]: https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs#Test_2:_Using_nocow_option https://wiki.tnonline.net/w/Blog/SQLite_Performance_on_Btrfs...

lazide 4y ago
Yup. It’s a pretty fundamental thing. COW and data checksums (and usually automatic/inline compression) co-exist that way because it’s otherwise too expensive performance wise, and potentially dangerous corruption wise. For instance, if you modify a single byte in a large file, you need to update the data on disk as well as the checksum in the block header, and other related data. Chances are, these are in different sectors, and also require re-reading in all the other data in the block to compute the checksum. Anywhere in that process is a chance for corruption of the original data and the update. If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation. You could end up with the original data and update both invalid. Writing out a new COW block is done all at once, and if it fails, the write failed atomically, with the original data still intact.
- tjoff 4y ago
  > Chances are, these are in different sectors, and also require re-reading in all the other data in the block to compute the checksum. Anywhere in that process is a chance for corruption of the original data and the update. Not much different than any interrupted write though. And a COW needs to reread just as much. > If the byte changes the final compressed size, it may not fit in the current block at all, causing an expensive (or impossible) re-allocation. Something that you must always pay in a COW filesystem anyway? Is handled by other non-COW filesystems anyway. Just because a filesystem isn't COW doesn't mean every change needs to be in place either. Of course, a filesystem that is primarily COW might not want to maintain compression for non-COW edge-cases and that is quite reasonable.
  - lazide 4y ago
    Literally none of what you are saying is true. A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values). Additionally, a non-COW filesystem needs to update all metadata checksums/values for existing blocks. It's a much more pathological case (interrupted write wise) than a COW filesystem, because if it writes data, but hasn't written the checksum yet - the block is now corrupt. If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere. Which has the exact same overhead or worse as COW. And the data in a FS block is usually many large multiples of the sector size, which makes writes pretty hard to do in any sort of atomic way. Journaling helps, but not with performance here! Since you'd need to store the prior values + the new values, or you're still guaranteed to lose data. Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized. But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere.
    - tjoff 4y ago
      >A COW write only needs to checksum the newly written bytes. A non-cow filesystem needs to checksum all data contained in the block (unchanged prior with now new values). No, a COW will re-read the entire block. Make the change and update the checksum and then write it back (to a new location, obviously). Way more than the newly written bytes - but way less than the entire file of course. Just as a non-COW fs will. >If it writes the checksum, but not the data, the block is now corrupt. And there is no way to know which one is correct post-facto, without storing the old data and the new data somewhere. Which is exactly what the journal is for - and you already have a journal. Or you don't update it in-place. Just because it isn't COW doesn't mean you always have to update in place. >Compression wise, this isn't (as much) of an issue for COW filesystems, because it only needs to compress the newly written data, which can be allocated without concern to the previous allocation size, which is still there, allocated. It can mean less efficient compression if these are small, fragmented writes of course, which is why most of them have some sort of batching mechanism in place. Alternatively, it can copy a chunk of the block, though that can cause write amplification, and is usually minimized. You can do exactly the same for a non-COW filesystem. >But you don't run across potential pathological fragmentation issues, like where you compress prior blocks which now take significantly less space, or new blocks take more space and require reshuffling everywhere. COW has the same issue. COW always need to "reshuffle"(?) data somewhere.
Arnavion 4y ago
While filesystem-integrated RAID makes sense since the filesystem can do filesystem-specific RAID placements (eg zfs), for now the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem.
- mustache_kimono 4y ago
  > the safest RAID experience seems to be filesystem on mdadm on dm-integrity on disk partition, so that the RAID and RAID errors are invisible to the filesystem. I suppose I don't understand this. Why would this be the case?
  - Arnavion 4y ago
    dm-integrity solves the problem of identifying which replica is good and which is bad. mdadm solves the problem of reading from the replica identified as good and fixing / reporting the replica identified as bad. The filesystem doesn't notice or care.
    - mustache_kimono 4y ago
      Ahh, so you intend, "If you can't use ZFS/btrfs, use dm-integrity"?
      - Arnavion 4y ago
        No. I don't use ZFS since it's not licensed correctly, so I have no opinion on it. And BTRFS raid is not safe enough for use. So I'm saying "Use filesystem on mdadm on dm-integrity".
        nix23 4y ago
        >I don't use ZFS since it's not licensed correctly, Oh look a hobby-lawyer!! Please Linus, license your code "correctly" it's called ISC not GPL.