6 ms·

Considering the size of disks now-a-days, the chance of bit rot is high. And (I don't have the original source) on SSD, bit rots probability is higher still.

Considering the size of disks now-a-days, the chance of bit rot is high. And (I don't have the original source) on SSD, bit rots probability is higher still. So... ZFS and BTRF have meta-data as well as data checksumming. From what I've read, XFS may have metadata checksumming, but not on the data side of things. I consider checksumming important. Do others? What is the solution? What other file systems offer that sort of capability? Snapshotting is a second go-to function. Particularly when it is integrated into the LXC container creation process. (There was a comment elsewhere here which said LXC is on it's way out.... huh? what?)

legulere 9y ago
> on SSD, bit rots probability is higher still Do you have a source for this? So far I believed that bit-rot rates are pretty similar.
- stargrazer 9y ago
  A google search turns up a number of sources. Another concept which could use justification is whether SSDs bit rot more over long term than do spinning disks. I heard that somewhere as well.
  - legulere 9y ago
    Using google was the first thing I did. I couldn't find something substantial.
ars 9y ago
Use mirror raid and have mdadm do a full disk compare/check every month (this is the default on Debian). Additionally use smartmontools and configure it to do a short self test each night, and a long self test (i.e. full disk read) each week. This will catch/flag errors early, which mdadm will then detect.
- Wicher 9y ago
  Disk A and disk B both contain file SomeFile. On disk B this file has rotted. When reading the file SomeFile into memory, the read will be distributed among the disks (for performance reasons) (and it will probby need to span a multiple of the stripe size). Ok, file is read into memory, including the bitrotted part from disk B. Now we write the file blocks back - as one does. Voila! Both disks now contain the bitrot. And mdadm will not complain - disk A and B are identical for the area of file SomeFile.
  - IgorPartola 9y ago
    Moreover, even if you don't read the file, and the bit rot is discovered during the monthly compare, at least on Linux the disk that is considered correct will be chosen at random. So you need at least three disks to have some semblance of protection. Have you guys seen many laptops that come with three or more drives? Just use ZFS. Even on a single disk setup you will at least not get silent bit rot.
- binaryphile 9y ago
  "Never go to sea with two chronometers; take one or three." - adage cited in the Mythical Man Month
- gigatexal 9y ago
  Or just do raidz6 in ZFS and call it a day.
  - feld 9y ago
    Actually it's better to just do mirrors. Avoid RAIDZ at all costs if you care about performance and the ability to resilver in a reasonable amount of time.
    - gigatexal 9y ago
      Sure I agree. But nested mirrors still suffer from the same issue of losing a drive and you lose everything.
tytso 9y ago
There are many different ways that storage can be layered, and depending on your use case, you can put various advanced features (snapshots, checksum/data integrity, encryption, etc.) functionality in different places in the storage stack. You can put functionality the block device layer (e.g., lvm, dm-thin, dm-verity), you can put functionality into the file system, you can put functionality into the cluster filesystem (if you have such a thing), or you can put iti in the application level. Depending on the requirements of your use case different choices will make more sense. It's important to remember that RHEL is used for enterprise customers, and what might be common in the enterprise world might not be common for yours, and vice versa. Certainly, if you are using a cluster file system, it makes no sense to do checksum protections at the disk file system level, because you will be using some kind of erasure coding (e.g., Reed Solomon error correcting codes) to protect against node failure. This will also take care of bit flips. If you are using cloud VM's, or if you are using Docker / Kubernetes, then LXC won't make sense. It all depends on your technology choices, and so it's important to look at the big picture, not just at the individual file system's features.
- rob-olmos 9y ago
  Given a stock (or additional packages?) RHEL 7.4 install on non-clustered storage, what would be the best combination to detect & correct bitrot at the filesystem and lower level?
mrob 9y ago
Linux 4.12 introduced dm-integrity, which adds integrity checking at the block device level, so it will work with any file system: https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity
- zerd 9y ago
  One good thing about ZFS integrity checking is that when it finds an error it can repair the bit rot from another disk if you have parity or mirroring. Can dm-integrity do that?
  - harshreality 9y ago
    Not so far, it seems. https://www.spinics.net/lists/dm-devel/msg31482.html https://www.spinics.net/lists/dm-devel/msg31482.html > this target do not provide error correction, only detection of error (such a tool could be written on top of dm-integrity though)
  - Veratyr 9y ago
    dm-integrity will only operate on a single disk so no, not on its own. It does however return an error if the integrity check fails, so if you put mdadm on top, mdadm can repair the erroneous block. I've tested this and am currently running it on a 32TB array.
  - rleigh 9y ago
    Or multiple copies of the data (copies=n property).
bradmwalker 9y ago
Development focus is switching to LXD.

legulere 9y ago

> on SSD, bit rots probability is higher still Do you have a source for this? So far I believed that bit-rot rates are pretty similar.

stargrazer 9y ago
A google search turns up a number of sources. Another concept which could use justification is whether SSDs bit rot more over long term than do spinning disks. I heard that somewhere as well.
- legulere 9y ago
  Using google was the first thing I did. I couldn't find something substantial.

ars 9y ago

Use mirror raid and have mdadm do a full disk compare/check every month (this is the default on Debian). Additionally use smartmontools and configure it to do a short self test each night, and a long self test (i.e. full disk read) each week. This will catch/flag errors early, which mdadm will then detect.

Wicher 9y ago
Disk A and disk B both contain file SomeFile. On disk B this file has rotted. When reading the file SomeFile into memory, the read will be distributed among the disks (for performance reasons) (and it will probby need to span a multiple of the stripe size). Ok, file is read into memory, including the bitrotted part from disk B. Now we write the file blocks back - as one does. Voila! Both disks now contain the bitrot. And mdadm will not complain - disk A and B are identical for the area of file SomeFile.
- IgorPartola 9y ago
  Moreover, even if you don't read the file, and the bit rot is discovered during the monthly compare, at least on Linux the disk that is considered correct will be chosen at random. So you need at least three disks to have some semblance of protection. Have you guys seen many laptops that come with three or more drives? Just use ZFS. Even on a single disk setup you will at least not get silent bit rot.
binaryphile 9y ago
"Never go to sea with two chronometers; take one or three." - adage cited in the Mythical Man Month
gigatexal 9y ago
Or just do raidz6 in ZFS and call it a day.
- feld 9y ago
  Actually it's better to just do mirrors. Avoid RAIDZ at all costs if you care about performance and the ability to resilver in a reasonable amount of time.
  - gigatexal 9y ago
    Sure I agree. But nested mirrors still suffer from the same issue of losing a drive and you lose everything.

tytso 9y ago

There are many different ways that storage can be layered, and depending on your use case, you can put various advanced features (snapshots, checksum/data integrity, encryption, etc.) functionality in different places in the storage stack. You can put functionality the block device layer (e.g., lvm, dm-thin, dm-verity), you can put functionality into the file system, you can put functionality into the cluster filesystem (if you have such a thing), or you can put iti in the application level. Depending on the requirements of your use case different choices will make more sense. It's important to remember that RHEL is used for enterprise customers, and what might be common in the enterprise world might not be common for yours, and vice versa. Certainly, if you are using a cluster file system, it makes no sense to do checksum protections at the disk file system level, because you will be using some kind of erasure coding (e.g., Reed Solomon error correcting codes) to protect against node failure. This will also take care of bit flips. If you are using cloud VM's, or if you are using Docker / Kubernetes, then LXC won't make sense. It all depends on your technology choices, and so it's important to look at the big picture, not just at the individual file system's features.

rob-olmos 9y ago
Given a stock (or additional packages?) RHEL 7.4 install on non-clustered storage, what would be the best combination to detect & correct bitrot at the filesystem and lower level?

mrob 9y ago

Linux 4.12 introduced dm-integrity, which adds integrity checking at the block device level, so it will work with any file system: https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity

zerd 9y ago
One good thing about ZFS integrity checking is that when it finds an error it can repair the bit rot from another disk if you have parity or mirroring. Can dm-integrity do that?
- harshreality 9y ago
  Not so far, it seems. https://www.spinics.net/lists/dm-devel/msg31482.html https://www.spinics.net/lists/dm-devel/msg31482.html > this target do not provide error correction, only detection of error (such a tool could be written on top of dm-integrity though)
- Veratyr 9y ago
  dm-integrity will only operate on a single disk so no, not on its own. It does however return an error if the integrity check fails, so if you put mdadm on top, mdadm can repair the erroneous block. I've tested this and am currently running it on a 32TB array.
- rleigh 9y ago
  Or multiple copies of the data (copies=n property).

bradmwalker 9y ago

Development focus is switching to LXD.