7 ms·

The magnetized material on the platter can slowly lose its direction (and thus the data) while laying around. So you should read all your data every 3 years or

The magnetized material on the platter can slowly lose its direction (and thus the data) while laying around. So you should read all your data every 3 years or so to "refresh" the data.

rasz 5y ago
Where did you learn that? Its incorrect.
- adrian_b 5y ago
  There is a temperature-dependent probability for the magnetization of a magnetic domain to rotate spontaneously, causing one erroneous data bit. For materials with high coercitivity and at temperatures much lower than their Curie temperature, the probability is low but it is not null. A few years ago, I went back to archiving my data on magnetic tapes (LTO). However, before that and after optical discs became to small to be usable as an archival medium, I have used HDDs for backup and archival, for more than a decade. After returning to tapes, I have gradually transferred the old HDDs to tapes. I had a case with one HDD that would no longer spin after a few years of offline storage, but most functioned, at least enough to be able to transfer them on tapes. All the archival HDDs were duplicated, for redundance, and all the files were stored with checksums to detect data corruption. The checksums have been absolutely essential, because after several years of storage there were a lot of corrupted files, but for none of them the HDD reported any error. The HDDs were about half WD and half Seagate, all of them Raid Edition or equivalent. I had a HDD archive with about 200 TB of data, and on average, after 5 years of storage, there was a corrupted file about every couple of TB. However, because each HDD was duplicated and the errors did not affect the same file in both copies, I have recovered all files. All the corrupted files were almost certainly caused by spontaneous magnetization rotation affecting a few bits, so what the previous poster said is not only theoretically correct, but proved in practice by at least one example. Most people do not realize how frequent data corruption is on HDDs, because they do not use file checksums or extra file copies, to detect corrupted data. Bit errors are frequent on HDDs, but most are corrected by the HDD firmware. However, from time to time the correction capabilities of the codes used in HDDs are exceeded and silent data corruption happens. After a lot of experience, I never trust the firmware of any storage media, so I store checksums, in extended attributes, for all the files that I will keep for more than a day.
  - Santosh83 5y ago
    What is your recommendation for a cross platform program that can automatically create and store checksums for all the files in a HDD (or folder) and compare them with the files later and restore damaged files from another copy?
    - adrian_b 5y ago
      Ideally, whenever a file is modified, the checksums should be recomputed when the file is closed, after having been opened as read-write. However, this behavior can obtained only by replacing the file system drivers with a custom version. That could be done on Linux but it cannot be done as a cross-platform solution. So for cross-platform, you need a pair of scripts, one for verifying the checksums, recursively for all the files in a directory, and another script for updating all the checksums for the files in some directory, which you need to either invoke manually after editing some files or downloading some new files, or you could set the script to run periodically, e.g. once per day, maybe prior to a backup script. Nowadays a bash script can run on any operating system, so it is cross-platform enough, but writing a new pair of scripts for each operating system for whatever tool is used there for scripting is not much work. The 2 scripts should consist each of a pair of scripts, a top level script that gets all the files from a directory (e.g. on UNIX using 'find "$1" -type f') and invokes the second script for updating or verifying the checksums for a single file. The scripts that do the work for one file just need to invoke some standard checksumming programs, e.g. sha256sum and so on, which are now available on any platform, if you take care to install them. For verifying, missing checksums or checksums that differ from a newly computed checksum give an error, for updating, the checksums are recomputed if they are missing or if their timestamp differs from the file modification timestamp. In the beginning, I was storing the checksums in one extra file per directory, but that had many disadvantages. Then I have switched to the use of extended attributes, which allow the storage of the checksums together with the file, transparently. Extended attributes are much more convenient, but their cross-platform use is more complex. For updating or verifying the extended attributes, you need 3 commands, list, read and write extended attributes. These not only are different on each operating system, but they are some times different on a single operating system for different file systems. So you either need to write different scripts for each platform, or you must detect the platform and choose the appropriate extended attribute commands. The work is about the same for both choices, so it might be simpler to just have different scripts for each platform. If you use per platform scripts, they can be small, less than a page each, so it is less likely to make mistakes in them. When you use extended attributes, you must take care that on open-source systems, like Linux or FreeBSD, not all file systems support extended attributes and those that support them might have been configured to ignore them. So, especially on Linux, you must check that you have enabled support for extended attributes and you might have to recompile the common utilities, like fileutils, because by default you might have commands like cp and mv that lose the extended attributes. Also neither all GUI file managers nor all archiver versions (e.g. of tar or pax), handle correctly the extended attributes. The main pitfall on Linux is that tmpfs has only partial support for extended attributes, so on many Linux systems, copying a file to /tmp will lose its extended attributes. When copying files through the network, the most reliable way is to use rsync, preferably rsync over ssh. That will copy any file together with its extended attributes, regardless what operating systems are at the 2 ends. Otherwise, Samba can be used between most operating systems. I have used it in mixed configurations with Linux, FreeBSD and Windows computers, without problems with preserving the extended attributes. There are new NFS versions with extended attribute support, but all the computers must be configured correctly, with non-default options. I have not used such a version of NFS. So the conclusion is that the files themselves are cross-platform if you take care to not use for transferring them methods that would silently lose the extended attributes, but the simplest way is to rewrite a pair of updating/verification scripts for each platform, because those can be very simple, while a cross-platform script would be large and cumbersome.
weiner1 5y ago
Citation very much Needed. Magnets only lose their magnetic powers very slowly. For example, samarium cobalt magnets might decrease their magnetic strength about 1 percent over a decade, which contradicts your claim.
- adrian_b 5y ago
  Samarium cobalt are actually some of the best from this point of view, because they have a very high Curie temperature. Most magnets made from cheaper materials with lower Curie temperatures, e.g. Nd-Fe-B or ferrites, lose their magnetization at higher rates at room temperature. Losing a few percent of the magnetic strength for the whole magnet would be equivalent with massive data corruption on a HDD, because that is the percentage of magnetic domains whose magnetization has rotated, i.e. the percentage of corrupt bits. The magnetic materials used in HDDs have actually a much lower rate of spontaneous change of magnetization. Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD, which is close to the typical rates of corruption actually seen in reality.
  - gruez 5y ago
    >Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD Hard drives add ECC info to each sector to prevent random bitflips from corrupting data[1]. Also IME actual rates of data corruption is far lower than 1 bit per year. This is on a filesystem with checksums, so any corruption would be easily detected. [1] https://en.wikipedia.org/wiki/File:Advanced_format_(4Kib)_HDD_sector.svg https://en.wikipedia.org/wiki/File:Advanced_format_(4Kib)_HD...
  - dannyw 5y ago
    If you're encountering 1 bitflip per year per HDD, something is horrifically wrong. I've only observed 1 incident of corruption in 5 years of 00+ drives with ZFS, and that was due to a dying HBA controller, not the HDD.

The magnetized material on the platter can slowly lose its direction (and thus the data) while laying around. So you should read all your data every 3 years or so to "refresh" the data.

rasz 5y ago

Where did you learn that? Its incorrect.

adrian_b 5y ago
There is a temperature-dependent probability for the magnetization of a magnetic domain to rotate spontaneously, causing one erroneous data bit. For materials with high coercitivity and at temperatures much lower than their Curie temperature, the probability is low but it is not null. A few years ago, I went back to archiving my data on magnetic tapes (LTO). However, before that and after optical discs became to small to be usable as an archival medium, I have used HDDs for backup and archival, for more than a decade. After returning to tapes, I have gradually transferred the old HDDs to tapes. I had a case with one HDD that would no longer spin after a few years of offline storage, but most functioned, at least enough to be able to transfer them on tapes. All the archival HDDs were duplicated, for redundance, and all the files were stored with checksums to detect data corruption. The checksums have been absolutely essential, because after several years of storage there were a lot of corrupted files, but for none of them the HDD reported any error. The HDDs were about half WD and half Seagate, all of them Raid Edition or equivalent. I had a HDD archive with about 200 TB of data, and on average, after 5 years of storage, there was a corrupted file about every couple of TB. However, because each HDD was duplicated and the errors did not affect the same file in both copies, I have recovered all files. All the corrupted files were almost certainly caused by spontaneous magnetization rotation affecting a few bits, so what the previous poster said is not only theoretically correct, but proved in practice by at least one example. Most people do not realize how frequent data corruption is on HDDs, because they do not use file checksums or extra file copies, to detect corrupted data. Bit errors are frequent on HDDs, but most are corrected by the HDD firmware. However, from time to time the correction capabilities of the codes used in HDDs are exceeded and silent data corruption happens. After a lot of experience, I never trust the firmware of any storage media, so I store checksums, in extended attributes, for all the files that I will keep for more than a day.
- Santosh83 5y ago
  What is your recommendation for a cross platform program that can automatically create and store checksums for all the files in a HDD (or folder) and compare them with the files later and restore damaged files from another copy?
  - adrian_b 5y ago
    Ideally, whenever a file is modified, the checksums should be recomputed when the file is closed, after having been opened as read-write. However, this behavior can obtained only by replacing the file system drivers with a custom version. That could be done on Linux but it cannot be done as a cross-platform solution. So for cross-platform, you need a pair of scripts, one for verifying the checksums, recursively for all the files in a directory, and another script for updating all the checksums for the files in some directory, which you need to either invoke manually after editing some files or downloading some new files, or you could set the script to run periodically, e.g. once per day, maybe prior to a backup script. Nowadays a bash script can run on any operating system, so it is cross-platform enough, but writing a new pair of scripts for each operating system for whatever tool is used there for scripting is not much work. The 2 scripts should consist each of a pair of scripts, a top level script that gets all the files from a directory (e.g. on UNIX using 'find "$1" -type f') and invokes the second script for updating or verifying the checksums for a single file. The scripts that do the work for one file just need to invoke some standard checksumming programs, e.g. sha256sum and so on, which are now available on any platform, if you take care to install them. For verifying, missing checksums or checksums that differ from a newly computed checksum give an error, for updating, the checksums are recomputed if they are missing or if their timestamp differs from the file modification timestamp. In the beginning, I was storing the checksums in one extra file per directory, but that had many disadvantages. Then I have switched to the use of extended attributes, which allow the storage of the checksums together with the file, transparently. Extended attributes are much more convenient, but their cross-platform use is more complex. For updating or verifying the extended attributes, you need 3 commands, list, read and write extended attributes. These not only are different on each operating system, but they are some times different on a single operating system for different file systems. So you either need to write different scripts for each platform, or you must detect the platform and choose the appropriate extended attribute commands. The work is about the same for both choices, so it might be simpler to just have different scripts for each platform. If you use per platform scripts, they can be small, less than a page each, so it is less likely to make mistakes in them. When you use extended attributes, you must take care that on open-source systems, like Linux or FreeBSD, not all file systems support extended attributes and those that support them might have been configured to ignore them. So, especially on Linux, you must check that you have enabled support for extended attributes and you might have to recompile the common utilities, like fileutils, because by default you might have commands like cp and mv that lose the extended attributes. Also neither all GUI file managers nor all archiver versions (e.g. of tar or pax), handle correctly the extended attributes. The main pitfall on Linux is that tmpfs has only partial support for extended attributes, so on many Linux systems, copying a file to /tmp will lose its extended attributes. When copying files through the network, the most reliable way is to use rsync, preferably rsync over ssh. That will copy any file together with its extended attributes, regardless what operating systems are at the 2 ends. Otherwise, Samba can be used between most operating systems. I have used it in mixed configurations with Linux, FreeBSD and Windows computers, without problems with preserving the extended attributes. There are new NFS versions with extended attribute support, but all the computers must be configured correctly, with non-default options. I have not used such a version of NFS. So the conclusion is that the files themselves are cross-platform if you take care to not use for transferring them methods that would silently lose the extended attributes, but the simplest way is to rewrite a pair of updating/verification scripts for each platform, because those can be very simple, while a cross-platform script would be large and cumbersome.

weiner1 5y ago

Citation very much Needed. Magnets only lose their magnetic powers very slowly. For example, samarium cobalt magnets might decrease their magnetic strength about 1 percent over a decade, which contradicts your claim.

adrian_b 5y ago
Samarium cobalt are actually some of the best from this point of view, because they have a very high Curie temperature. Most magnets made from cheaper materials with lower Curie temperatures, e.g. Nd-Fe-B or ferrites, lose their magnetization at higher rates at room temperature. Losing a few percent of the magnetic strength for the whole magnet would be equivalent with massive data corruption on a HDD, because that is the percentage of magnetic domains whose magnetization has rotated, i.e. the percentage of corrupt bits. The magnetic materials used in HDDs have actually a much lower rate of spontaneous change of magnetization. Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD, which is close to the typical rates of corruption actually seen in reality.
- gruez 5y ago
  >Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD Hard drives add ECC info to each sector to prevent random bitflips from corrupting data[1]. Also IME actual rates of data corruption is far lower than 1 bit per year. This is on a filesystem with checksums, so any corruption would be easily detected. [1] https://en.wikipedia.org/wiki/File:Advanced_format_(4Kib)_HDD_sector.svg https://en.wikipedia.org/wiki/File:Advanced_format_(4Kib)_HD...
- dannyw 5y ago
  If you're encountering 1 bitflip per year per HDD, something is horrifically wrong. I've only observed 1 incident of corruption in 5 years of 00+ drives with ZFS, and that was due to a dying HBA controller, not the HDD.