4 ms·

On overwriting disks (on OpenBSD 7.0-current)

29 points by hucste 5y ago

BenjiWiebe 5y ago
Doesn't pv give you an ETA if the output device size is known? I recently did pv </dev/zero >/dev/sdb to zero a drive and I'm pretty sure it gave me an ETA...
- wyldfire 5y ago
  That seems to be exactly why it's endorsed. > For larger disks, an ETA proves invaluable. I like pv for this task.
- LeoPanthera 5y ago
  You have to supply the input device (or file) as an argument, instead of using stdin. Then it will read the size of the device/file. As far as I know, this does not work for the output device, and the size must be specified. So: pv /path/file >/dev/device Will give you an ETA.
  - hddqsb 5y ago
    On Linux, pv can actually determine the size of the output device (using lseek on stdout), so `pv </dev/zero >/dev/sdb` does give a correct ETA (tested with pv 1.6.6 on Debian).
WayToDoor 5y ago
Or you can just use dc3dd that features a progress bar by default.
projektfu 5y ago
Why is Secure Erase never mentioned in these articles? Not every drive supports it but it is a more secure way to erase SSDs and a faster way to erase self-encrypting disks. Also, for disks that require a full overwrite, it puts the disk into a mode that will continue running to completion even after a power failure. On spinning disks, it might be good to cover your deletions as you go with random data, but you can’t use dd to do that. You’d need a filesystem-aware tool. On SSDs, the TRIM function may help you approach that. https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
- tedunangst 5y ago
  Personally, I would never trust the drive to implement secure erase in a secure manner.
- scrapheap 5y ago
  Secure Erase is a very good method of erasing drives. However, this article wasn't looking at how to erase the content of the drive, it was looking at how to fill a drive/partition with randomness in preparation for using an encrypted file system on it (the randomness of the data the partition is filled with is important to help hinder cryptanalysis attempts).
anjbe 5y ago
OpenBSD’s dd(1) natively supports printing status with SIGINFO. If you run “stty status ^T”, then during dd you can press Ctrl-T to get an informational message including progress and rate of transfer. $ dd if=/dev/zero of=/dev/null load: 0.27 cmd: dd 2358 [running] 0.37u 0.65s 1% 99k 528102+0 records in 528101+0 records out 270387712 bytes transferred in 1.051 secs (257262549 bytes/sec) load: 0.27 cmd: dd 2358 [running] 1.00u 1.93s 10% 109k 1467936+0 records in 1467935+0 records out 751582720 bytes transferred in 2.930 secs (256426100 bytes/sec) load: 0.32 cmd: dd 2358 [running] 1.68u 3.12s 19% 109k 2424899+0 records in 2424899+0 records out 1241548288 bytes transferred in 4.834 secs (256784572 bytes/sec)
- saagarjha 5y ago
  This is one of the niceties that macOS happens to inherit from the BSDs.
- hddqsb 5y ago
  GNU dd has a similar feature, it prints the status on SIGUSR1. There is no keyboard shortcut though, so you'd have to open another terminal and run `pkill -USR1 -x dd`.
PaulKeeble 5y ago
For the better part of a decade there was a challenge put out on the internet and put to every recovery company on the planet for a very standard 80GB drive that contained a standard FAT32 filesystem and a few Megabytes worth of files to be recovered after it had been written with zeros from /dev/zero on linux using DD. The challenge offered $10k for recovery of the files. Once the process had been explained not a single commercial company in the entire world believed they could recover that file and no one took up the challenge and it finished up about a decade ago. People have been doing some really extreme things to clear out drives including 5+ passes and using true random sources but the fact is it is well beyond commercial means to recover data on a drive that is simply set to all zero just once. Maybe a state actor somewhere could read the edges of the data or the theoretic differences of zeroing and its state before that 0 was written but so far no one in the past 2 decades has shown it happen. If you are destroying a drive that is highly secret that another nation might want then sure be paranoid, but otherwise there is no reason for anyone to do anything but zero out a drive and that includes almost every single company. Most drives these days come with a secure wipe and that is the fastest way to clear them. Use that or just zeros, anything else is not making the (ex) data more secure its just taking longer. [1]https://tinyapps.org/docs/recovering_data_from_zero_filled_hard_drive.html https://tinyapps.org/docs/recovering_data_from_zero_filled_h... is a similar more recent experiment but not the one referenced since that one disappeared years ago.
- upofadown 5y ago
  The article is about randomizing a drive before using it for full disk encryption. That does in fact require a random pattern. The idea is to hide the structure of the data. >...but otherwise there is no reason for anyone to do anything but zero out a drive and that includes almost every single company. Overwriting doesn't work with SSDs. They leave copies of data laying around on the disk if you do that. That is even if you zero the whole disk as they have extra space in the form of overprovisioning. >Most drives these days come with a secure wipe and that is the fastest way to clear them. Tests have shown that a significant proportion of SSDs do this badly.
- hnbear 5y ago
  A lot of this is driven by regulatory, compliance or other outside factors. Working in finance, healthcare, and other secure industries you must demonstrate control of the environment, which includes processes to clear up data. Once a standard is common it’s easier to follow it and demonstrate that adherence to a regulator than it is to invent your own process and convince them it’s safe. Even in current world with SSDs that don’t follow the same mechanics as spinning disks, regulators expect a 9-pass DoD wipe because that’s what they’re used to. It’s easiest to just pass drives to a certified data destruction company who will provide a cert of evidence they did a DoD, NIST, etc certified wipe. Reality is seldom part of the process. Similar is the process for password resets. Regulars ask for 90 day resets and crazy complexity schemes when NIST actually just recommends relatively sane complexity, slightly longer passwords and no rotation.

29 points by hucste 5y ago

Doesn't pv give you an ETA if the output device size is known? I recently did pv </dev/zero >/dev/sdb to zero a drive and I'm pretty sure it gave me an ETA...

wyldfire 5y ago
That seems to be exactly why it's endorsed. > For larger disks, an ETA proves invaluable. I like pv for this task.
LeoPanthera 5y ago
You have to supply the input device (or file) as an argument, instead of using stdin. Then it will read the size of the device/file. As far as I know, this does not work for the output device, and the size must be specified. So: pv /path/file >/dev/device Will give you an ETA.
- hddqsb 5y ago
  On Linux, pv can actually determine the size of the output device (using lseek on stdout), so `pv </dev/zero >/dev/sdb` does give a correct ETA (tested with pv 1.6.6 on Debian).

WayToDoor 5y ago

Or you can just use dc3dd that features a progress bar by default.

projektfu 5y ago

Why is Secure Erase never mentioned in these articles? Not every drive supports it but it is a more secure way to erase SSDs and a faster way to erase self-encrypting disks. Also, for disks that require a full overwrite, it puts the disk into a mode that will continue running to completion even after a power failure. On spinning disks, it might be good to cover your deletions as you go with random data, but you can’t use dd to do that. You’d need a filesystem-aware tool. On SSDs, the TRIM function may help you approach that. https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase

tedunangst 5y ago
Personally, I would never trust the drive to implement secure erase in a secure manner.
scrapheap 5y ago
Secure Erase is a very good method of erasing drives. However, this article wasn't looking at how to erase the content of the drive, it was looking at how to fill a drive/partition with randomness in preparation for using an encrypted file system on it (the randomness of the data the partition is filled with is important to help hinder cryptanalysis attempts).

anjbe 5y ago

OpenBSD’s dd(1) natively supports printing status with SIGINFO. If you run “stty status ^T”, then during dd you can press Ctrl-T to get an informational message including progress and rate of transfer. $ dd if=/dev/zero of=/dev/null load: 0.27 cmd: dd 2358 [running] 0.37u 0.65s 1% 99k 528102+0 records in 528101+0 records out 270387712 bytes transferred in 1.051 secs (257262549 bytes/sec) load: 0.27 cmd: dd 2358 [running] 1.00u 1.93s 10% 109k 1467936+0 records in 1467935+0 records out 751582720 bytes transferred in 2.930 secs (256426100 bytes/sec) load: 0.32 cmd: dd 2358 [running] 1.68u 3.12s 19% 109k 2424899+0 records in 2424899+0 records out 1241548288 bytes transferred in 4.834 secs (256784572 bytes/sec)

saagarjha 5y ago
This is one of the niceties that macOS happens to inherit from the BSDs.
hddqsb 5y ago
GNU dd has a similar feature, it prints the status on SIGUSR1. There is no keyboard shortcut though, so you'd have to open another terminal and run `pkill -USR1 -x dd`.

PaulKeeble 5y ago

For the better part of a decade there was a challenge put out on the internet and put to every recovery company on the planet for a very standard 80GB drive that contained a standard FAT32 filesystem and a few Megabytes worth of files to be recovered after it had been written with zeros from /dev/zero on linux using DD. The challenge offered $10k for recovery of the files. Once the process had been explained not a single commercial company in the entire world believed they could recover that file and no one took up the challenge and it finished up about a decade ago. People have been doing some really extreme things to clear out drives including 5+ passes and using true random sources but the fact is it is well beyond commercial means to recover data on a drive that is simply set to all zero just once. Maybe a state actor somewhere could read the edges of the data or the theoretic differences of zeroing and its state before that 0 was written but so far no one in the past 2 decades has shown it happen. If you are destroying a drive that is highly secret that another nation might want then sure be paranoid, but otherwise there is no reason for anyone to do anything but zero out a drive and that includes almost every single company. Most drives these days come with a secure wipe and that is the fastest way to clear them. Use that or just zeros, anything else is not making the (ex) data more secure its just taking longer. [1]https://tinyapps.org/docs/recovering_data_from_zero_filled_hard_drive.html https://tinyapps.org/docs/recovering_data_from_zero_filled_h... is a similar more recent experiment but not the one referenced since that one disappeared years ago.

upofadown 5y ago
The article is about randomizing a drive before using it for full disk encryption. That does in fact require a random pattern. The idea is to hide the structure of the data. >...but otherwise there is no reason for anyone to do anything but zero out a drive and that includes almost every single company. Overwriting doesn't work with SSDs. They leave copies of data laying around on the disk if you do that. That is even if you zero the whole disk as they have extra space in the form of overprovisioning. >Most drives these days come with a secure wipe and that is the fastest way to clear them. Tests have shown that a significant proportion of SSDs do this badly.
hnbear 5y ago
A lot of this is driven by regulatory, compliance or other outside factors. Working in finance, healthcare, and other secure industries you must demonstrate control of the environment, which includes processes to clear up data. Once a standard is common it’s easier to follow it and demonstrate that adherence to a regulator than it is to invent your own process and convince them it’s safe. Even in current world with SSDs that don’t follow the same mechanics as spinning disks, regulators expect a 9-pass DoD wipe because that’s what they’re used to. It’s easiest to just pass drives to a certified data destruction company who will provide a cert of evidence they did a DoD, NIST, etc certified wipe. Reality is seldom part of the process. Similar is the process for password resets. Regulars ask for 90 day resets and crazy complexity schemes when NIST actually just recommends relatively sane complexity, slightly longer passwords and no rotation.