4 ms·

I agree, there is only one reasonable method to consider a drive "wiped": encrypt all data written to it with a key; wipe == destroy the key. Drives are basical

I agree, there is only one reasonable method to consider a drive "wiped": encrypt all data written to it with a key; wipe == destroy the key. Drives are basically little computers now, they have full-featured CPUs, OSes, memory, write caches and persistent buffers, they compress data before writing it, lie to you about where the data is (ok fine, it's a legitimate abstraction), and who knows what else. My point is that drives have become way too complex for wiping by "overwriting" "every" "block" to ever be an option again (and yes all three of those words deserve scare-quotes). The answer to this problem has become: always encrypt, lose the key to wipe. Period.

flavius29663 5y ago
what if you just fill the whole disk with random bytes? Wouldn't that take care of it ?
- thelopa 5y ago
  There could be bits of plain text data left over in over-provisioned storage set up in case a block goes bad (e.g. a flash cell that can no longer erase).
  - flavius29663 5y ago
    Would that extra cell be used unless there is a need for it? If there is a need for it, then I assume it will be a need for it when you fill the drive too. But yeah, it depends on the manufacturer of the drive, so you don't want to take the chance, Full encryption is unbeatable.
    - dan000892 5y ago
      Flash devices maximize the longevity of the cells by wear leveling. The on-board controller maintains a map between the physical cell layout and the logical layout that's changing with every single write operation. Extra cells are used both reactively and proactively. Reactively: Error when erasing or write/verifying a cell? Copy contents to spare cell, flag old cell as bad, update the map, maybe complain if running low on spare cells but otherwise call it a day. Proactively: Cells are only rated for so many erase operations which directly informs the longevity (total bytes written) of a drive. MLC cells can be written to 3,000 times and SLC 20,000 but the latter is much more expensive. For use cases where greater longevity is required (without spending SLC money), why not put a disk together that has twice as many cells as it needs and wear level across all of them? If you `dd if=/dev/zero of=/dev/sdX bs=64k status=progress` your 32GB drive and it'll complete after 32GB but you have no way of knowing how many more cells that drive has that you didn't touch. (Also as one of the parents pointed out, disk controllers are getting smart. You don't know if it was transparently compressing data too causing you to touch even fewer cells. ATA Secure Erase is the way to go.)
- infogulch 5y ago
  You don't have access to "the whole disk". What the drive presents to the OS as an array of blocks is just an abstraction now. There are at least 3 commonly known ways that "filling" the disk with random bytes could fail to destroy the original data: * If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware. * To prevent uneven wear, ssd capacity is over provisioned which means it has more cells than what it tells the OS. In general, when you "overwrite" a block on your drive, that data likely does not actually overwrite the cell where the data used to be. The drive picks some other unused cell, writes the data there, and does some bookkeeping so it knows that this new cell is what the OS means when it asks for that block again. When the OS "fills the disk" it may not end up overwriting some of those cells due to the firmware's arbitrary wear leveling algorithm. * Many drives have a local write buffer that uses different, faster persistent storage to temporarily save writes to the drive as it takes its time to write out cells. This is to prevent data loss in the case of a power outage. In case the drive hadn't fully written all the blocks to the main storage when it boots back up again, it finds the data in the write buffer and finishes the job. The write buffer has both of the previous issues except its even more arbitrary and depends on load profile which your random filling routine is not guaranteed to hit. There is no such thing as securely overwriting blocks anymore, because "blocks" are a fiction presented to your OS by the drive's firmware. There is only one way to be sure the data on the drive is filled with random data: make it random to begin with, and ensure that it stays random by destroying the decryption key.
  - dylan604 5y ago
    >* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware. How useful is one block of recovered data?
    - KMnO4 5y ago
      Blocks can be up to 4mb. You could store all your passwords, private keys, contact/identity info, etc
glitchc 5y ago
How can one trust it though? Presumably the key has to make its way into the drive to actually decrypt the data. How can one be sure the key was actually deleted and not archived somewhere in the controller’s storage?
- infogulch 5y ago
  Ideally, data is encrypted at the CPU (or coprocessor or TPU chip) and sent to the drive encrypted. The drive never gets the key, its only job is to store encrypted blocks correctly. In the TPU case, the TPU chip controls decryption and the key never leaves the specialized, tamper-resistant chip; wiping means sending a command to the TPU to rotate keys, thus instantly destroying any access to the data.
neoncontrails 5y ago
A coworker once insisted on fondling discarded drives with extremely strong magnets, then taking a power sander to the magnetized part. I wasn't sure if this was overkill or not. Is there any wisdom in this?
- dylan604 5y ago
  Sounds like someone needing a reason to use a power sander
  - sizzle 5y ago
    Yup exactly why I needed to buy my Milwaukee right angle die grinder… ask my wife ;)
- sundvor 5y ago
  Well if they wanted to be 100% certain nothing could be recovered, then that is a very good way to make sure.
- WaxProlix 5y ago
  When I worked for the UC Observatories we physically destroyed most retired drives, just in case of sensitive information. Some personal machines' drives were excepted but there was a good deal of hammering.
dan000892 5y ago
From your comment I'm not sure if you're aware but there are two established standards that do just that: ATA Secure Erase is an optional security feature implemented by many flash-based disk devices specifically because write degradation and overprovisioning preclude them from being wiped via overwrite as you would magnetic media. More SSD disk controllers than you may realize have an AES encryption engine and persistent storage for the key. The controller is encrypting/decrypting on the fly transparent to the user/system. A secure erase simply rolls that key (and flags all the cells as empty). As noted in the sibling comment, there must be trust in the device itself that the old key is unrecoverable (or the data recoverable if the prior key had been compromised). hdparm on any linux liveusb (e.g. Clonezilla) and an eSATA dock and you can nuke a few disks a minute with two commands: hdparm --user-master u --security-set-pass p /dev/sdX hdparm --user-master u --security-erase-enhanced p /dev/sdX hdparm has other arguments that can tell you if this is supported and how long the drive says it'll take to do it. The other standard is TCG's OPAL. An AES media encryption key (MEK) is generated to encrypt/decrypt all data on the fly. The MEK is encrypted with a user-defined key encrypting key (KEK) and that encrypted MEK is stored on the drive. At boot the user (or some external process) needs to supply the drive with the KEK to decrypt the MEK and begin transparently decrypting/encrypting data on the fly. Splitting the disk into multiple "locking ranges" with independent MEKs and KEKs is also described in the spec enabling secure wipe of just a portion of the disk. Lose the KEK or destroy the MEK and the data is unrecoverable. I'd say this option still requires trust in the disk itself but since TCG dictates not just the interface but also the implementation and has a corresponding certification process (as well as reliance on an off-device key), I reckon TCG OPAL devices should be more trusted than those that simply claim to implement the ATA Secure Erase command, but I haven't had an opportunity to test any.
- infogulch 5y ago
  Cool, thanks for sharing. It's interesting that even the drives themselves use this strategy to "erase" data; it doesn't actually overwrite the cells, it just always encrypts all data that goes to cells and rotates the key when it's asked to erase. This strategy is clearly superior, it doesn't cost any wear on the cells to perform, and it's instant. The multiple locking ranges is pretty neat, I'd never heard of that.

flavius29663 5y ago

what if you just fill the whole disk with random bytes? Wouldn't that take care of it ?

thelopa 5y ago
There could be bits of plain text data left over in over-provisioned storage set up in case a block goes bad (e.g. a flash cell that can no longer erase).
- flavius29663 5y ago
  Would that extra cell be used unless there is a need for it? If there is a need for it, then I assume it will be a need for it when you fill the drive too. But yeah, it depends on the manufacturer of the drive, so you don't want to take the chance, Full encryption is unbeatable.
  - dan000892 5y ago
    Flash devices maximize the longevity of the cells by wear leveling. The on-board controller maintains a map between the physical cell layout and the logical layout that's changing with every single write operation. Extra cells are used both reactively and proactively. Reactively: Error when erasing or write/verifying a cell? Copy contents to spare cell, flag old cell as bad, update the map, maybe complain if running low on spare cells but otherwise call it a day. Proactively: Cells are only rated for so many erase operations which directly informs the longevity (total bytes written) of a drive. MLC cells can be written to 3,000 times and SLC 20,000 but the latter is much more expensive. For use cases where greater longevity is required (without spending SLC money), why not put a disk together that has twice as many cells as it needs and wear level across all of them? If you `dd if=/dev/zero of=/dev/sdX bs=64k status=progress` your 32GB drive and it'll complete after 32GB but you have no way of knowing how many more cells that drive has that you didn't touch. (Also as one of the parents pointed out, disk controllers are getting smart. You don't know if it was transparently compressing data too causing you to touch even fewer cells. ATA Secure Erase is the way to go.)
infogulch 5y ago
You don't have access to "the whole disk". What the drive presents to the OS as an array of blocks is just an abstraction now. There are at least 3 commonly known ways that "filling" the disk with random bytes could fail to destroy the original data: * If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware. * To prevent uneven wear, ssd capacity is over provisioned which means it has more cells than what it tells the OS. In general, when you "overwrite" a block on your drive, that data likely does not actually overwrite the cell where the data used to be. The drive picks some other unused cell, writes the data there, and does some bookkeeping so it knows that this new cell is what the OS means when it asks for that block again. When the OS "fills the disk" it may not end up overwriting some of those cells due to the firmware's arbitrary wear leveling algorithm. * Many drives have a local write buffer that uses different, faster persistent storage to temporarily save writes to the drive as it takes its time to write out cells. This is to prevent data loss in the case of a power outage. In case the drive hadn't fully written all the blocks to the main storage when it boots back up again, it finds the data in the write buffer and finishes the job. The write buffer has both of the previous issues except its even more arbitrary and depends on load profile which your random filling routine is not guaranteed to hit. There is no such thing as securely overwriting blocks anymore, because "blocks" are a fiction presented to your OS by the drive's firmware. There is only one way to be sure the data on the drive is filled with random data: make it random to begin with, and ensure that it stays random by destroying the decryption key.
- dylan604 5y ago
  >* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware. How useful is one block of recovered data?
  - KMnO4 5y ago
    Blocks can be up to 4mb. You could store all your passwords, private keys, contact/identity info, etc

glitchc 5y ago

How can one trust it though? Presumably the key has to make its way into the drive to actually decrypt the data. How can one be sure the key was actually deleted and not archived somewhere in the controller’s storage?

infogulch 5y ago
Ideally, data is encrypted at the CPU (or coprocessor or TPU chip) and sent to the drive encrypted. The drive never gets the key, its only job is to store encrypted blocks correctly. In the TPU case, the TPU chip controls decryption and the key never leaves the specialized, tamper-resistant chip; wiping means sending a command to the TPU to rotate keys, thus instantly destroying any access to the data.

neoncontrails 5y ago

A coworker once insisted on fondling discarded drives with extremely strong magnets, then taking a power sander to the magnetized part. I wasn't sure if this was overkill or not. Is there any wisdom in this?

dylan604 5y ago
Sounds like someone needing a reason to use a power sander
- sizzle 5y ago
  Yup exactly why I needed to buy my Milwaukee right angle die grinder… ask my wife ;)
sundvor 5y ago
Well if they wanted to be 100% certain nothing could be recovered, then that is a very good way to make sure.
WaxProlix 5y ago
When I worked for the UC Observatories we physically destroyed most retired drives, just in case of sensitive information. Some personal machines' drives were excepted but there was a good deal of hammering.

dan000892 5y ago

From your comment I'm not sure if you're aware but there are two established standards that do just that: ATA Secure Erase is an optional security feature implemented by many flash-based disk devices specifically because write degradation and overprovisioning preclude them from being wiped via overwrite as you would magnetic media. More SSD disk controllers than you may realize have an AES encryption engine and persistent storage for the key. The controller is encrypting/decrypting on the fly transparent to the user/system. A secure erase simply rolls that key (and flags all the cells as empty). As noted in the sibling comment, there must be trust in the device itself that the old key is unrecoverable (or the data recoverable if the prior key had been compromised). hdparm on any linux liveusb (e.g. Clonezilla) and an eSATA dock and you can nuke a few disks a minute with two commands: hdparm --user-master u --security-set-pass p /dev/sdX hdparm --user-master u --security-erase-enhanced p /dev/sdX hdparm has other arguments that can tell you if this is supported and how long the drive says it'll take to do it. The other standard is TCG's OPAL. An AES media encryption key (MEK) is generated to encrypt/decrypt all data on the fly. The MEK is encrypted with a user-defined key encrypting key (KEK) and that encrypted MEK is stored on the drive. At boot the user (or some external process) needs to supply the drive with the KEK to decrypt the MEK and begin transparently decrypting/encrypting data on the fly. Splitting the disk into multiple "locking ranges" with independent MEKs and KEKs is also described in the spec enabling secure wipe of just a portion of the disk. Lose the KEK or destroy the MEK and the data is unrecoverable. I'd say this option still requires trust in the disk itself but since TCG dictates not just the interface but also the implementation and has a corresponding certification process (as well as reliance on an off-device key), I reckon TCG OPAL devices should be more trusted than those that simply claim to implement the ATA Secure Erase command, but I haven't had an opportunity to test any.

infogulch 5y ago
Cool, thanks for sharing. It's interesting that even the drives themselves use this strategy to "erase" data; it doesn't actually overwrite the cells, it just always encrypts all data that goes to cells and rotates the key when it's asked to erase. This strategy is clearly superior, it doesn't cost any wear on the cells to perform, and it's instant. The multiple locking ranges is pretty neat, I'd never heard of that.