4 ms·

Secure deletion: a single overwrite will do it

34 points by ilitirit 18y ago

ErrantX 18y ago
Whilst I am in agreement 35 writes is serious overkill I would dispute that a single write is suitable. Because of how standard data wiping software works one single pass would leave lots of traces. Perhaps not enough to pull entire documents etc. but with professional reconstruction software quite a lot of date can be recovered (I do this every week in my job). Given that you can guess at the contents of portions of the data (the OS :)) rebuilding is fairly easy. 3 passes is the correct method. One pass writing random data, one pass writing the 2's complement data (these passes ensure every bit has been "moved") then write it out with ) 0's. This ensures nearly untraceable data.
- karl11 18y ago
  I would be inclined to agree with you. I don't think the inconvenience is too great to wipe with three passes just to make sure.
  - ErrantX 18y ago
    I had a bigt discussion about this not very long ago with a client we were destroying some disks for. He had read something along similar lines (one pass is fine). I agree: 1 pass will tend to hide a lot of stuff. But when I did a quick example and showed him us recovering SAM files (windows password files) from one of his HDD's containing the MSCACHE hashes of several employees on his Windows Domain he was convinced (edit: of course that was a lucky break - and you do have to crack the passwords too - but we got some contact info and other document segments too :)).
- Chronos 18y ago
  Complements (either 1's or 2's) are no longer particularly useful patterns with modern HDD codings, which are wrapped in Reed-Solomon and then heavily encoded before hitting the disk. I'm more familiar with CD-ROMs than HDDs, so I'll explain the need for encoding in those terms. The Red and Yellow Books specify that a track consists of "pits" and "lands", where pit-staying-pit and land-staying-land both represent "0", while pit-becoming-land and land-becoming-pit both represent "1". Due to limitations of the pressing process (which involves physically squeezing a piece of metal in a press to create the pits), you need a specific minimum number of "0" bits separating each "1" bit (because the metal in question isn't ductile enough). But people want to write arbitrary data to their CDs. This is a problem. The solution: an encoding (Yellow Book, Annex D, Table D.1) maps 8-bit bytes into 14-bit "bytes". Because they had 16384 outputs for only 256 inputs, it was easy to find a list of outputs that spaced out the "1" bits with sufficient distance. (The other 16128 bit patterns are forbidden.) Venturing out of familiar territory and back on topic, HDDs have a similar problem: high frequency bit changes are more difficult to write to disk, because the neighboring bit domains bleed into each other. That is, if you choose "north-up" to mean "1" and "north-down" to mean "0", then writing the pattern "10101010" will tend to muddle out the magnetic field, strongly increasing the likelihood that you'll lose the data (especially if it sits for a while before you write to that spot again). The solution is another mapping code: I think hard drives use something more tame (like 8-to-11) and have different requirements, but the principles are the same. The downside from a data destruction POV: there's no guarantees that manufacturers use the same codes, even within their own product lines, since HDDs are sealed and have integrated controllers. The manufacturers don't even bother to tell you the physical encoding, because there's no need for anyone else to know it. The net result is that there is no longer any "magic code" you can write to a disk to stress sectors, guarantee data destruction, or have any other effect whatsoever upon the physical disk. All that stuff died out back in the days when IDE and SCSI hit the scene, after RLL and MFM drives died out.
m_eiman 18y ago
If you think that you can recover overwritten data, feel free to accept 'The Great Zero Challenge' over at http://16systems.com/zero/ http://16systems.com/zero/ "Q. What is this? A. A challenge to confirm whether or not a professional, established data recovery firm can recover data from a hard drive that has been overwritten with zeros once. We used the 32 year-old Unix dd command using /dev/zero as input to overwrite the drive. [...]" It's been over a year, and nobody has accepted the challenge yet. Even if there isn't any prize money to win, I'd think that the PR opportunity would be quite enough for any data recovery firm to do it. So my conclusion is: overwriting once is plenty good enough. You want to overwrite the whole disk though, otherwise the filesystem might leave metadata clues even after the file has been overwritten and unlinked.
- ErrantX 18y ago
  I'd love to take them on. However they are right it is highly unlikely data can be recovered from that drive. They have only a few folders/files on there it seems leaving little to go on to rebuild the image. It's not worth it - who needs the PR.. it's worthless when we recover zeroed disks frequently anyway :) If their willing to meet the cost to recover it (I'd do it cheap for £600). We are UK based so I guess they wont. I dont think you can draw the conclusion that overwriting once is plenty fine based on their conclusions. At least not till: - Someone has tried their disk (people not wanting too is different from giving it a shot :)) - Someone trying a more real life example (install an OS, then copy some files in, then DD it). Considering the second one has been done... :) They are quite welcome to clone any old OS drive onto a disk and wipe it the same way with DD. Then fly it over to us here and pay the £1000 (ish) cost to recover it. (yes, I know that is a bit outrageous but so is their "challenge").
  - m_eiman 18y ago
    "Considering the second one has been done... :)" It has? Do you have any source/article you can link to? I'd like to read about it and how it's done, seems to me like you'd need a fair amount of black magic to do it!
    - 18y ago
rythie 18y ago
I can kind of believe this is true. Since I know if someone formats a drive, I can get bits of the data back with strings /dev/sda but if they dd it with zeros, I can't do that. I suspect most/all data recovery firms will only deal with the 95%* of easier cases - which involve them basically running some software and not opening the drive. If they are bit better they might attempt the 4.9%* of cases where they have to replace/fix/bypass the drive's firmware. They write off the 0.1%* with dd style problems because they would cost too much for the customer and would need a better class of staff (recovery typically only costs a three-digit sum) That said it depends, how secure you need your data to be. I think given a budget of $100k (if not less) for one drive this could be possible (i.e. a government, competitor company). For example you could employ people from the company who makes the drives, reprogram the firmware to read the weak magnetic data etc. * percentages made up, to illustrate the point.
jcromartie 18y ago
I learned a nice trick with OS X's Disk Utility the other day. You can securely delete already-deleted files, after the fact, with this tool. Just select a volume, go to the Erase tab, and click "Erase Free Space..."
alecco 18y ago
Most people forget overwriting on the directory file name holding structures (of the different file systems.) The names of the deleted files stay there for long time. This is more so on FAT and FFS-type directories (guesstimate ~99% of file systems.) If a prosecutor (or government torturer) can prove the suspect had a file and wiped it out (e.g. /documents/superillegalfile.txt), I wouldn't like to be in his place.
- ErrantX 18y ago
  > If a prosecutor (or government torturer) can prove the suspect had a file and wiped it out (e.g. /documents/superillegalfile.txt), I wouldn't like to be in his place. In the UK that would be only sideline evidence, you'd have a hard time getting the CPS (crown prosecution service) to actually prosecute based on that evidence. Recovering the file is 9/10ths of the law :P (this is our main revenue stream btw - forensics for law enforcement).
  - alecco 18y ago
    If [the proof of file name deleted] is backed by other evidence, like ISP logs, CPS will likely hold [the case] valid. In this example, the suspect cant deny it was him instead a housemate or neighbour. It always adds.
barrkel 18y ago
I'd like to add something to this discussion, though, so others won't feel the pain I once felt some time ago: Sdelete from SysInternals is not safe to use if you are also using EFS (encrypted files on XP Pro / Vista etc.). You will get random corruption of random files on your file system. SDelete works by using the NT defragmentation API to discover what disk sectors are allocated to a file, so it can write to them directly. However, for whatever reason, in practice this does not work with encrypted (EFS) files. I've looked at the source for sdelete, and I can't see how it's going wrong, but I do know from experience (twice) that it does. The symptoms include blue-screens from corrupted OS files, overwritten documents, etc. There is further corroboration here: http://forum.sysinternals.com/forum_posts.asp?TID=3878 http://forum.sysinternals.com/forum_posts.asp?TID=3878 http://www.tech-archive.net/Archive/Win2000/microsoft.public.win2000.general/2004-10/2258.html http://www.tech-archive.net/Archive/Win2000/microsoft.public...

34 points by ilitirit 18y ago

Whilst I am in agreement 35 writes is serious overkill I would dispute that a single write is suitable. Because of how standard data wiping software works one single pass would leave lots of traces. Perhaps not enough to pull entire documents etc. but with professional reconstruction software quite a lot of date can be recovered (I do this every week in my job). Given that you can guess at the contents of portions of the data (the OS :)) rebuilding is fairly easy. 3 passes is the correct method. One pass writing random data, one pass writing the 2's complement data (these passes ensure every bit has been "moved") then write it out with ) 0's. This ensures nearly untraceable data.

karl11 18y ago
I would be inclined to agree with you. I don't think the inconvenience is too great to wipe with three passes just to make sure.
- ErrantX 18y ago
  I had a bigt discussion about this not very long ago with a client we were destroying some disks for. He had read something along similar lines (one pass is fine). I agree: 1 pass will tend to hide a lot of stuff. But when I did a quick example and showed him us recovering SAM files (windows password files) from one of his HDD's containing the MSCACHE hashes of several employees on his Windows Domain he was convinced (edit: of course that was a lucky break - and you do have to crack the passwords too - but we got some contact info and other document segments too :)).
Chronos 18y ago
Complements (either 1's or 2's) are no longer particularly useful patterns with modern HDD codings, which are wrapped in Reed-Solomon and then heavily encoded before hitting the disk. I'm more familiar with CD-ROMs than HDDs, so I'll explain the need for encoding in those terms. The Red and Yellow Books specify that a track consists of "pits" and "lands", where pit-staying-pit and land-staying-land both represent "0", while pit-becoming-land and land-becoming-pit both represent "1". Due to limitations of the pressing process (which involves physically squeezing a piece of metal in a press to create the pits), you need a specific minimum number of "0" bits separating each "1" bit (because the metal in question isn't ductile enough). But people want to write arbitrary data to their CDs. This is a problem. The solution: an encoding (Yellow Book, Annex D, Table D.1) maps 8-bit bytes into 14-bit "bytes". Because they had 16384 outputs for only 256 inputs, it was easy to find a list of outputs that spaced out the "1" bits with sufficient distance. (The other 16128 bit patterns are forbidden.) Venturing out of familiar territory and back on topic, HDDs have a similar problem: high frequency bit changes are more difficult to write to disk, because the neighboring bit domains bleed into each other. That is, if you choose "north-up" to mean "1" and "north-down" to mean "0", then writing the pattern "10101010" will tend to muddle out the magnetic field, strongly increasing the likelihood that you'll lose the data (especially if it sits for a while before you write to that spot again). The solution is another mapping code: I think hard drives use something more tame (like 8-to-11) and have different requirements, but the principles are the same. The downside from a data destruction POV: there's no guarantees that manufacturers use the same codes, even within their own product lines, since HDDs are sealed and have integrated controllers. The manufacturers don't even bother to tell you the physical encoding, because there's no need for anyone else to know it. The net result is that there is no longer any "magic code" you can write to a disk to stress sectors, guarantee data destruction, or have any other effect whatsoever upon the physical disk. All that stuff died out back in the days when IDE and SCSI hit the scene, after RLL and MFM drives died out.

m_eiman 18y ago

If you think that you can recover overwritten data, feel free to accept 'The Great Zero Challenge' over at http://16systems.com/zero/ http://16systems.com/zero/ "Q. What is this? A. A challenge to confirm whether or not a professional, established data recovery firm can recover data from a hard drive that has been overwritten with zeros once. We used the 32 year-old Unix dd command using /dev/zero as input to overwrite the drive. [...]" It's been over a year, and nobody has accepted the challenge yet. Even if there isn't any prize money to win, I'd think that the PR opportunity would be quite enough for any data recovery firm to do it. So my conclusion is: overwriting once is plenty good enough. You want to overwrite the whole disk though, otherwise the filesystem might leave metadata clues even after the file has been overwritten and unlinked.

ErrantX 18y ago
I'd love to take them on. However they are right it is highly unlikely data can be recovered from that drive. They have only a few folders/files on there it seems leaving little to go on to rebuild the image. It's not worth it - who needs the PR.. it's worthless when we recover zeroed disks frequently anyway :) If their willing to meet the cost to recover it (I'd do it cheap for £600). We are UK based so I guess they wont. I dont think you can draw the conclusion that overwriting once is plenty fine based on their conclusions. At least not till: - Someone has tried their disk (people not wanting too is different from giving it a shot :)) - Someone trying a more real life example (install an OS, then copy some files in, then DD it). Considering the second one has been done... :) They are quite welcome to clone any old OS drive onto a disk and wipe it the same way with DD. Then fly it over to us here and pay the £1000 (ish) cost to recover it. (yes, I know that is a bit outrageous but so is their "challenge").
- m_eiman 18y ago
  "Considering the second one has been done... :)" It has? Do you have any source/article you can link to? I'd like to read about it and how it's done, seems to me like you'd need a fair amount of black magic to do it!
  - 18y ago

rythie 18y ago

I can kind of believe this is true. Since I know if someone formats a drive, I can get bits of the data back with strings /dev/sda but if they dd it with zeros, I can't do that. I suspect most/all data recovery firms will only deal with the 95%* of easier cases - which involve them basically running some software and not opening the drive. If they are bit better they might attempt the 4.9%* of cases where they have to replace/fix/bypass the drive's firmware. They write off the 0.1%* with dd style problems because they would cost too much for the customer and would need a better class of staff (recovery typically only costs a three-digit sum) That said it depends, how secure you need your data to be. I think given a budget of $100k (if not less) for one drive this could be possible (i.e. a government, competitor company). For example you could employ people from the company who makes the drives, reprogram the firmware to read the weak magnetic data etc. * percentages made up, to illustrate the point.

jcromartie 18y ago

I learned a nice trick with OS X's Disk Utility the other day. You can securely delete already-deleted files, after the fact, with this tool. Just select a volume, go to the Erase tab, and click "Erase Free Space..."

alecco 18y ago

Most people forget overwriting on the directory file name holding structures (of the different file systems.) The names of the deleted files stay there for long time. This is more so on FAT and FFS-type directories (guesstimate ~99% of file systems.) If a prosecutor (or government torturer) can prove the suspect had a file and wiped it out (e.g. /documents/superillegalfile.txt), I wouldn't like to be in his place.

ErrantX 18y ago
> If a prosecutor (or government torturer) can prove the suspect had a file and wiped it out (e.g. /documents/superillegalfile.txt), I wouldn't like to be in his place. In the UK that would be only sideline evidence, you'd have a hard time getting the CPS (crown prosecution service) to actually prosecute based on that evidence. Recovering the file is 9/10ths of the law :P (this is our main revenue stream btw - forensics for law enforcement).
- alecco 18y ago
  If [the proof of file name deleted] is backed by other evidence, like ISP logs, CPS will likely hold [the case] valid. In this example, the suspect cant deny it was him instead a housemate or neighbour. It always adds.

barrkel 18y ago

I'd like to add something to this discussion, though, so others won't feel the pain I once felt some time ago: Sdelete from SysInternals is not safe to use if you are also using EFS (encrypted files on XP Pro / Vista etc.). You will get random corruption of random files on your file system. SDelete works by using the NT defragmentation API to discover what disk sectors are allocated to a file, so it can write to them directly. However, for whatever reason, in practice this does not work with encrypted (EFS) files. I've looked at the source for sdelete, and I can't see how it's going wrong, but I do know from experience (twice) that it does. The symptoms include blue-screens from corrupted OS files, overwritten documents, etc. There is further corroboration here: http://forum.sysinternals.com/forum_posts.asp?TID=3878 http://forum.sysinternals.com/forum_posts.asp?TID=3878 http://www.tech-archive.net/Archive/Win2000/microsoft.public.win2000.general/2004-10/2258.html http://www.tech-archive.net/Archive/Win2000/microsoft.public...