4 ms·

rm just unlinks the files at the inode level, seems like a disk forensics utility like the imagerec suite could have restored alot of the 'lost' data. In fact i

rm just unlinks the files at the inode level, seems like a disk forensics utility like the imagerec suite could have restored alot of the 'lost' data. In fact i've done it on source code after learning that the default behavior of untar was to overwrite all of your current directory structure. since it was text i didnt need anything fancy like imagerec, instead i just piped the raw disk to grep, and looked for parts i knew were in files i needed, then output them and the surrounding data to an external hard drive.

colanderman 10y ago
Back then yes. These days with SSDs, the OS will issue trim commands to the disk, zeroing the blocks from the OS point of view, and on SSDs with "secure delete", from a forensics point of view as well.
- drakenot 10y ago
  That is very interesting, I didn't know there were SSDs with "secure delete." I remember that Apple removed the "Secure Empty Trash" feature in OS X 10.11 because they didn't feel like they could guarantee secure deletion with their new fast SSDs present in most of their computers.
  - throwaway7767 10y ago
    "secure delete" (actually "secure erase") is a term from the ATA standard, it's supported by practically all ATA and later SATA devices for a very long time. The idea was you'd tell the disk to erase everything, it would do the actual deletion in the background but would not allow reading the old data. This was also the way to reset passworded harddrives, you could send a secure erase command without authentication, the drive would wipe itself and once done, the password is gone. Once SSDs arrived on the scene, they were limited by the interface as ATA didn't specify a way to mark sectors as unused. People found that their performance would degrade with use as all the sectors became utilized. But a secure erase command would mark all sectors as erased, so the drive would work like new. Later on, ATA got the TRIM (and later queued versions of TRIM) so the OS can mark specific sectors as erased. But the result is that a lot of people confuse flash sector erasing with secure erase.
- jnwatson 10y ago
  Perhaps I'm misunderstanding you, but trim doesn't secure delete anything. It merely indicates to the SSD that the sector is unused so that it can reap the underlying NAND location the next time it decides to garbage collect. In other words, the data is still there in the flash, but only the SSD firmware (and physical access) has access to it.
  - emcq 10y ago
    I think the reality is that some blocks will be lazily erased. I believe modern flash firmware periodically GCs in the background to prevent write amplification. If this is triggered you may overwrite data. NAND requires an erase before write so I wouldn't be surprised if some controllers are lazily erasing blocks to get better long term write speeds and prevent GC hiccups.
  - throwaway7767 10y ago
    The reason for marking it as erased is so the firmware can physically erase the flash sector. It could happen immediately, a minute later or next week. But a well-written SSD firmware will try to erase blocks any time it's not busy doing something else, as erasing blocks takes way longer than writing them. You might be able to recover something from the physical flash, but there's definitely no guarantees.
    - K0SM0S 10y ago
      > erasing blocks takes way longer than writing them Honest question, could you elaborate on that? Intuitively I would've thought writing and erasing are _the same_ from a physical standpoint, insofar as "erasing" means writing zeros.
      - colanderman 10y ago
        There are two operations on flash memory: "erase" (set all bits to 0) and "write" (set some bits to 1). The former is expensive, the latter cheap. Note that "write" can't set bits to 0. Hence the benefit of keeping zeroed blocks around.
tixzdk 10y ago
The same question was asked on Quora: https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob/comment/941166 https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
- leecarraher 10y ago
  thanks! oh those destitute days of swap as an actual resource instead of just a, something must be broken, i just filled up 32 gigs of main memory.
- patcheudor 10y ago
  Yup, disk arrays can be quite a problem when it comes to forensics recovery. This has been a bit of a nemesis of mine over the years. Friends or family will decide to buy a single RAID solution for backup and configure it to write files across the disks for performance because they don't know any better. Four years later they'll come to me because something happened to the array like a failed or corrupted controller NVRAM and they want to recover the files. For backup I recommend mirrored single individual spinning disks, preferably in multiple locations.
mprovost 10y ago
That's really only possible on a single user system. VFX studios usually have large NFS servers (filers) typically running proprietary filesystems, and with hundreds or thousands of clients writing files simultaneously they don't get all laid down on the attached arrays in nice neat chunks representing entire files. Typically they would even try and distribute writes to multiple raid groups/arrays for a single file. Also consider the size, you were able to copy it onto an external drive to recover. Studios don't have a spare, empty filer sitting around to dump the random contents of a few racks worth of disks onto.

colanderman 10y ago

Back then yes. These days with SSDs, the OS will issue trim commands to the disk, zeroing the blocks from the OS point of view, and on SSDs with "secure delete", from a forensics point of view as well.

drakenot 10y ago
That is very interesting, I didn't know there were SSDs with "secure delete." I remember that Apple removed the "Secure Empty Trash" feature in OS X 10.11 because they didn't feel like they could guarantee secure deletion with their new fast SSDs present in most of their computers.
- throwaway7767 10y ago
  "secure delete" (actually "secure erase") is a term from the ATA standard, it's supported by practically all ATA and later SATA devices for a very long time. The idea was you'd tell the disk to erase everything, it would do the actual deletion in the background but would not allow reading the old data. This was also the way to reset passworded harddrives, you could send a secure erase command without authentication, the drive would wipe itself and once done, the password is gone. Once SSDs arrived on the scene, they were limited by the interface as ATA didn't specify a way to mark sectors as unused. People found that their performance would degrade with use as all the sectors became utilized. But a secure erase command would mark all sectors as erased, so the drive would work like new. Later on, ATA got the TRIM (and later queued versions of TRIM) so the OS can mark specific sectors as erased. But the result is that a lot of people confuse flash sector erasing with secure erase.
jnwatson 10y ago
Perhaps I'm misunderstanding you, but trim doesn't secure delete anything. It merely indicates to the SSD that the sector is unused so that it can reap the underlying NAND location the next time it decides to garbage collect. In other words, the data is still there in the flash, but only the SSD firmware (and physical access) has access to it.
- emcq 10y ago
  I think the reality is that some blocks will be lazily erased. I believe modern flash firmware periodically GCs in the background to prevent write amplification. If this is triggered you may overwrite data. NAND requires an erase before write so I wouldn't be surprised if some controllers are lazily erasing blocks to get better long term write speeds and prevent GC hiccups.
- throwaway7767 10y ago
  The reason for marking it as erased is so the firmware can physically erase the flash sector. It could happen immediately, a minute later or next week. But a well-written SSD firmware will try to erase blocks any time it's not busy doing something else, as erasing blocks takes way longer than writing them. You might be able to recover something from the physical flash, but there's definitely no guarantees.
  - K0SM0S 10y ago
    > erasing blocks takes way longer than writing them Honest question, could you elaborate on that? Intuitively I would've thought writing and erasing are _the same_ from a physical standpoint, insofar as "erasing" means writing zeros.
    - colanderman 10y ago
      There are two operations on flash memory: "erase" (set all bits to 0) and "write" (set some bits to 1). The former is expensive, the latter cheap. Note that "write" can't set bits to 0. Hence the benefit of keeping zeroed blocks around.

tixzdk 10y ago

The same question was asked on Quora: https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Story-2-during-production/answer/Oren-Jacob/comment/941166 https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...

leecarraher 10y ago
thanks! oh those destitute days of swap as an actual resource instead of just a, something must be broken, i just filled up 32 gigs of main memory.
patcheudor 10y ago
Yup, disk arrays can be quite a problem when it comes to forensics recovery. This has been a bit of a nemesis of mine over the years. Friends or family will decide to buy a single RAID solution for backup and configure it to write files across the disks for performance because they don't know any better. Four years later they'll come to me because something happened to the array like a failed or corrupted controller NVRAM and they want to recover the files. For backup I recommend mirrored single individual spinning disks, preferably in multiple locations.

mprovost 10y ago

That's really only possible on a single user system. VFX studios usually have large NFS servers (filers) typically running proprietary filesystems, and with hundreds or thousands of clients writing files simultaneously they don't get all laid down on the attached arrays in nice neat chunks representing entire files. Typically they would even try and distribute writes to multiple raid groups/arrays for a single file. Also consider the size, you were able to copy it onto an external drive to recover. Studios don't have a spare, empty filer sitting around to dump the random contents of a few racks worth of disks onto.