6 ms·

This looks great! So I will ask the obvious question - how common is the failure scenario this protects against compared to others?

simcop2387 9y ago
Depends on the media. I think this would be awesome for archival usage on optical and tape media. Or even printed out in paper, since this looks like it'd let you rescan it in any order.
- dom0 9y ago
  Tape storage normally uses either an index (linear FS or off-tape, e.g. for object storage) or a more-or-less recoverable stream format ("tar").
  - simcop2387 9y ago
    In this case it'd be like losing the index of the tape, but still needing to recover data off it.
jaclaz 9y ago
I would say rather common. The whole issue is that file carving (once filesystem structures are destroyed) works fine with contiguous data but it doesn't work (or it works only very partially, and depending on a number of factors) with any fragmented data. But I will give you a common enough example, that I have seen happen many times. The user wants to format a USB stick, but by mistake he/she selects a data volume instead. Two possibilities (under Windows later than XP): 1) user chose NOT the "/q" (cli) or "quick" (GUI) format, all data is lost forever as starting from Vista if a "full" format is cosen the whole volume will be overwritten with 00's 2) user chose to "quick" format, only the filesystem structures are recreated (blank) and - given that the volume was originally formatted on the same OS - they will be exactly where previous filesystem structure were, thus cleanly replacing them. In this latter situation all files are still there, what is lost is where they are, the address info for a contiguous file is two pieces of data, where it begins and how big in size it is, there are a number of programs that can "recognize" a file (or its beginning) by its "signature" (usually a header, but sometimes a combination of header and footer), among them the excellent TriD tool by the same Author Marco Pontello, and a large number of file formats do include some info on the filesize, and there are slightly more sophisticated programs that can usually recognize if data belongs to a given filetype. The address info of a fragmented data is as many starting points and as many extensions as the fragments, and since a fragment - bar the first one - have no headers it is extremely difficult to find all the fragments of a file (and knowing to which file a fragment belongs) and re-build the file by re-assembling the fragments in the right order, often impossible. In a nutshell, if the volume was only made of contiguous files, they can be recovered 100% or very near 100%, only losing their filenames and their path in the "previous" filesystem structure. BUT any file that was fragmented will either be lost completely or will need (in some cases only this is possible) manual reconstruction, something that may take days or weeks of work and very often with only very scarce or partial success. The idea of a "self-referencing" archive with sector-level granularity is simply great, you won't ever need it, but if you do, it could be a lifesaver.
- dr_zoidberg 9y ago
  When I was younger I worked a while in file carving problems and algorithms, and while your take on it is accurate, it's not complete. There are algorithms that can do what you call "manual reconstruction" in a more or less automated way. Of course, as with everything that is automated, there are edge cases where it doesn't work correctly, but it'll save you a lot of time. What has been discussed here and in the OC, is mostly header-footer carving, but file-structure-based carving has been around for a long time already (I think Foremost included it in 2005/2006, and PhotoRec supports it for some formats). Using the file command (*NIX) or TRiD to find the headers is a bit of a hack, and useful only if you want to sit a long time with dd (or an equivalent tool) and direct yourself the carving process. As for the fragmentation issue, there used to be around a great program called "Adroit Photo Forensic" which implemented the amazing SmartCarving(TM) approach (academically known as graph-theoretic-carving) in which the disk is scanned, pools of segments by file format are populated, and then a graph is constructed and analyzed with a Dijkstra's shortest-path variant which works on multiple paths from multiple starts-to-ends. Very interesting, though a bit computationally expensive. Plus you need an extremely precise similarity metric between the segments, which was actually the weak-spot for this. However, it seems as if the Adroit product has died (I can only find mirrors now which host old versions of it), and as it was a try-and-buy program I don't think you'd be able to use it now. Thing is, there are easier ways to handle fragmentation. For example, PhotoRec itself lets you pass an agressive validation step, to discard most of the "probably wrong" recovered files, and then dump the remaining to an alternative disk image, on which you can try carving again. Of course, this iterative process takes time, but it can work, and of course it can be automated (and thus, saving you from manual intervention). While my work has taken me away from it, I never quite understood why the digital forensics community "forgot" about file carving, or disregarded it as an important problem -- it seems as if there are still the same issues around as when I worked on it, while having less tools available. As for the SeqBox format, more than an alternative to this tools, I find it as an interesting complement to them.
  - jaclaz 9y ago
    Sure, I tested a couple of times Adroit, anyway it is (was) only about some specific file formats (photos, aka jpeg's), and even the Authors' "pitch" was about (only) 20% more photos recovered, I remember not being that much impressed by the results of the actual tests (but maybe the devices on which I tested it at the time also suffered from other forms of corruption): https://web-beta.archive.org/web/20120313073659/http://digital-assembly.com/products/adroit-photo-forensics/ https://web-beta.archive.org/web/20120313073659/http://digit... https://web-beta.archive.org/web/20120806031446/http://photo-recovery.info:80/ https://web-beta.archive.org/web/20120806031446/http://photo... Some info about Smart Carving (for those interested): http://www.forensicswiki.org/wiki/File_Carving:SmartCarving http://www.forensicswiki.org/wiki/File_Carving:SmartCarving and GuidedCarving: http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving I am pretty sure that a similar tool (specific for photos/jpeg's) would be very useful, but extending the same principles to different file formats is - as I see it - extremely hard and in a large number of cases simply impossible (due to the actual structure of the file format itself).
danblick 9y ago
I wonder if maybe it's intended for use when the filesystem is intentionally wiped out, maybe so you can deny any data exists on a disk at all. (Like anti-forensics?) I don't know, I've never really wanted to do anything like that. https://en.wikipedia.org/wiki/Anti-computer_forensics https://en.wikipedia.org/wiki/Anti-computer_forensics

simcop2387 9y ago

Depends on the media. I think this would be awesome for archival usage on optical and tape media. Or even printed out in paper, since this looks like it'd let you rescan it in any order.

dom0 9y ago
Tape storage normally uses either an index (linear FS or off-tape, e.g. for object storage) or a more-or-less recoverable stream format ("tar").
- simcop2387 9y ago
  In this case it'd be like losing the index of the tape, but still needing to recover data off it.

jaclaz 9y ago

I would say rather common. The whole issue is that file carving (once filesystem structures are destroyed) works fine with contiguous data but it doesn't work (or it works only very partially, and depending on a number of factors) with any fragmented data. But I will give you a common enough example, that I have seen happen many times. The user wants to format a USB stick, but by mistake he/she selects a data volume instead. Two possibilities (under Windows later than XP): 1) user chose NOT the "/q" (cli) or "quick" (GUI) format, all data is lost forever as starting from Vista if a "full" format is cosen the whole volume will be overwritten with 00's 2) user chose to "quick" format, only the filesystem structures are recreated (blank) and - given that the volume was originally formatted on the same OS - they will be exactly where previous filesystem structure were, thus cleanly replacing them. In this latter situation all files are still there, what is lost is where they are, the address info for a contiguous file is two pieces of data, where it begins and how big in size it is, there are a number of programs that can "recognize" a file (or its beginning) by its "signature" (usually a header, but sometimes a combination of header and footer), among them the excellent TriD tool by the same Author Marco Pontello, and a large number of file formats do include some info on the filesize, and there are slightly more sophisticated programs that can usually recognize if data belongs to a given filetype. The address info of a fragmented data is as many starting points and as many extensions as the fragments, and since a fragment - bar the first one - have no headers it is extremely difficult to find all the fragments of a file (and knowing to which file a fragment belongs) and re-build the file by re-assembling the fragments in the right order, often impossible. In a nutshell, if the volume was only made of contiguous files, they can be recovered 100% or very near 100%, only losing their filenames and their path in the "previous" filesystem structure. BUT any file that was fragmented will either be lost completely or will need (in some cases only this is possible) manual reconstruction, something that may take days or weeks of work and very often with only very scarce or partial success. The idea of a "self-referencing" archive with sector-level granularity is simply great, you won't ever need it, but if you do, it could be a lifesaver.

dr_zoidberg 9y ago
When I was younger I worked a while in file carving problems and algorithms, and while your take on it is accurate, it's not complete. There are algorithms that can do what you call "manual reconstruction" in a more or less automated way. Of course, as with everything that is automated, there are edge cases where it doesn't work correctly, but it'll save you a lot of time. What has been discussed here and in the OC, is mostly header-footer carving, but file-structure-based carving has been around for a long time already (I think Foremost included it in 2005/2006, and PhotoRec supports it for some formats). Using the file command (*NIX) or TRiD to find the headers is a bit of a hack, and useful only if you want to sit a long time with dd (or an equivalent tool) and direct yourself the carving process. As for the fragmentation issue, there used to be around a great program called "Adroit Photo Forensic" which implemented the amazing SmartCarving(TM) approach (academically known as graph-theoretic-carving) in which the disk is scanned, pools of segments by file format are populated, and then a graph is constructed and analyzed with a Dijkstra's shortest-path variant which works on multiple paths from multiple starts-to-ends. Very interesting, though a bit computationally expensive. Plus you need an extremely precise similarity metric between the segments, which was actually the weak-spot for this. However, it seems as if the Adroit product has died (I can only find mirrors now which host old versions of it), and as it was a try-and-buy program I don't think you'd be able to use it now. Thing is, there are easier ways to handle fragmentation. For example, PhotoRec itself lets you pass an agressive validation step, to discard most of the "probably wrong" recovered files, and then dump the remaining to an alternative disk image, on which you can try carving again. Of course, this iterative process takes time, but it can work, and of course it can be automated (and thus, saving you from manual intervention). While my work has taken me away from it, I never quite understood why the digital forensics community "forgot" about file carving, or disregarded it as an important problem -- it seems as if there are still the same issues around as when I worked on it, while having less tools available. As for the SeqBox format, more than an alternative to this tools, I find it as an interesting complement to them.
- jaclaz 9y ago
  Sure, I tested a couple of times Adroit, anyway it is (was) only about some specific file formats (photos, aka jpeg's), and even the Authors' "pitch" was about (only) 20% more photos recovered, I remember not being that much impressed by the results of the actual tests (but maybe the devices on which I tested it at the time also suffered from other forms of corruption): https://web-beta.archive.org/web/20120313073659/http://digital-assembly.com/products/adroit-photo-forensics/ https://web-beta.archive.org/web/20120313073659/http://digit... https://web-beta.archive.org/web/20120806031446/http://photo-recovery.info:80/ https://web-beta.archive.org/web/20120806031446/http://photo... Some info about Smart Carving (for those interested): http://www.forensicswiki.org/wiki/File_Carving:SmartCarving http://www.forensicswiki.org/wiki/File_Carving:SmartCarving and GuidedCarving: http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving I am pretty sure that a similar tool (specific for photos/jpeg's) would be very useful, but extending the same principles to different file formats is - as I see it - extremely hard and in a large number of cases simply impossible (due to the actual structure of the file format itself).

danblick 9y ago

I wonder if maybe it's intended for use when the filesystem is intentionally wiped out, maybe so you can deny any data exists on a disk at all. (Like anti-forensics?) I don't know, I've never really wanted to do anything like that. https://en.wikipedia.org/wiki/Anti-computer_forensics https://en.wikipedia.org/wiki/Anti-computer_forensics