3 ms·
When I was younger I worked a while in file carving problems and algorithms, and while your take on it is accurate, it's not complete. There are algorithms that
by dr_zoidberg 10y ago
When I was younger I worked a while in file carving problems and algorithms, and while your take on it is accurate, it's not complete. There are algorithms that can do what you call "manual reconstruction" in a more or less automated way. Of course, as with everything that is automated, there are edge cases where it doesn't work correctly, but it'll save you a lot of time.
What has been discussed here and in the OC, is mostly header-footer carving, but file-structure-based carving has been around for a long time already (I think Foremost included it in 2005/2006, and PhotoRec supports it for some formats). Using the file command (*NIX) or TRiD to find the headers is a bit of a hack, and useful only if you want to sit a long time with dd (or an equivalent tool) and direct yourself the carving process.
As for the fragmentation issue, there used to be around a great program called "Adroit Photo Forensic" which implemented the amazing SmartCarving(TM) approach (academically known as graph-theoretic-carving) in which the disk is scanned, pools of segments by file format are populated, and then a graph is constructed and analyzed with a Dijkstra's shortest-path variant which works on multiple paths from multiple starts-to-ends. Very interesting, though a bit computationally expensive. Plus you need an extremely precise similarity metric between the segments, which was actually the weak-spot for this. However, it seems as if the Adroit product has died (I can only find mirrors now which host old versions of it), and as it was a try-and-buy program I don't think you'd be able to use it now.
Thing is, there are easier ways to handle fragmentation. For example, PhotoRec itself lets you pass an agressive validation step, to discard most of the "probably wrong" recovered files, and then dump the remaining to an alternative disk image, on which you can try carving again. Of course, this iterative process takes time, but it can work, and of course it can be automated (and thus, saving you from manual intervention).
While my work has taken me away from it, I never quite understood why the digital forensics community "forgot" about file carving, or disregarded it as an important problem -- it seems as if there are still the same issues around as when I worked on it, while having less tools available.
As for the SeqBox format, more than an alternative to this tools, I find it as an interesting complement to them.
- jaclaz 10y agoSure, I tested a couple of times Adroit, anyway it is (was) only about some specific file formats (photos, aka jpeg's), and even the Authors' "pitch" was about (only) 20% more photos recovered, I remember not being that much impressed by the results of the actual tests (but maybe the devices on which I tested it at the time also suffered from other forms of corruption): https://web-beta.archive.org/web/20120313073659/http://digital-assembly.com/products/adroit-photo-forensics/ https://web-beta.archive.org/web/20120313073659/http://digit... https://web-beta.archive.org/web/20120806031446/http://photo-recovery.info:80/ https://web-beta.archive.org/web/20120806031446/http://photo... Some info about Smart Carving (for those interested): http://www.forensicswiki.org/wiki/File_Carving:SmartCarving http://www.forensicswiki.org/wiki/File_Carving:SmartCarving and GuidedCarving: http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving http://www.forensicswiki.org/wiki/File_Carving:GuidedCarving I am pretty sure that a similar tool (specific for photos/jpeg's) would be very useful, but extending the same principles to different file formats is - as I see it - extremely hard and in a large number of cases simply impossible (due to the actual structure of the file format itself).
- dr_zoidberg 10y agoDepends on the format, for example PNG has checksums per segment which make it extremely easy to find if the segment has been correctly rebuilt. Same goes for ZIP and RAR, and JPEGs can be matched against the Huffman tables for invalid byte sequences. Also for JPEG there was a (sort of experimental) carver that validated the image against the thumbnail, and then decided if it was correctly reconstructed or not (and maybe it also helped to choose the correct block when there was an error). Still, most of this tools never left the alpha stage, and ended as curiositys presented in a conference and then left to rot.
- jaclaz 10y agoYes :), it is queer how some programs/tools that should exist and of which we do have the basic algorithms were never properly written/developed while we have tens or hundreds of tools similar among them missing these functionalities. Another kind of tool that was never properly developed is interactive jpeg repair, besides the good jpegsnoop: http://www.impulseadventure.com/photo/jpeg-snoop.html http://www.impulseadventure.com/photo/jpeg-snoop.html (which has only some of these functionalities) there are a handful of tools that are of little or "very narrow" use, but nothing much effective at rebuilding a damaged jpeg.
- dr_zoidberg 10y agoI've seen this in the steganography/steganalysis "community": almost every time a script or program is written, its purpose is to rapidly fix the needs by some highly specialized expert who deems that development too simple (that is, to himself) to further develop into a full-fledged tool.