4 ms·

I'm surprised that the lost file was simple continuous area on the raw device, and would've expected the pieces to be intermingled with data from other files.

I'm surprised that the lost file was simple continuous area on the raw device, and would've expected the pieces to be intermingled with data from other files. Maybe if the disk is being filled for the first time, space for the file is allocated in one go (as opposed to gradually appending to the end of the file), and there's a large enough unfragmented area of free space, and the disk isn't very busy, the file would be allocated like this on the device. But I sure wouldn't count on it. Not sure about the flv format, and how it recovers from errors, but in the worst case it might look like the correct file but might have pieces from other videos mixed in.

marcan_42 6y ago
Modern file systems try fairly hard to keep files contiguous for performance, and if recordings are always written and deleted semi sequentially, you'd expect the disk to end up fairly not fragmented. That said, two or three big fragments would be pretty common in this situation.
- Beldin 6y ago
  We recently performed a large study on fragmentation on the NTFS file system [1]. On NTFS, we found that slightly over a quarter (26.8%) of FLV files were fragmented. So if EXT4 performs similarly, it would be far more common not to have fragments. [1] http://www.open.ou.nl/hjo/papers/di20.pdf http://www.open.ou.nl/hjo/papers/di20.pdf (Paper accepted but conference and therefore publication delayed due to corona)
  - jeroenhd 6y ago
    NTFS is known to allow more fragmentation than ext4 (as long as there is plenty of disk space left; full ext4 volumes will fragment). This is one reason why running Windows from a hard drive requires occasional defragmentation (done automatically in the background without you noticing) while Linux will happily run without any defragmentation at all in most cases. Fragmentation will still happen, of course, but instead of preferring write performance and accepting fragmentation as a consequence, like NTFS, the ext4 system tries to avoid fragmentation where it can. Now, in the age of SSDs and flash storage, fragmentation isn't as big a problem as it used to be, but it's nice to see the optimizations done for running an OS from a hard drive make it easier to recover byte streams from ext4 than it would be to recover them from NTFS.
herpderperator 6y ago
I too was surprised that the article mentioned nothing about blocks or fragmentation. The deleted file has an inode yes but that inode also describes all the blocks that contain the data for that inode and it is no guarantee that the blocks are contiguous. It seems that was completely overlooked, but in this case it turns out everything apart from one part in the middle of the video was recovered leading me to believe that a few of the blocks were in fact fragmented.

marcan_42 6y ago

Modern file systems try fairly hard to keep files contiguous for performance, and if recordings are always written and deleted semi sequentially, you'd expect the disk to end up fairly not fragmented. That said, two or three big fragments would be pretty common in this situation.

Beldin 6y ago
We recently performed a large study on fragmentation on the NTFS file system [1]. On NTFS, we found that slightly over a quarter (26.8%) of FLV files were fragmented. So if EXT4 performs similarly, it would be far more common not to have fragments. [1] http://www.open.ou.nl/hjo/papers/di20.pdf http://www.open.ou.nl/hjo/papers/di20.pdf (Paper accepted but conference and therefore publication delayed due to corona)
- jeroenhd 6y ago
  NTFS is known to allow more fragmentation than ext4 (as long as there is plenty of disk space left; full ext4 volumes will fragment). This is one reason why running Windows from a hard drive requires occasional defragmentation (done automatically in the background without you noticing) while Linux will happily run without any defragmentation at all in most cases. Fragmentation will still happen, of course, but instead of preferring write performance and accepting fragmentation as a consequence, like NTFS, the ext4 system tries to avoid fragmentation where it can. Now, in the age of SSDs and flash storage, fragmentation isn't as big a problem as it used to be, but it's nice to see the optimizations done for running an OS from a hard drive make it easier to recover byte streams from ext4 than it would be to recover them from NTFS.

herpderperator 6y ago

I too was surprised that the article mentioned nothing about blocks or fragmentation. The deleted file has an inode yes but that inode also describes all the blocks that contain the data for that inode and it is no guarantee that the blocks are contiguous. It seems that was completely overlooked, but in this case it turns out everything apart from one part in the middle of the video was recovered leading me to believe that a few of the blocks were in fact fragmented.