5 ms·

Aren’t you selling it short? Spinny disks, especially in the days before NCQ/TCQ have a vast imbalance between reading sectors sequentially and positioning the

by anyfoo 5y ago

Aren’t you selling it short? Spinny disks, especially in the days before NCQ/TCQ have a vast imbalance between reading sectors sequentially and positioning the head to a new track. NCQ/TCQ later helped a bit by optimizing the path the head takes (With Physics!) when multiple operations are in the queue, but a completely fragmented disk drive seems like it will underperform noticeable against one where everything is laid out neatly in a linear fashion, at least in some access patterns (reading large files in one go). But I’d be interested in any real data anyway.

wenc 5y ago
I can't conduct the experiment anymore obviously, but to qualify my perceptions -- if you try to do a linear read from a 90% fragmented filesystem, performance would definitely suck and defragging would obviously provide perceptible benefit. Most of the time however, my disk was probably only about 30% fragmented (if I recall correctly from Speed Disk). I was running DOS on a FAT file system where there was no swap file or anything that would cause massive fragmentation except deleting and installing programs, and maybe some temp files created by certain programs. The performance delta for me was barely noticeable on a 40MB Seagate IDE drive.
- rwmj 5y ago
  > I can't conduct the experiment anymore obviously Why not? $ virt-make-fs --type=msdos /usr/share/doc /var/tmp/disk.img $ export file=/var/tmp/disk.img $ nbdkit eval \ after_fork=' echo 0 > $tmpdir/read ' \ get_size=' stat -Lc %s $file ' \ pread=' diff=$(($4 - `cat $tmpdir/read`)) echo $4 > $tmpdir/read if [ $diff -lt -10000 ] || [ $diff -gt 10000 ]; then # simulate a seek sleep 0.1 fi dd if=$file skip=$4 count=$3 iflag=count_bytes,skip_bytes ' $ sudo nbd-client localhost /dev/nbd0 $ sudo mount /dev/nbd0 /tmp/mnt The tricky thing is probably making a deliberately fragmented disk image for testing using modern tools.
- sumtechguy 5y ago
  In Vista defragging was one of my goto's for getting the thing to act nicely. I would use process monitor from sysinternals to measure which files are loaded as the system boots. Then use jkdefrag/mydefrag to take said list and rebuild drive into different zones. Try to put small files that are used together in the same area of the disk. Put huge things like 'isos/vhds/acrivezips' etc near the end of the drive. Put small frequently used files near the front in used order if possible. Smash out any gaps if possible. This helps NTFS from creating more fragments. Defrag any files that 7zip unarchives immediately as it has a bad habit of creating highly fragmented files (contig from sysinternals). Try to put file index in its own 'area' as one reason it runs so badly is the drive fragmentation it causes to itself. It got so bad I would put the index on its own partition sometimes. Try to put files that fragment frequently in their own 'zone' as NTFS will tend to pick the next available gap to where the file is (not always). That made a real noticeable difference in perceived speed. These days with nvme and ssd I might just defrag the files themselves once a year if I feel like it. There is a difference but the perceived is pretty much not there anymore. The difference is there is a chain of sectors in the MFT that windows will have to traverse if you ask for something in the middle of a file. That is about the only thing you save with newer drives. ext4 has a similar issue. But it seems to be better about picking spots where it will not fragment. It will however fragment badly in low space conditions. Most filesystems will.
rzzzt 5y ago
With FAT, you are constantly jumping between the table at the beginning of the disk to get the next cluster number and the data, unless disk caching pulled it in RAM.
- chj 5y ago
  But a defrag FAT jumps much much less, because the cluster slots allocated to the same file will be close to each other, often in the same sector. I would be surprised if DOS doesn't optimize for this case.
- ygra 5y ago
  I remember that being a reason why the MFT in NTFS is positioned in the middle of the drive so that on average the head has to travel less. And NTFS also supports putting small file contents directly into the MFT entry, which also cuts down on head movement. Nowadays all that probably is moot and a relic from ancient times (although I still have enough hard drives running at home).
  - jaclaz 5y ago
    Not really. First thing the middle of the volume/partition is not the middle of the disk in any multi-partitioned disk, then NTFS $MFT is not (and never has been) in the middle of the partition, its location varies depending on the size of the volume/partiton but above a certain size, around 5/6 GB it is at a fixed offset, usually on LCN #786432 which - on a normal 8 KB/cluster NTFS volume - amounts to offset 6,291,456 sectors or 6,442,450,944 bytes, rarely the middle of the volume, unless its size is around 13 GB.
    - fuzzfactor 5y ago
      Have you found the difference between formatting NTFS in older Windows versions compared to the recent releases? Seems to me the MFT was in the middle of volumes commonly in older versions, and lately much closer to the beginning of the volume. Could have something to do with the feature of shrinking a volume which became more common, and you can now more often shrink an NTFS volume by more than half, which was often impossible in earlier years.

by anyfoo 5y ago

wenc 5y ago

I can't conduct the experiment anymore obviously, but to qualify my perceptions -- if you try to do a linear read from a 90% fragmented filesystem, performance would definitely suck and defragging would obviously provide perceptible benefit. Most of the time however, my disk was probably only about 30% fragmented (if I recall correctly from Speed Disk). I was running DOS on a FAT file system where there was no swap file or anything that would cause massive fragmentation except deleting and installing programs, and maybe some temp files created by certain programs. The performance delta for me was barely noticeable on a 40MB Seagate IDE drive.

rwmj 5y ago
> I can't conduct the experiment anymore obviously Why not? $ virt-make-fs --type=msdos /usr/share/doc /var/tmp/disk.img $ export file=/var/tmp/disk.img $ nbdkit eval \ after_fork=' echo 0 > $tmpdir/read ' \ get_size=' stat -Lc %s $file ' \ pread=' diff=$(($4 - `cat $tmpdir/read`)) echo $4 > $tmpdir/read if [ $diff -lt -10000 ] || [ $diff -gt 10000 ]; then # simulate a seek sleep 0.1 fi dd if=$file skip=$4 count=$3 iflag=count_bytes,skip_bytes ' $ sudo nbd-client localhost /dev/nbd0 $ sudo mount /dev/nbd0 /tmp/mnt The tricky thing is probably making a deliberately fragmented disk image for testing using modern tools.
sumtechguy 5y ago
In Vista defragging was one of my goto's for getting the thing to act nicely. I would use process monitor from sysinternals to measure which files are loaded as the system boots. Then use jkdefrag/mydefrag to take said list and rebuild drive into different zones. Try to put small files that are used together in the same area of the disk. Put huge things like 'isos/vhds/acrivezips' etc near the end of the drive. Put small frequently used files near the front in used order if possible. Smash out any gaps if possible. This helps NTFS from creating more fragments. Defrag any files that 7zip unarchives immediately as it has a bad habit of creating highly fragmented files (contig from sysinternals). Try to put file index in its own 'area' as one reason it runs so badly is the drive fragmentation it causes to itself. It got so bad I would put the index on its own partition sometimes. Try to put files that fragment frequently in their own 'zone' as NTFS will tend to pick the next available gap to where the file is (not always). That made a real noticeable difference in perceived speed. These days with nvme and ssd I might just defrag the files themselves once a year if I feel like it. There is a difference but the perceived is pretty much not there anymore. The difference is there is a chain of sectors in the MFT that windows will have to traverse if you ask for something in the middle of a file. That is about the only thing you save with newer drives. ext4 has a similar issue. But it seems to be better about picking spots where it will not fragment. It will however fragment badly in low space conditions. Most filesystems will.

rzzzt 5y ago

With FAT, you are constantly jumping between the table at the beginning of the disk to get the next cluster number and the data, unless disk caching pulled it in RAM.

chj 5y ago
But a defrag FAT jumps much much less, because the cluster slots allocated to the same file will be close to each other, often in the same sector. I would be surprised if DOS doesn't optimize for this case.
ygra 5y ago
I remember that being a reason why the MFT in NTFS is positioned in the middle of the drive so that on average the head has to travel less. And NTFS also supports putting small file contents directly into the MFT entry, which also cuts down on head movement. Nowadays all that probably is moot and a relic from ancient times (although I still have enough hard drives running at home).
- jaclaz 5y ago
  Not really. First thing the middle of the volume/partition is not the middle of the disk in any multi-partitioned disk, then NTFS $MFT is not (and never has been) in the middle of the partition, its location varies depending on the size of the volume/partiton but above a certain size, around 5/6 GB it is at a fixed offset, usually on LCN #786432 which - on a normal 8 KB/cluster NTFS volume - amounts to offset 6,291,456 sectors or 6,442,450,944 bytes, rarely the middle of the volume, unless its size is around 13 GB.
  - fuzzfactor 5y ago
    Have you found the difference between formatting NTFS in older Windows versions compared to the recent releases? Seems to me the MFT was in the middle of volumes commonly in older versions, and lately much closer to the beginning of the volume. Could have something to do with the feature of shrinking a volume which became more common, and you can now more often shrink an NTFS volume by more than half, which was often impossible in earlier years.