6 ms·

I always thought NTFS was declared to be relatively immune from fragmentation problems. That’s why there was no in-built defragmenting tool for NTFS-formatted d

by qubex 9y ago

I always thought NTFS was declared to be relatively immune from fragmentation problems. That’s why there was no in-built defragmenting tool for NTFS-formatted drives until much later on.

executesorder66 9y ago
NTFS is the only filesystem I've ever had to defrag.
- dspillett 9y ago
  Either you never used FAT* or were very very lucky! It (and other old, naive, filesystems) used a pure first-available-block strategy that almost invites pathological fragmentation especially in a multi tasking (or worse, multi-user) environment. Also, I've seen ext* filesystems just as badly fragmented as NTFS ones in similar environments. One key problem I've seen on NTFS volumes in the past is with those that were converted from FAT (IIRC old Windows NT installations would do this even for fresh installs: install on FAT* for the first stage then convert to NTFS before the second) - the conversion process didn't move files around when converting to 4KByte blocks from larger ones so left a small chunk of free space after every file.
  - executesorder66 9y ago
    I've used FAT32, and yeah. I guess I got lucky. > Also, I've seen ext* filesystems just as badly fragmented as NTFS ones in similar environments. What kind of environment is that? I've rewritten terabytes of data over the same blocks, and never had any issues.
    - dspillett 9y ago
      > rewritten terabytes of data over the same blocks It isn't about the amount of data throughput or the reuse of blocks: it is how files are allocated space as they grow, particularly when there is concurrent activity, and how free space fragments as files shrink or, more likely, are removed. > What kind of environment is that? Busy near-full write-heavy multi-user file servers particularly. Multitasking exacerbates the "how files grow" issue, multiple users exacerbates both, and multiple concurrent users multiplies the issue further, being near full means that data fragmentation countermeasures are circumvented because due to worsened free-space fragmentation (all the space left is in small units). Ignoring a few pathological cases (that are usually unrealistic thought experiments), if the filesystem never gets particularly full then neither NTFS nor ext* are likely to fragment significantly. But in the circumstances mentioned above both are just as likely to. Their key mechanic for avoiding data fragmentation (keeping some free space after files where possible to allow for growth) is very similar, other technicalities where they vary more widely (like storing small objects in directory structures directly or the MFT) have an effect, but a much smaller effect than that attributable to the methods they share. Of course even some near-full multi-use volumes aren't going to fragment significantly at the filesystem level. If a volume is hosting large database files then they are likely to be reasonably contiguous from the filesystem's point of view and fragmentation issues within the data files is massively more significant. Also if most of your concurrent users are only reading (perhaps a media archive system maintained my a small team, perhaps one person, or just at most one person at any given time usually, but accessed by many?) they can be ignored from a fragmentation PoV (readers aren't going to affect data layout unless you have active writers and some exotic MVCC arrangement going on at the filesystem level).
AnIdiotOnTheNet 9y ago
The allocation strategy in use by the OS doesn't necessarily have any relation to the filesystem it is being used with. IIRC, NTFS being considered "immune" to fragmentation concerns had a lot more to do with a shift in the Windows allocation strategy at about the same time than anything in the filesystem itself.
- dspillett 9y ago
  The allocation strategy is generally part of the filesystem design, not the OS design. NTFS was properly designed with real use cases in mind to avoid fragmentation where possible. Earlier Windows filesystems (FAT) were designed to just do the job simply. You can* use a more effective strategy with FAT* but you need to jump through some hoops as it doesn't have any built-in structure to help. It is far more practical to just ask your users to use a more modern filesystem, except for smaller removable devices where the issue is far less significant anyway.
  - jaclaz 9y ago
    >NTFS was properly designed with real use cases in mind to avoid fragmentation where possible. Once said that NTFS was very properly designed (and still works fine after some 25 years with only a very few modifications and on relatively huge volumes) it does have a few "strange" characteristics. JFYI, a strange case when there was apparently no solution (due to some queer placement of the NTFS filesstem structures), a 6 Gb file couldn't be stored contiguous on a 8 Gb USB stick (and of course FAT32 wouldn't have allowed a bigger than 4 Gb file): http://reboot.pro/topic/18022-error-60-file-for-disk-emulation/ http://reboot.pro/topic/18022-error-60-file-for-disk-emulati...
    - dspillett 9y ago
      > due to some queer placement of the NTFS filesstem structures It isn't really queer: some core, regularly accessed, structures are deliberately in the middle of the filesystem to reduce head movements in the common use case of it being a single volume (or the most significant volume for the currently IO tasks) on a spinning-disk-and-floating-heads based drive. This essentially splits free space in two from the very start so an 8Gb volume is going to start with two ~4Gb ranges of unallocated space not one single 8Gb range. ext* filesystems distribute group descriptors, superblock backups, and other structures, around volumes instead of keeping them at one end, for simialar reasons. See https://www.slashroot.in/understanding-file-system-superblock-linux https://www.slashroot.in/understanding-file-system-superbloc... and much other online documentation. (though spreading the superblocks around isn't a performance thing, it is to reduce the damage potentially caused by localised physical disk corruption) The queer part is software coupling itself so tightly with the storage layer such that it can't cope with a file not being physically 100% contiguous. The example in that link sounds like a virtualisation method trying to be very aggressive about optimising the mapping between guest disk access and physical storage (which might convey a small performance benefit) - if you are optimising like that then you are probably better off trying to use raw block devices rather than files on a host filesystem.
dspillett 9y ago
> thought NTFS was declared to be relatively immune from fragmentation problems The operative word there there is relatively. In terms of main file content storage it did (and still does) a very good job, especially relative to FAT's naive first-available-block allocation strategy, until the volume gets near-full. How much is "near full" very much depends on your local use cases (rules of thumb like "the larger of 10% or 20Gb" are banded about, but they are just myth-like guesswork at an old-wive's-tale level IMO as they are usually traceable back to times when volumes were much smaller and common data patterns vastly different). Once volumes are near-full not much helps - this is still true for modern variants of NTFS, ext, and any other filesystem (I'm not counting filesystems with "active defragmentation" as that is essentially a filesystem plus a tool). Some have tweaks that help reduce the effects (NTFS being able to store small files in directory blocks directly for instance) but it is not practical to completely rid yourself of them especially in pathological cases. Furthermore once a file is fragmented it will stay that way until removed & replaced or acted upon by a defragment process on the filesystem, and deleting fragmented files may leave the free space around them a mess of small blocks so delete & reallocate may not help on a near-full filesystem. Any filesystem that claims to be fragmentation free without a list of caveats attached to that claim is snake-oil.

by qubex 9y ago

I always thought NTFS was declared to be relatively immune from fragmentation problems. That’s why there was no in-built defragmenting tool for NTFS-formatted drives until much later on.

executesorder66 9y ago

NTFS is the only filesystem I've ever had to defrag.

dspillett 9y ago
Either you never used FAT* or were very very lucky! It (and other old, naive, filesystems) used a pure first-available-block strategy that almost invites pathological fragmentation especially in a multi tasking (or worse, multi-user) environment. Also, I've seen ext* filesystems just as badly fragmented as NTFS ones in similar environments. One key problem I've seen on NTFS volumes in the past is with those that were converted from FAT (IIRC old Windows NT installations would do this even for fresh installs: install on FAT* for the first stage then convert to NTFS before the second) - the conversion process didn't move files around when converting to 4KByte blocks from larger ones so left a small chunk of free space after every file.
- executesorder66 9y ago
  I've used FAT32, and yeah. I guess I got lucky. > Also, I've seen ext* filesystems just as badly fragmented as NTFS ones in similar environments. What kind of environment is that? I've rewritten terabytes of data over the same blocks, and never had any issues.
  - dspillett 9y ago
    > rewritten terabytes of data over the same blocks It isn't about the amount of data throughput or the reuse of blocks: it is how files are allocated space as they grow, particularly when there is concurrent activity, and how free space fragments as files shrink or, more likely, are removed. > What kind of environment is that? Busy near-full write-heavy multi-user file servers particularly. Multitasking exacerbates the "how files grow" issue, multiple users exacerbates both, and multiple concurrent users multiplies the issue further, being near full means that data fragmentation countermeasures are circumvented because due to worsened free-space fragmentation (all the space left is in small units). Ignoring a few pathological cases (that are usually unrealistic thought experiments), if the filesystem never gets particularly full then neither NTFS nor ext* are likely to fragment significantly. But in the circumstances mentioned above both are just as likely to. Their key mechanic for avoiding data fragmentation (keeping some free space after files where possible to allow for growth) is very similar, other technicalities where they vary more widely (like storing small objects in directory structures directly or the MFT) have an effect, but a much smaller effect than that attributable to the methods they share. Of course even some near-full multi-use volumes aren't going to fragment significantly at the filesystem level. If a volume is hosting large database files then they are likely to be reasonably contiguous from the filesystem's point of view and fragmentation issues within the data files is massively more significant. Also if most of your concurrent users are only reading (perhaps a media archive system maintained my a small team, perhaps one person, or just at most one person at any given time usually, but accessed by many?) they can be ignored from a fragmentation PoV (readers aren't going to affect data layout unless you have active writers and some exotic MVCC arrangement going on at the filesystem level).

AnIdiotOnTheNet 9y ago

The allocation strategy in use by the OS doesn't necessarily have any relation to the filesystem it is being used with. IIRC, NTFS being considered "immune" to fragmentation concerns had a lot more to do with a shift in the Windows allocation strategy at about the same time than anything in the filesystem itself.

dspillett 9y ago
The allocation strategy is generally part of the filesystem design, not the OS design. NTFS was properly designed with real use cases in mind to avoid fragmentation where possible. Earlier Windows filesystems (FAT) were designed to just do the job simply. You can* use a more effective strategy with FAT* but you need to jump through some hoops as it doesn't have any built-in structure to help. It is far more practical to just ask your users to use a more modern filesystem, except for smaller removable devices where the issue is far less significant anyway.
- jaclaz 9y ago
  >NTFS was properly designed with real use cases in mind to avoid fragmentation where possible. Once said that NTFS was very properly designed (and still works fine after some 25 years with only a very few modifications and on relatively huge volumes) it does have a few "strange" characteristics. JFYI, a strange case when there was apparently no solution (due to some queer placement of the NTFS filesstem structures), a 6 Gb file couldn't be stored contiguous on a 8 Gb USB stick (and of course FAT32 wouldn't have allowed a bigger than 4 Gb file): http://reboot.pro/topic/18022-error-60-file-for-disk-emulation/ http://reboot.pro/topic/18022-error-60-file-for-disk-emulati...
  - dspillett 9y ago
    > due to some queer placement of the NTFS filesstem structures It isn't really queer: some core, regularly accessed, structures are deliberately in the middle of the filesystem to reduce head movements in the common use case of it being a single volume (or the most significant volume for the currently IO tasks) on a spinning-disk-and-floating-heads based drive. This essentially splits free space in two from the very start so an 8Gb volume is going to start with two ~4Gb ranges of unallocated space not one single 8Gb range. ext* filesystems distribute group descriptors, superblock backups, and other structures, around volumes instead of keeping them at one end, for simialar reasons. See https://www.slashroot.in/understanding-file-system-superblock-linux https://www.slashroot.in/understanding-file-system-superbloc... and much other online documentation. (though spreading the superblocks around isn't a performance thing, it is to reduce the damage potentially caused by localised physical disk corruption) The queer part is software coupling itself so tightly with the storage layer such that it can't cope with a file not being physically 100% contiguous. The example in that link sounds like a virtualisation method trying to be very aggressive about optimising the mapping between guest disk access and physical storage (which might convey a small performance benefit) - if you are optimising like that then you are probably better off trying to use raw block devices rather than files on a host filesystem.

dspillett 9y ago

> thought NTFS was declared to be relatively immune from fragmentation problems The operative word there there is relatively. In terms of main file content storage it did (and still does) a very good job, especially relative to FAT's naive first-available-block allocation strategy, until the volume gets near-full. How much is "near full" very much depends on your local use cases (rules of thumb like "the larger of 10% or 20Gb" are banded about, but they are just myth-like guesswork at an old-wive's-tale level IMO as they are usually traceable back to times when volumes were much smaller and common data patterns vastly different). Once volumes are near-full not much helps - this is still true for modern variants of NTFS, ext, and any other filesystem (I'm not counting filesystems with "active defragmentation" as that is essentially a filesystem plus a tool). Some have tweaks that help reduce the effects (NTFS being able to store small files in directory blocks directly for instance) but it is not practical to completely rid yourself of them especially in pathological cases. Furthermore once a file is fragmented it will stay that way until removed & replaced or acted upon by a defragment process on the filesystem, and deleting fragmented files may leave the free space around them a mess of small blocks so delete & reallocate may not help on a near-full filesystem. Any filesystem that claims to be fragmentation free without a list of caveats attached to that claim is snake-oil.