11 ms·

Why Linux Doesn’t Need Defragmenting

96 points by plessthanpt05 14y ago

carterschonwald 14y ago
this doesn't sound right... but maybe thats because I sort of assume any "sufficiently smart file system" has sort of COW (copy on write) like append semantics plus some sort of amortized compaction similar to a reasonable point in the GC design space of copying vs mark-sweep. (realistically most File systems don't do something like that, but its what I imagine "should be" the case)
rbanffy 14y ago
"If fragmentation does occur, the file system will attempt to move the files around to reduce fragmentation in normal use" Can anyone give more details on this? Oh... And the love in the comments on the original article is quite moving.
- tedunangst 14y ago
  I can only speak to how BSD does it, but if you append 8k (any small amount) to a file that doesn't have adjacent space available, the filesystem will collect up to the last 64k and rewrite it all out in one block.
ars 14y ago
The trouble is I did use a filesystem all the way to completely full, and now it's severely fragmented (despite me resizing it larger) and I have no practical way of unfragmenting it.
- joejohnson 14y ago
  From the article: "If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk."
  - gouranga 14y ago
    Isn't that a form of defragmenting? It also contradicts the point that it is unecessary to defragment the volume.
    - dredmorbius 14y ago
      It's extremely rare to need to do this. It's generally a sign that you're running your disks / filesystems overutilized. Retaining 10-20% free space is generally encouraged. If you do elect to do the "copy out, wipe, copy in" method, you can do this, in general, while not completely losing the use of your system. There are some tricks (union mounts, remounting filesystems) which will allow you to flip between filesystems, in most cases. Specifics will depend on your partitioning scheme, use case, and the filesystems under consideration (usually you'd be dealing with user files under /home, spool or variable files under /var, source trees, etc.). As the article states, Linux systems generally experience very minimal fragmentation, and dynamically resolve the issue for you, unlike Windows. You can typically run a Linux system for years, given a 10-20% disk reserve, without running into appreciable fragmentation issues. The gains from any sort of manual defragmentation would be minimal compared to the cost of performing the task.
  - Lerc 14y ago
    That is true for any filesystem that needs defragging. The entire notion of a disk defragmentor is that it fixes the problem without needing to do this step. It's like saying "Our car never breaks down so you don't need to be able to repair it. Even if it does break down you can still walk."
- sltkr 14y ago
  There are tools like Shake [1] that can semi-intelligently copy fragmented files around to defragment them, which typically results in less fragments if there is enough free space on the volume. 1. http://www.vleu.net/shake/ http://www.vleu.net/shake/
- drostie 14y ago
  I had that problem at one point. I vaguely recall that I unmounted my home partition and fscked it at one point to find the fragmented inodes, then there was complicated magic to find the filenames that those inodes referred to, and then there was a script which copied them all to a tmpfs (i.e. into memory), deleted the original, and copied the copy back out. One of the goals of recent ext4 work was to get support for "online defragmentation" but I don't know to what extent it's automatic, what tools exist, et cetera. It sounds like pyfragtools also has attempted to solve this problem.
- ajross 14y ago
  Uh... attaching new media and using "tar" isn't a practical way of unfragmenting it? Hard drives are almost free in the modern world.
  - shawnz 14y ago
    When compared to having a (reasonably simple) tool which will do it for free, using only one volume, while still allowing r/w access? Yes, it's impractical.
- gcr 14y ago
  If your filesystem is completely full, fragmentation should be the last of your worries.
  - ars 14y ago
    It was full, now I expanded it, but the old data is still heavily fragmented.
trentmb 14y ago
"If you do have problems with fragmentation on Linux, you probably need a larger hard disk. If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk." So... buy a larger hard disk, or get a second disk to copy everything to instead of a defrag utility? I think I'll take the Windows approach here.
- dunecn 14y ago
  Well, in that case, Windows can't do any better. In order to defrag a filesystem, that requires free space to copy and move files around. If that space isn't there (because the HDD is nearly full), there isn't much you can do.
  - tzs 14y ago
    The Windows defrag API allows you to work at the cluster level. You only in theory need a single free cluster in order to defrag.
    - cavilling_elite 14y ago
      At this rate, it would be faster to move everything over to a new harddrive.
      - deleted 14y ago
        [deleted]
    - ZoFreX 14y ago
      It's worth noting that while the Windows defrag utility is completely useless without a lot of free space available, many of the third party defrag tools can still work (albeit much more slowly) in that situation. Of course with SSDs becoming more and more prevalent, this is becoming a non-issue anyway.
- 14y ago
Travis 14y ago
Does this imply for certain contexts, the Windows model is superior? Say you had a "write/update infrequently, read many times" system profile with 10% disk space avail. It seems like Windows would store these files closer to each other on the drive, making reads more efficient. Or would that be such a negligible difference that it's not worth mentioning?
- gouranga 14y ago
  It's pretty negligible. I genuinely can't tell the difference between NTFS and ext3 at least performance-wise. However, the locking semantics on NTFS are shitty unless you know what you are doing (i.e. use KTM/Transactional NTFS) which sometimes skews the statistics a lot.
yoasif_ 14y ago
Users can defragment files (copying files to a new location) automatically by running something like pyfragtools: http://ubuntuforums.org/showthread.php?t=169551 http://ubuntuforums.org/showthread.php?t=169551 Other filesystems like ZFS and btrfs utilize copy-on-write to virtually eliminate file fragmentation -- files are copied whenever edits are made to the file.
- wch 14y ago
  My understanding is that COW filesystems have really bad fragmentation if you're changing portions of large files -- for example, with virtual machine disk images.
  - wch 14y ago
    Actually, I just discovered that on btrfs, you can disable COW on a volume, or even on individual files. To do it on a per-file basis, you have to compile a small program available here: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg14877.html http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg1... This makes btrfs suitable for virtual machine images.
- drostie 14y ago
  Maybe ZFS/btrfs users don't care, but I would be really hesitant before using a filesystem which does not respect the Unix `shred` tool. Like real paper-shredders, I don't use them very much but when I do use them I really hope that they work.
  - icebraining 14y ago
    Why doesn't shred work on ZFS/Btrfs?
    - barrkel 14y ago
      ZFS is copy-on-write with snapshot support. If you have files you may need to shred, put them on a different file system; ZFS is oriented towards making it hard to lose data.
sciurus 14y ago
On ext2/3/4 filesystems, there are a couple handy tools for measuring fragmentation that you probably already have installed To see how fragmented a file is: filefrag -v filename To see how fragmented your free space is e2freefrag devicename http://linux.die.net/man/8/e2freefrag http://linux.die.net/man/8/e2freefrag http://linux.die.net/man/8/filefrag http://linux.die.net/man/8/filefrag To learn about the intricacies of how fragmentation can occur in ext3 and some better tools for measuring it, see the fragmentation section of Oliver Diedrich's article 'Tuning the Linux file system Ext3'. http://www.h-online.com/open/features/Fragmentation-746701.html http://www.h-online.com/open/features/Fragmentation-746701.h... Extents, multiblock allocation, and delayed allocation make ext4 more resistant to fragmentation than ext3, but the developers are still working on a defrag utility. http://kernelnewbies.org/Ext4 http://kernelnewbies.org/Ext4
ciupicri 14y ago
XFS has some sort of a defragmentation tool - xfs_fsr: xfs_fsr [-v] [-t seconds] [-f leftoff] [-m mtab] xfs_fsr [-v] [xfsdev | file] ... xfs_fsr improves the organization of mounted filesystems. The reorganization algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).
- pyre 14y ago
  xfs also has a tendency to fill up the kernel slab space when stat'ing a bunch of files/directories. This is really annoying because most "memory full" tools don't delve into the slab space to see if the usage is something that will free up automatically.
wwrap 14y ago
A lot of questions being asked here are already covered in this thread from a day ago: http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt_need_defragmenting/ http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt...
pooriaazimi 14y ago
The comment thread on this article made me appreciate HN a lot more... It reminded me of those silly schoolyard arguments.
- nitrogen 14y ago
  I was surprised to see someone still commenting about how you can't use Linux in the "real world" in 2012. So my world's not the "real world?"
naner 14y ago
How does ext* move files around to reduce fragmentation if the disk is full? I'm glad you asked. Take a look at your filesystem properties: # tune2fs -l /dev/blah In there you'll see a property called "Reserved Block Count". That is space reserved for kernel use to avoid fragmentation, continue logging, and for any other tasks the kernel/root user may need to do on a full filesystem. Users cannot write to this and your disk utilities will report your disk being full even if this reserved space is still available. I believe it is typically 5% of the partition size. If your partition is hundreds of gigabytes then the reserved portion may be tens of gigabytes (multiply the reserved blocks by the block size to get total byte size). You can change the percentage of this portion with the tune2fs -m flag. I have filled up a 500 GB backup drive with 0% reserved blocks (%5 reserved would have meant that 25GB was unusable) with no issues. Obviously, you probably don't want it set to 0% if you are constantly changing the contents of that partition.
cabirum 14y ago
1. It's not hard to implement such file allocation strategy. If it was superior, other OS'es would have adopted it long ago. 2. Scattering files all over the partition makes the longest seek times. This is actually the most inefficient solution for linux with its apps and sources (usually) consisting of thousands small files. 3. Windows does not need to be defragmented manually as it defragments itself when PC is not in use.
- bnegreve 14y ago
  > 2. Scattering files all over the partition makes the longest seek times. Yes but it isn't guaranteed that there will be less fragmentation on a windows file system. It depends on when was the last defrag, on the way file are created, and your disk occupation. Also, when you scatter files on different disks it can be more efficient because you can read several cluster simultaneously. But I guess Windows does this as well.
TwoBit 14y ago
Windows seems superior to me. It performs better due to seek time reduction, and it defrags and optimizes during idle periods. With Linux you have the worst of both worlds: it's slower and when the disk gets near full you supposedly have no choice but to re-do the disk.
thyrsus 14y ago
In 30 years working with Unix, I've never encountered a situation where "defragmentation" was the answer. File system too full? Dozens of times. So, what usage pattern am I avoiding to live such a charmed life? Anecdotally, I run a couple virtual machines on my laptop whose storage is on an external drive that sounds like Geiger counter at Fukushima Reactor 1 when the drive is busy. That is, when I've booted Windows 7. Not when I'm running anything under Windows, Windows just needs to be booted. When I'm running only a Linux VM, it drops to background level.
ams6110 14y ago
What I haven't seen mentioned is that historically, Unix (which inspired Linux) is a server/multi-user OS. All this thinking about "my" disk drives and how "my" large files are allocated is pretty irrelevant when you have dozens of users as well as system daemons accessing different files all of the time. Your drives are seeking all over the place constantly anyway. Fragmentation in such a scenario is much less of an issue.
steve8918 14y ago
So Linux DOES need to be defragmented, but the only difference is that it defragments automatically, unlike Windows which requires an external defragmentation tool. So really the title should be "Why Linux Doesn't Need Defragmentation Tools".
nitrogen 14y ago
This might be more accurately titled "Why ext2/3/4 doesn't need defragmenting." The advice given in the article: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk. does not work for ntfs-3g. Whatever strategy it uses to allocate blocks, it's terrible for streaming reads of large files (last time I checked about 1-2 years ago). So, if you're trying to use the copy-off copy-on method of defragmenting an NTFS partition, don't do it from Linux.

96 points by plessthanpt05 14y ago

this doesn't sound right... but maybe thats because I sort of assume any "sufficiently smart file system" has sort of COW (copy on write) like append semantics plus some sort of amortized compaction similar to a reasonable point in the GC design space of copying vs mark-sweep. (realistically most File systems don't do something like that, but its what I imagine "should be" the case)

rbanffy 14y ago

"If fragmentation does occur, the file system will attempt to move the files around to reduce fragmentation in normal use" Can anyone give more details on this? Oh... And the love in the comments on the original article is quite moving.

tedunangst 14y ago
I can only speak to how BSD does it, but if you append 8k (any small amount) to a file that doesn't have adjacent space available, the filesystem will collect up to the last 64k and rewrite it all out in one block.

ars 14y ago

The trouble is I did use a filesystem all the way to completely full, and now it's severely fragmented (despite me resizing it larger) and I have no practical way of unfragmenting it.

joejohnson 14y ago
From the article: "If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk."
- gouranga 14y ago
  Isn't that a form of defragmenting? It also contradicts the point that it is unecessary to defragment the volume.
  - dredmorbius 14y ago
    It's extremely rare to need to do this. It's generally a sign that you're running your disks / filesystems overutilized. Retaining 10-20% free space is generally encouraged. If you do elect to do the "copy out, wipe, copy in" method, you can do this, in general, while not completely losing the use of your system. There are some tricks (union mounts, remounting filesystems) which will allow you to flip between filesystems, in most cases. Specifics will depend on your partitioning scheme, use case, and the filesystems under consideration (usually you'd be dealing with user files under /home, spool or variable files under /var, source trees, etc.). As the article states, Linux systems generally experience very minimal fragmentation, and dynamically resolve the issue for you, unlike Windows. You can typically run a Linux system for years, given a 10-20% disk reserve, without running into appreciable fragmentation issues. The gains from any sort of manual defragmentation would be minimal compared to the cost of performing the task.
- Lerc 14y ago
  That is true for any filesystem that needs defragging. The entire notion of a disk defragmentor is that it fixes the problem without needing to do this step. It's like saying "Our car never breaks down so you don't need to be able to repair it. Even if it does break down you can still walk."
sltkr 14y ago
There are tools like Shake [1] that can semi-intelligently copy fragmented files around to defragment them, which typically results in less fragments if there is enough free space on the volume. 1. http://www.vleu.net/shake/ http://www.vleu.net/shake/
drostie 14y ago
I had that problem at one point. I vaguely recall that I unmounted my home partition and fscked it at one point to find the fragmented inodes, then there was complicated magic to find the filenames that those inodes referred to, and then there was a script which copied them all to a tmpfs (i.e. into memory), deleted the original, and copied the copy back out. One of the goals of recent ext4 work was to get support for "online defragmentation" but I don't know to what extent it's automatic, what tools exist, et cetera. It sounds like pyfragtools also has attempted to solve this problem.
ajross 14y ago
Uh... attaching new media and using "tar" isn't a practical way of unfragmenting it? Hard drives are almost free in the modern world.
- shawnz 14y ago
  When compared to having a (reasonably simple) tool which will do it for free, using only one volume, while still allowing r/w access? Yes, it's impractical.
gcr 14y ago
If your filesystem is completely full, fragmentation should be the last of your worries.
- ars 14y ago
  It was full, now I expanded it, but the old data is still heavily fragmented.

trentmb 14y ago

"If you do have problems with fragmentation on Linux, you probably need a larger hard disk. If you actually need to defragment a file system, the simplest way is probably the most reliable: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk." So... buy a larger hard disk, or get a second disk to copy everything to instead of a defrag utility? I think I'll take the Windows approach here.

dunecn 14y ago
Well, in that case, Windows can't do any better. In order to defrag a filesystem, that requires free space to copy and move files around. If that space isn't there (because the HDD is nearly full), there isn't much you can do.
- tzs 14y ago
  The Windows defrag API allows you to work at the cluster level. You only in theory need a single free cluster in order to defrag.
  - cavilling_elite 14y ago
    At this rate, it would be faster to move everything over to a new harddrive.
    - deleted 14y ago
      [deleted]
  - ZoFreX 14y ago
    It's worth noting that while the Windows defrag utility is completely useless without a lot of free space available, many of the third party defrag tools can still work (albeit much more slowly) in that situation. Of course with SSDs becoming more and more prevalent, this is becoming a non-issue anyway.
14y ago

Travis 14y ago

Does this imply for certain contexts, the Windows model is superior? Say you had a "write/update infrequently, read many times" system profile with 10% disk space avail. It seems like Windows would store these files closer to each other on the drive, making reads more efficient. Or would that be such a negligible difference that it's not worth mentioning?

gouranga 14y ago
It's pretty negligible. I genuinely can't tell the difference between NTFS and ext3 at least performance-wise. However, the locking semantics on NTFS are shitty unless you know what you are doing (i.e. use KTM/Transactional NTFS) which sometimes skews the statistics a lot.

yoasif_ 14y ago

Users can defragment files (copying files to a new location) automatically by running something like pyfragtools: http://ubuntuforums.org/showthread.php?t=169551 http://ubuntuforums.org/showthread.php?t=169551 Other filesystems like ZFS and btrfs utilize copy-on-write to virtually eliminate file fragmentation -- files are copied whenever edits are made to the file.

wch 14y ago
My understanding is that COW filesystems have really bad fragmentation if you're changing portions of large files -- for example, with virtual machine disk images.
- wch 14y ago
  Actually, I just discovered that on btrfs, you can disable COW on a volume, or even on individual files. To do it on a per-file basis, you have to compile a small program available here: http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg14877.html http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg1... This makes btrfs suitable for virtual machine images.
drostie 14y ago
Maybe ZFS/btrfs users don't care, but I would be really hesitant before using a filesystem which does not respect the Unix `shred` tool. Like real paper-shredders, I don't use them very much but when I do use them I really hope that they work.
- icebraining 14y ago
  Why doesn't shred work on ZFS/Btrfs?
  - barrkel 14y ago
    ZFS is copy-on-write with snapshot support. If you have files you may need to shred, put them on a different file system; ZFS is oriented towards making it hard to lose data.

sciurus 14y ago

On ext2/3/4 filesystems, there are a couple handy tools for measuring fragmentation that you probably already have installed To see how fragmented a file is: filefrag -v filename To see how fragmented your free space is e2freefrag devicename http://linux.die.net/man/8/e2freefrag http://linux.die.net/man/8/e2freefrag http://linux.die.net/man/8/filefrag http://linux.die.net/man/8/filefrag To learn about the intricacies of how fragmentation can occur in ext3 and some better tools for measuring it, see the fragmentation section of Oliver Diedrich's article 'Tuning the Linux file system Ext3'. http://www.h-online.com/open/features/Fragmentation-746701.html http://www.h-online.com/open/features/Fragmentation-746701.h... Extents, multiblock allocation, and delayed allocation make ext4 more resistant to fragmentation than ext3, but the developers are still working on a defrag utility. http://kernelnewbies.org/Ext4 http://kernelnewbies.org/Ext4

ciupicri 14y ago

XFS has some sort of a defragmentation tool - xfs_fsr: xfs_fsr [-v] [-t seconds] [-f leftoff] [-m mtab] xfs_fsr [-v] [xfsdev | file] ... xfs_fsr improves the organization of mounted filesystems. The reorganization algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).

pyre 14y ago
xfs also has a tendency to fill up the kernel slab space when stat'ing a bunch of files/directories. This is really annoying because most "memory full" tools don't delve into the slab space to see if the usage is something that will free up automatically.

wwrap 14y ago

A lot of questions being asked here are already covered in this thread from a day ago: http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt_need_defragmenting/ http://dd.reddit.com/r/linux/comments/uc1o9/why_linux_doesnt...

pooriaazimi 14y ago

The comment thread on this article made me appreciate HN a lot more... It reminded me of those silly schoolyard arguments.

nitrogen 14y ago
I was surprised to see someone still commenting about how you can't use Linux in the "real world" in 2012. So my world's not the "real world?"

naner 14y ago

How does ext* move files around to reduce fragmentation if the disk is full? I'm glad you asked. Take a look at your filesystem properties: # tune2fs -l /dev/blah In there you'll see a property called "Reserved Block Count". That is space reserved for kernel use to avoid fragmentation, continue logging, and for any other tasks the kernel/root user may need to do on a full filesystem. Users cannot write to this and your disk utilities will report your disk being full even if this reserved space is still available. I believe it is typically 5% of the partition size. If your partition is hundreds of gigabytes then the reserved portion may be tens of gigabytes (multiply the reserved blocks by the block size to get total byte size). You can change the percentage of this portion with the tune2fs -m flag. I have filled up a 500 GB backup drive with 0% reserved blocks (%5 reserved would have meant that 25GB was unusable) with no issues. Obviously, you probably don't want it set to 0% if you are constantly changing the contents of that partition.

cabirum 14y ago

1. It's not hard to implement such file allocation strategy. If it was superior, other OS'es would have adopted it long ago. 2. Scattering files all over the partition makes the longest seek times. This is actually the most inefficient solution for linux with its apps and sources (usually) consisting of thousands small files. 3. Windows does not need to be defragmented manually as it defragments itself when PC is not in use.

bnegreve 14y ago
> 2. Scattering files all over the partition makes the longest seek times. Yes but it isn't guaranteed that there will be less fragmentation on a windows file system. It depends on when was the last defrag, on the way file are created, and your disk occupation. Also, when you scatter files on different disks it can be more efficient because you can read several cluster simultaneously. But I guess Windows does this as well.

TwoBit 14y ago

Windows seems superior to me. It performs better due to seek time reduction, and it defrags and optimizes during idle periods. With Linux you have the worst of both worlds: it's slower and when the disk gets near full you supposedly have no choice but to re-do the disk.

thyrsus 14y ago

In 30 years working with Unix, I've never encountered a situation where "defragmentation" was the answer. File system too full? Dozens of times. So, what usage pattern am I avoiding to live such a charmed life? Anecdotally, I run a couple virtual machines on my laptop whose storage is on an external drive that sounds like Geiger counter at Fukushima Reactor 1 when the drive is busy. That is, when I've booted Windows 7. Not when I'm running anything under Windows, Windows just needs to be booted. When I'm running only a Linux VM, it drops to background level.

ams6110 14y ago

What I haven't seen mentioned is that historically, Unix (which inspired Linux) is a server/multi-user OS. All this thinking about "my" disk drives and how "my" large files are allocated is pretty irrelevant when you have dozens of users as well as system daemons accessing different files all of the time. Your drives are seeking all over the place constantly anyway. Fragmentation in such a scenario is much less of an issue.

steve8918 14y ago

So Linux DOES need to be defragmented, but the only difference is that it defragments automatically, unlike Windows which requires an external defragmentation tool. So really the title should be "Why Linux Doesn't Need Defragmentation Tools".

nitrogen 14y ago

This might be more accurately titled "Why ext2/3/4 doesn't need defragmenting." The advice given in the article: Copy all the files off the partition, erase the partition, then copy the files back onto the partition. The file system will intelligently allocate the files as you copy them back onto the disk. does not work for ntfs-3g. Whatever strategy it uses to allocate blocks, it's terrible for streaming reads of large files (last time I checked about 1-2 years ago). So, if you're trying to use the copy-off copy-on method of defragmenting an NTFS partition, don't do it from Linux.