12 ms·

So I just started experimenting with ZFS, because it seemed required for container snapshots. Then I found out it fragments badly, and nobody can figure out ho

by dakami 10y ago

So I just started experimenting with ZFS, because it seemed required for container snapshots. Then I found out it fragments badly, and nobody can figure out how to write a defragmenter. So, uh, keep the FS below 60-80% full apparently. Yeah.

gcr 10y ago
Have you tried btrfs? It also has support for super-cheap copy-on-write operations which should make container images and the like a snap. Not sure if your container tool supports btrfs snapshots of course, but it's conceptually simple, right?
- aorth 10y ago
  For development containers, systemd-nspawn has had support for btrfs snapshots since 2014 or 2015. Simple to get going if you're on a Linux box with systemd, no other daemons or tools required.
- cmurf 10y ago
  I think the main issue here is Btrfs is still developing. Its kernel doc file still says it's for benchmarking and review. [1] CoreOS devs decided to switch from Btrfs to overlay(fs) about 17 months ago. That's a long, long time in Btrfs development "years" that's how much development happens on Btrfs. But I can't say if CoreOS would, had today's Btrfs been what they were using in 2014, would have changed their decision. RH/Fedora are very dm/LVM thinp snapshots with XFS centric for backing their containers. I think what you're seeing is distros are doing something different with their container backing approach in order to differentiate from other distros. Maybe it's a stab in the dark or spaghetti on the wall approach but in the end all of these storage backends are going to mature a lot in the interim, so ultimately it'll be good for everyone. [1] https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/filesystems/btrfs.txt?id=refs/tags/v4.5.4 https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st...
- takeda 10y ago
  Isn't this issue inherent to all COW systems (ZFS, WAFL, btrfs)?
- Annatar 10y ago
  Btrfs has been in beta for how many years now? ZFS is protecting data in enterprise production environments since 2006 (Solaris 10 update 2).
  - krondor 10y ago
    Btrfs is not beta, see the discussion re: maturity below. ZFS is an excellent filesystem, btrfs is an excellent filesystem. There is room for both excellent options for users.
hhw 10y ago
Using dedicated ZIL significantly reduces fragmentation: http://www.racktopsystems.com/dedicated-zfs-intent-log-aka-slogzil-and-data-fragmentation/ http://www.racktopsystems.com/dedicated-zfs-intent-log-aka-s... Anyone using ZFS in a serious capacity would have both dedicated ARC and ZIL.
- sneak 10y ago
  ARC yes, ZIL no.
  - cthalupa 10y ago
    In a thread that is about the perils of ZFS fragmentation, you are replying to a link saying that a ZIL seriously reduces the risk of fragmentation, and saying that someone worried about fragmentation does not need to use a ZIL. Why? If there's a legitimate reason, please expand.
    - gnoway 10y ago
      I think he meant that they might not have one. It's been a while since I looked at using ZFS for anything meaningful, but at the time (~6 years ago), while losing L2ARC was no big deal, losing dedicated ZIL was catastrophic. I think that's still true today. So you need at least two ZIL devices in a mirror. On top of that, you really need something faster and lower latency for your ZIL vs. the ARC or main pool; people were trying to use SSDs but most commonly-available drives at the time would either degrade or fail in a hurry under load. So the options were RAM-based, e.g. STEC ZeusRAM on the high end, or some sort of PCI-X/PCIe RAM device. The former was not easy or cheap to acquire for testing stuff, and the latter made failover configs impossible. I think that ZIL is also not soaking up all writes, just most writes meeting a certain criteria. Some just stream through to the pool. So I was always thinking of it as a protection device that also converted random writes to sequential. Some people don't think they need that. I remember the fragmentation issue being a problem at the time, but also thinking it was probably going to get solved soon because there was so much interest and a whole company behind it. Then Oracle happened. My guess is that if it were still Sun and all the key people were still there, this would be a solved problem right now. As it is, Oracle probably wants you to buy all the extra storage anyway, and would love to offer professional services to get you out of the fragmentation bind you're in.
KaiserPro 10y ago
have you actually benchmarked it? anything with copy of write is going to fragment. are you still running on spinning rust? I've not seen any real performance hits until 90% full, but then any file system with large images suffers at that point.
rsync 10y ago
"Then I found out it fragments badly, and nobody can figure out how to write a defragmenter. So, uh, keep the FS below 60-80% full apparently." Confirmed. Not FUD. Our experience[1] is that things go to hell around 90% and even if you bring it below 90% there is a permanent performance degradation to the pool. In order to be safe, we try to keep things below 80%, just to be safe. That's probably a bit conservative, though. ZFS needs defrag. It is not reasonable to give up 3 drives worth of capacity for the parity (raidz3, for instance) and then on top of that set aside another 10-20% as the "angels share". [1] rsync.net
- craigyk 10y ago
  I've seen this repeated a lot, but have not had quite the same experience with "permanent" performance degradation. Especially if I eventually expand the pool with another vdev. Not sure about ZFSonLinux, but: 1) Having a ZIL helps with this, and in general. 2) ZFS changes strategy depending on how full it is, it spends more time avoiding further fragmentation rather than grabbing the first empty slot. This hit would go away if you get the free space back up. 3) Finally, there is a way[1] to have ZFS keep all the info it needs in RAM to greatly alleviate the times when it starts hunting harder to prevent more fragmentation. It looks like the RAM requirements are 32GB/1PB... so not too bad IMO. [1] https://blogs.oracle.com/bonwick/entry/space_maps https://blogs.oracle.com/bonwick/entry/space_maps
  - rsync 10y ago
    "I've seen this repeated a lot, but have not had quite the same experience with "permanent" performance degradation. Especially if I eventually expand the pool with another vdev. Not sure about ZFSonLinux" Look, I'll admit that we haven't done a lot of scientific comparisons between healthy pools and presumed-wrecked-but-back-below-80-percent pools ... but I know what I saw. I think if you break the 90% barrier and either: a) get back below it quickly, or b) don't do much on the filesystem while it's above 90%, you'll probably be just fine once you get back below 90%. However, if you've got a busy, busy, churning filesystem, and you grow above 90% and you keep on churning it while above 90%, your performance problems will continue one you go back below, presuming the workload is constant. Which makes sense ... and, anecdotally, is the same behavior we saw with UFS2 when we tune2fs'd minfree down to 0% and ran on that for a while ... freeing up space and setting minfree back to 5-6% didn't make things go back to normal ... I am receptive to the idea that a ZIL solves this. I don't know if it does or not.
cmurf 10y ago
What I think would help any COW file system is delay snapshot (and clone) deletion longer, and delete in groups, which would result in larger contiguous regions being freed. When one container is deleted, a small amount of space is freed, and then may be used for writes, quickly filling up, and thus increasing localized fragmentation - or more problematically for spinning drives it's increasing seek times by causing recent writes to be scattered farther apart. To reduce read and write seeks, it's better to have larger free areas for COW to write to sequentially. So it'd be nice if there were something like a "remove/hide" feature for containers, separate from delete or clean up, with a command that makes it easier to select many containers for deletion. At least on Btrfs this should be quite fast, and the background cleaner process that does the actual work of updating the ref count and freeing extents should have a priority such that it doesn't overly negatively impact other processes. Some of this behavior may change on Btrfs as the free space tracking has been recently rewritten. Right now the default is the original space cache implementation, while the new free space b-tree implementation is a mount time option intended only for testing.
ChuckMcM 10y ago
So to understand why this is you have to appreciate the goals behind write-anywhere-file-layout (aka WAFL) file systems. [1] One of the goals of such systems is that copy of the file system on disk is always consistent, turn power off at any point and you can come right back up with a valid file system. This is accomplished by only writing to the 'free block list'. You construct updated inodes from the file change all the way up to the root inode out of new blocks and then to "step" forward you write a new root block. This is really neat and it means that when you've done that step, you still have the old inodes and datablocks around, they just aren't linked but you can link them to another "holder" inode attached to the name ".snapshot" and it will show you the file system just before the change. Write the old root block back into the real root block and "poof!" you have reverted the file system back to the previous snapshot. Ok, so that is pretty sweet and really awesome in a lot of ways, but it has a couple of problems. The first, as noted, is that it pretty much guarantees fragmentation as its always reaching for free blocks and they can be anywhere. On NetApp boxes of old, that wasn't too much of a big deal because everything was done "per RAID stripe" so you were fragmented, but you were also reading/writing full stripes in RAID so you had the bandwidth you needed and fragmentation was absorbed by the efficiencies of full stripe reads/writes. But the second issue arises when you start getting close to full, managing the free block list gets harder and harder. You are constantly getting low block pressure, so you are constantly trying to reclaim old blocks (on unused snapshots, or expired ones) and that leads to a big drop in performance. The math is you can't change more of the data between snapshot steps than the amount of space you have free. That is why NetApp filers would get cranky using them in build environments where automated builds would delete scads of intermediate files, only to rebuild them and then relink them. Big percentage change in the overall storage. On the positive side, storage is pretty darn cheap these days, so a swapping in 3TB drives instead of 2TB drives means you could use all the storage you "planned" to use and keep the drives at 66% occupancy. Hard on users though who will yell at you "It says it has 10TB of storage and is only using 6TB but you won't expand my quota?" At such times it would be useful for the tools to lie but that doesn't happen. [1] Disclosure 1, I worked for 5 years at NetApp with systems that worked this way. Disclosure 2, an intern with NetApp (we'll call him Matt) was very impressed with this and went on to work at Sun for Jeff and similar solutions appeared in ZFS.
- 10y ago
davidgerard 10y ago
We got bitten by this back in the Solaris days in 2009, on a TV broadcasting production box with quite stringent uptime requirements: what happens is the defragmenter gets itself tied in knots and starts thrashing, and the symptom is 50% system CPU with no apparent cause. Got a Sun kernel engineer on call and all. SPOILER: it did in fact require a reboot to unfuck the system. Then we kept the disk in question being wasted.
- ryao 10y ago
  A reboot implies a bug had to be fixed. That can be assumed to be fixed everywhere by now.
  - davidgerard 10y ago
    ... no, it doesn't imply anything of the sort. As far as I know, ZFS still has this issue - it can get itself tied in knots, and only a reboot will stop this from happening.
ryao 10y ago
On my SSDs, I can go to 96% full without issue using ZoL. ZFSOnLinux is patched to disable ZFS' LBA weighting on solid state storage though. Non-solid state storage tends to reach 96% in metaslabs early due to LBA weighting, although I would be fine with filling a pool to 90% with the recent code. Going much higher than that is probably not a good idea though. I have yet to see evidence that 60%-80% causes issues unless the system is so overloaded that having performance drop a small amount is noticeable. On spinning disks, such a thing is only natural because there is not much space left in the outer platters. That said, older versions years ago would enter best fit behavior at 80%, which is where the 80% talk originated.
pbarnes_1 10y ago
We used ZFS in production for a year and it was the worst decision we'd ever made, precisely for this reason. Also, removing files uses up an insane amount of CPU and can block all FS operations if you get into the 80%+ full situation. I'll also note that the interplay between the ARC and Linux MM is... "interesting".

by dakami 10y ago

gcr 10y ago

Have you tried btrfs? It also has support for super-cheap copy-on-write operations which should make container images and the like a snap. Not sure if your container tool supports btrfs snapshots of course, but it's conceptually simple, right?

aorth 10y ago
For development containers, systemd-nspawn has had support for btrfs snapshots since 2014 or 2015. Simple to get going if you're on a Linux box with systemd, no other daemons or tools required.
cmurf 10y ago
I think the main issue here is Btrfs is still developing. Its kernel doc file still says it's for benchmarking and review. [1] CoreOS devs decided to switch from Btrfs to overlay(fs) about 17 months ago. That's a long, long time in Btrfs development "years" that's how much development happens on Btrfs. But I can't say if CoreOS would, had today's Btrfs been what they were using in 2014, would have changed their decision. RH/Fedora are very dm/LVM thinp snapshots with XFS centric for backing their containers. I think what you're seeing is distros are doing something different with their container backing approach in order to differentiate from other distros. Maybe it's a stab in the dark or spaghetti on the wall approach but in the end all of these storage backends are going to mature a lot in the interim, so ultimately it'll be good for everyone. [1] https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/Documentation/filesystems/btrfs.txt?id=refs/tags/v4.5.4 https://git.kernel.org/cgit/linux/kernel/git/stable/linux-st...
takeda 10y ago
Isn't this issue inherent to all COW systems (ZFS, WAFL, btrfs)?
Annatar 10y ago
Btrfs has been in beta for how many years now? ZFS is protecting data in enterprise production environments since 2006 (Solaris 10 update 2).
- krondor 10y ago
  Btrfs is not beta, see the discussion re: maturity below. ZFS is an excellent filesystem, btrfs is an excellent filesystem. There is room for both excellent options for users.

hhw 10y ago

Using dedicated ZIL significantly reduces fragmentation: http://www.racktopsystems.com/dedicated-zfs-intent-log-aka-slogzil-and-data-fragmentation/ http://www.racktopsystems.com/dedicated-zfs-intent-log-aka-s... Anyone using ZFS in a serious capacity would have both dedicated ARC and ZIL.

sneak 10y ago
ARC yes, ZIL no.
- cthalupa 10y ago
  In a thread that is about the perils of ZFS fragmentation, you are replying to a link saying that a ZIL seriously reduces the risk of fragmentation, and saying that someone worried about fragmentation does not need to use a ZIL. Why? If there's a legitimate reason, please expand.
  - gnoway 10y ago
    I think he meant that they might not have one. It's been a while since I looked at using ZFS for anything meaningful, but at the time (~6 years ago), while losing L2ARC was no big deal, losing dedicated ZIL was catastrophic. I think that's still true today. So you need at least two ZIL devices in a mirror. On top of that, you really need something faster and lower latency for your ZIL vs. the ARC or main pool; people were trying to use SSDs but most commonly-available drives at the time would either degrade or fail in a hurry under load. So the options were RAM-based, e.g. STEC ZeusRAM on the high end, or some sort of PCI-X/PCIe RAM device. The former was not easy or cheap to acquire for testing stuff, and the latter made failover configs impossible. I think that ZIL is also not soaking up all writes, just most writes meeting a certain criteria. Some just stream through to the pool. So I was always thinking of it as a protection device that also converted random writes to sequential. Some people don't think they need that. I remember the fragmentation issue being a problem at the time, but also thinking it was probably going to get solved soon because there was so much interest and a whole company behind it. Then Oracle happened. My guess is that if it were still Sun and all the key people were still there, this would be a solved problem right now. As it is, Oracle probably wants you to buy all the extra storage anyway, and would love to offer professional services to get you out of the fragmentation bind you're in.

KaiserPro 10y ago

have you actually benchmarked it? anything with copy of write is going to fragment. are you still running on spinning rust? I've not seen any real performance hits until 90% full, but then any file system with large images suffers at that point.

rsync 10y ago

"Then I found out it fragments badly, and nobody can figure out how to write a defragmenter. So, uh, keep the FS below 60-80% full apparently." Confirmed. Not FUD. Our experience[1] is that things go to hell around 90% and even if you bring it below 90% there is a permanent performance degradation to the pool. In order to be safe, we try to keep things below 80%, just to be safe. That's probably a bit conservative, though. ZFS needs defrag. It is not reasonable to give up 3 drives worth of capacity for the parity (raidz3, for instance) and then on top of that set aside another 10-20% as the "angels share". [1] rsync.net

craigyk 10y ago
I've seen this repeated a lot, but have not had quite the same experience with "permanent" performance degradation. Especially if I eventually expand the pool with another vdev. Not sure about ZFSonLinux, but: 1) Having a ZIL helps with this, and in general. 2) ZFS changes strategy depending on how full it is, it spends more time avoiding further fragmentation rather than grabbing the first empty slot. This hit would go away if you get the free space back up. 3) Finally, there is a way[1] to have ZFS keep all the info it needs in RAM to greatly alleviate the times when it starts hunting harder to prevent more fragmentation. It looks like the RAM requirements are 32GB/1PB... so not too bad IMO. [1] https://blogs.oracle.com/bonwick/entry/space_maps https://blogs.oracle.com/bonwick/entry/space_maps
- rsync 10y ago
  "I've seen this repeated a lot, but have not had quite the same experience with "permanent" performance degradation. Especially if I eventually expand the pool with another vdev. Not sure about ZFSonLinux" Look, I'll admit that we haven't done a lot of scientific comparisons between healthy pools and presumed-wrecked-but-back-below-80-percent pools ... but I know what I saw. I think if you break the 90% barrier and either: a) get back below it quickly, or b) don't do much on the filesystem while it's above 90%, you'll probably be just fine once you get back below 90%. However, if you've got a busy, busy, churning filesystem, and you grow above 90% and you keep on churning it while above 90%, your performance problems will continue one you go back below, presuming the workload is constant. Which makes sense ... and, anecdotally, is the same behavior we saw with UFS2 when we tune2fs'd minfree down to 0% and ran on that for a while ... freeing up space and setting minfree back to 5-6% didn't make things go back to normal ... I am receptive to the idea that a ZIL solves this. I don't know if it does or not.

cmurf 10y ago

What I think would help any COW file system is delay snapshot (and clone) deletion longer, and delete in groups, which would result in larger contiguous regions being freed. When one container is deleted, a small amount of space is freed, and then may be used for writes, quickly filling up, and thus increasing localized fragmentation - or more problematically for spinning drives it's increasing seek times by causing recent writes to be scattered farther apart. To reduce read and write seeks, it's better to have larger free areas for COW to write to sequentially. So it'd be nice if there were something like a "remove/hide" feature for containers, separate from delete or clean up, with a command that makes it easier to select many containers for deletion. At least on Btrfs this should be quite fast, and the background cleaner process that does the actual work of updating the ref count and freeing extents should have a priority such that it doesn't overly negatively impact other processes. Some of this behavior may change on Btrfs as the free space tracking has been recently rewritten. Right now the default is the original space cache implementation, while the new free space b-tree implementation is a mount time option intended only for testing.

ChuckMcM 10y ago

So to understand why this is you have to appreciate the goals behind write-anywhere-file-layout (aka WAFL) file systems. [1] One of the goals of such systems is that copy of the file system on disk is always consistent, turn power off at any point and you can come right back up with a valid file system. This is accomplished by only writing to the 'free block list'. You construct updated inodes from the file change all the way up to the root inode out of new blocks and then to "step" forward you write a new root block. This is really neat and it means that when you've done that step, you still have the old inodes and datablocks around, they just aren't linked but you can link them to another "holder" inode attached to the name ".snapshot" and it will show you the file system just before the change. Write the old root block back into the real root block and "poof!" you have reverted the file system back to the previous snapshot. Ok, so that is pretty sweet and really awesome in a lot of ways, but it has a couple of problems. The first, as noted, is that it pretty much guarantees fragmentation as its always reaching for free blocks and they can be anywhere. On NetApp boxes of old, that wasn't too much of a big deal because everything was done "per RAID stripe" so you were fragmented, but you were also reading/writing full stripes in RAID so you had the bandwidth you needed and fragmentation was absorbed by the efficiencies of full stripe reads/writes. But the second issue arises when you start getting close to full, managing the free block list gets harder and harder. You are constantly getting low block pressure, so you are constantly trying to reclaim old blocks (on unused snapshots, or expired ones) and that leads to a big drop in performance. The math is you can't change more of the data between snapshot steps than the amount of space you have free. That is why NetApp filers would get cranky using them in build environments where automated builds would delete scads of intermediate files, only to rebuild them and then relink them. Big percentage change in the overall storage. On the positive side, storage is pretty darn cheap these days, so a swapping in 3TB drives instead of 2TB drives means you could use all the storage you "planned" to use and keep the drives at 66% occupancy. Hard on users though who will yell at you "It says it has 10TB of storage and is only using 6TB but you won't expand my quota?" At such times it would be useful for the tools to lie but that doesn't happen. [1] Disclosure 1, I worked for 5 years at NetApp with systems that worked this way. Disclosure 2, an intern with NetApp (we'll call him Matt) was very impressed with this and went on to work at Sun for Jeff and similar solutions appeared in ZFS.

10y ago

davidgerard 10y ago

We got bitten by this back in the Solaris days in 2009, on a TV broadcasting production box with quite stringent uptime requirements: what happens is the defragmenter gets itself tied in knots and starts thrashing, and the symptom is 50% system CPU with no apparent cause. Got a Sun kernel engineer on call and all. SPOILER: it did in fact require a reboot to unfuck the system. Then we kept the disk in question being wasted.

ryao 10y ago
A reboot implies a bug had to be fixed. That can be assumed to be fixed everywhere by now.
- davidgerard 10y ago
  ... no, it doesn't imply anything of the sort. As far as I know, ZFS still has this issue - it can get itself tied in knots, and only a reboot will stop this from happening.

ryao 10y ago

On my SSDs, I can go to 96% full without issue using ZoL. ZFSOnLinux is patched to disable ZFS' LBA weighting on solid state storage though. Non-solid state storage tends to reach 96% in metaslabs early due to LBA weighting, although I would be fine with filling a pool to 90% with the recent code. Going much higher than that is probably not a good idea though. I have yet to see evidence that 60%-80% causes issues unless the system is so overloaded that having performance drop a small amount is noticeable. On spinning disks, such a thing is only natural because there is not much space left in the outer platters. That said, older versions years ago would enter best fit behavior at 80%, which is where the 80% talk originated.

pbarnes_1 10y ago

We used ZFS in production for a year and it was the worst decision we'd ever made, precisely for this reason. Also, removing files uses up an insane amount of CPU and can block all FS operations if you get into the 80%+ full situation. I'll also note that the interplay between the ARC and Linux MM is... "interesting".