4 ms·

We had a bunch of Thumpers (SunFire X4200) with 48 disks at work, running ZFS on Solaris. It was dog slow and awful, tuning performance was complicated and took

by HorstG 7y ago

We had a bunch of Thumpers (SunFire X4200) with 48 disks at work, running ZFS on Solaris. It was dog slow and awful, tuning performance was complicated and took ages. One had to use just the right disks in just the right order in RaidZs with striping over them. Swap in a hotspare: things slow to a crawl (i.e. not even Gbit/s). After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust. Sorry, don't have numbers around anymore, stuff has been trashed since. Oh, and btw., snapshots and larger numbers of filesystems (which Sun recommended instead of the missing Quota support) also slow things down to a crawl. ZFS is nice on paper and maybe nice to play with. Definitely simpler to use than anything else. But performance-wise it sucked big time, at least on Solaris.

Annatar 7y ago
"After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust." Please let me know which company this is, so I can ensure that I never end up working there by accident. Much obliged in advance, thank you kindly.
- HorstG 7y ago
  Why? What is bad about playing around with leftover hardware?
  - Annatar 7y ago
    Nothing at all; it's what was done to that hardware that's the travesty here. It takes an extraordinary level of incompetence and ignorance to even get the idea to slap Linux with dmraid and LVM on that hardware and then claim that it was faster and more robust without understanding how unreliable and fragile that constelation is and that it was faster because all the reliability was gone.
wazoox 7y ago
ZFS performs quite well if you give it boatloads of RAM. It uses its own cache layer, and eats RAM like hotcakes. XFS OTOH is as fast as the hardware can go with any amount of RAM.
- esoterae 7y ago
  Sort of. But no snapshots. Wanna use LVM for snapshots? 33% performance hit for the entire LV per snapshot, by implementation. ZFS? ~1% hit. I've never been able to see any difference at the workloads I run, whereas with LVM it was pervasive and inescapable.
  - denimboy 7y ago
    I don't doubt this but do you have any documentation? Asking for a friend who uses XFS on LVM for disk heavy applications like database, file server, etc.
    - rleigh 7y ago
      You would have to look at the implementation directly. The user documentation isn't great for documenting performance considerations, sadly. Essentially it comes down to this: a snapshot LV contains copies of old blocks which have been modified in the source LV. Whenever a block is updated in the source LV, LVM will need to check if that block been previously copied into all corresponding snapshot LVs. For each source LV where this is not the case, it will need to copy the block to the snapshot LV. This means that there is O(n) complexity in the checking and copying. And in the case of "thin" LVs, it will also need to allocate the block to copy to, potentially for every snapshot LV in existence, making the process even slower. The effect is write amplification effectively proportional to the total number of snapshots. ZFS snapshots, in comparison, cost essentially the same no matter how many you create, because the old blocks are put onto a "deadlist" of the most recent snapshot, and it doesn't need repeating for every other snapshot in existence. Older snapshots can reference them when needed, and if a snapshot is deleted, any blocks still referenced are moved to the next oldest snapshot. Blocks are never copied and only have a single direct owner. This makes the operations cheap.
    - snvzz 7y ago
      FreeNAS has good documentation on which hardware to pick and how to set up ZFS.
  - 7y ago
__jal 7y ago
Sounds like you turned on dedupe, or had an absurdly wide stripe size. You do need to match your array structure to your needs as well as tune ZFS. On our backup servers (45 disks, 6-wide Z2 stripes) easily handle wire-speed 10G with 32G ARC. And you're just wrong about snapshots and filesystem counts. ZFS is no speed demon, but it performs just fine if you set it up correctly and tune it.
- HorstG 7y ago
  Stripe size could have been a problem, though we just went with the default there afair. Most of the first tries was just along the Sun docs, we later only changed things until performance was sufficient. Dedupe wasn't even implemented back then. Maybe you also don't see as massive an impact because your hardware is a lot faster. X4200s were predominantly meant to be cheap, not fast. No cache, insufficient RAM, slow controllers, etc.
  - hunta2097 7y ago
    X4200s were the devil's work. Terrible BMC, raid controller, even the disk caddies were poorly designed. The BMC controller couldn't speak to the disk controller so you had no out-of-band storage management. I had to Run a fleet of 300 of them, truly an awful time.
lmm 7y ago
dmraid raid5/6 lose data, sometimes catastrophically, in normal failure scenarios that the ZFS equivalent handles just fine. If a sector goes bad between the time when you last scrubbed and the time when you get a disk failure (which is pretty much inevitable with modern disk sizes), you're screwed.
jordanbeiber 7y ago
ZFS, on Solaris, not robust? ZFS for “play”?! This... is just plain uninformed. Not just me and my employer, but many (many) others rely on ZFS for critical production storage, and have done so for many years. It’s actually very robust on Linux as well - considering the fact that freeBSD have started to use the ZoL code base is quite telling. Would freeBSD also be in the “play” and “not robust” category as well, hanging out together with Solaris? Will it perform better than all in terms of writes/s? Most likely not - although by staying away from de-dup, enough RAM and adhere the pretty much general recommendation to use mirror vdevs only in your pools, it can be competitive. Something solid with data integrity guarantees? You can’t beat ZFS, imo.
- HorstG 7y ago
  Performance wasn't robust, especially on dead disks and rebuilds, but also on pools with many (>100) filesystems or snapshots. Performance would often degrade heavily and unpredictably on such occasions. We didn't loose data more often than with other systems. "play" comes from my distinct impression that the most vocal ZFS proponents are hobbyists and admins herding their pet servers (as opposed to cattle). ZFS comes at low/no cost nowadays and is easy to use, therefore ideal in this world.
  - jordanbeiber 7y ago
    Fair enough, I can’t argue with your personal experience, but I can assure you that ZFS is used ”for real” at many shops. I’ve only used zfs in two or three way mirror setup, on beefy boxes, where the issues you describe are minimal. Also JBOD only. The thing is that without checksumming you’ve actually no idea if you lose data. I’ve had several pools over the years report automatic resilvering on checksum mismatches. Usually it’s been disks acting up well before smart can tell, and reporting this has been invaluable.
- montjoy 7y ago
  > Something solid with data integrity guarantees? You can’t beat ZFS, imo. This reminds me. We had one file server used mostly for package installs that used ZFS for storage. One day our java package stops installing. The package had become corrupt. So I force a manual ZFS scrub. No dice. Ok fine I’ll just replace the package. It seems to work but the next day it’s corrupt again. Weird. Ok I’ll download the package directly from Oracle again. The next day again it’s corrupt. I download a slightly different version. No problems. I grab the previous problematic package and put it in a different directory (with no other copies on the file system) - again it becomes corrupt. There was something specific about the java package that ZFS just thought it needed to “fix”. If I had to guess it was getting the file hash confused. I’m pretty sure we had dedupe turned on so that may have factored into it. Anyway that’s the first and only time I’ve seen a file system munge up a regular file for no reason - and it was on ZFS.

by HorstG 7y ago

Annatar 7y ago

"After EoL a colleague installed Linux with dmraid, LVM and xfs on the same hardware: much faster, more robust." Please let me know which company this is, so I can ensure that I never end up working there by accident. Much obliged in advance, thank you kindly.

HorstG 7y ago
Why? What is bad about playing around with leftover hardware?
- Annatar 7y ago
  Nothing at all; it's what was done to that hardware that's the travesty here. It takes an extraordinary level of incompetence and ignorance to even get the idea to slap Linux with dmraid and LVM on that hardware and then claim that it was faster and more robust without understanding how unreliable and fragile that constelation is and that it was faster because all the reliability was gone.

wazoox 7y ago

ZFS performs quite well if you give it boatloads of RAM. It uses its own cache layer, and eats RAM like hotcakes. XFS OTOH is as fast as the hardware can go with any amount of RAM.

esoterae 7y ago
Sort of. But no snapshots. Wanna use LVM for snapshots? 33% performance hit for the entire LV per snapshot, by implementation. ZFS? ~1% hit. I've never been able to see any difference at the workloads I run, whereas with LVM it was pervasive and inescapable.
- denimboy 7y ago
  I don't doubt this but do you have any documentation? Asking for a friend who uses XFS on LVM for disk heavy applications like database, file server, etc.
  - rleigh 7y ago
    You would have to look at the implementation directly. The user documentation isn't great for documenting performance considerations, sadly. Essentially it comes down to this: a snapshot LV contains copies of old blocks which have been modified in the source LV. Whenever a block is updated in the source LV, LVM will need to check if that block been previously copied into all corresponding snapshot LVs. For each source LV where this is not the case, it will need to copy the block to the snapshot LV. This means that there is O(n) complexity in the checking and copying. And in the case of "thin" LVs, it will also need to allocate the block to copy to, potentially for every snapshot LV in existence, making the process even slower. The effect is write amplification effectively proportional to the total number of snapshots. ZFS snapshots, in comparison, cost essentially the same no matter how many you create, because the old blocks are put onto a "deadlist" of the most recent snapshot, and it doesn't need repeating for every other snapshot in existence. Older snapshots can reference them when needed, and if a snapshot is deleted, any blocks still referenced are moved to the next oldest snapshot. Blocks are never copied and only have a single direct owner. This makes the operations cheap.
  - snvzz 7y ago
    FreeNAS has good documentation on which hardware to pick and how to set up ZFS.
- 7y ago

__jal 7y ago

Sounds like you turned on dedupe, or had an absurdly wide stripe size. You do need to match your array structure to your needs as well as tune ZFS. On our backup servers (45 disks, 6-wide Z2 stripes) easily handle wire-speed 10G with 32G ARC. And you're just wrong about snapshots and filesystem counts. ZFS is no speed demon, but it performs just fine if you set it up correctly and tune it.

HorstG 7y ago
Stripe size could have been a problem, though we just went with the default there afair. Most of the first tries was just along the Sun docs, we later only changed things until performance was sufficient. Dedupe wasn't even implemented back then. Maybe you also don't see as massive an impact because your hardware is a lot faster. X4200s were predominantly meant to be cheap, not fast. No cache, insufficient RAM, slow controllers, etc.
- hunta2097 7y ago
  X4200s were the devil's work. Terrible BMC, raid controller, even the disk caddies were poorly designed. The BMC controller couldn't speak to the disk controller so you had no out-of-band storage management. I had to Run a fleet of 300 of them, truly an awful time.

lmm 7y ago

dmraid raid5/6 lose data, sometimes catastrophically, in normal failure scenarios that the ZFS equivalent handles just fine. If a sector goes bad between the time when you last scrubbed and the time when you get a disk failure (which is pretty much inevitable with modern disk sizes), you're screwed.

jordanbeiber 7y ago

ZFS, on Solaris, not robust? ZFS for “play”?! This... is just plain uninformed. Not just me and my employer, but many (many) others rely on ZFS for critical production storage, and have done so for many years. It’s actually very robust on Linux as well - considering the fact that freeBSD have started to use the ZoL code base is quite telling. Would freeBSD also be in the “play” and “not robust” category as well, hanging out together with Solaris? Will it perform better than all in terms of writes/s? Most likely not - although by staying away from de-dup, enough RAM and adhere the pretty much general recommendation to use mirror vdevs only in your pools, it can be competitive. Something solid with data integrity guarantees? You can’t beat ZFS, imo.

HorstG 7y ago
Performance wasn't robust, especially on dead disks and rebuilds, but also on pools with many (>100) filesystems or snapshots. Performance would often degrade heavily and unpredictably on such occasions. We didn't loose data more often than with other systems. "play" comes from my distinct impression that the most vocal ZFS proponents are hobbyists and admins herding their pet servers (as opposed to cattle). ZFS comes at low/no cost nowadays and is easy to use, therefore ideal in this world.
- jordanbeiber 7y ago
  Fair enough, I can’t argue with your personal experience, but I can assure you that ZFS is used ”for real” at many shops. I’ve only used zfs in two or three way mirror setup, on beefy boxes, where the issues you describe are minimal. Also JBOD only. The thing is that without checksumming you’ve actually no idea if you lose data. I’ve had several pools over the years report automatic resilvering on checksum mismatches. Usually it’s been disks acting up well before smart can tell, and reporting this has been invaluable.
montjoy 7y ago
> Something solid with data integrity guarantees? You can’t beat ZFS, imo. This reminds me. We had one file server used mostly for package installs that used ZFS for storage. One day our java package stops installing. The package had become corrupt. So I force a manual ZFS scrub. No dice. Ok fine I’ll just replace the package. It seems to work but the next day it’s corrupt again. Weird. Ok I’ll download the package directly from Oracle again. The next day again it’s corrupt. I download a slightly different version. No problems. I grab the previous problematic package and put it in a different directory (with no other copies on the file system) - again it becomes corrupt. There was something specific about the java package that ZFS just thought it needed to “fix”. If I had to guess it was getting the file hash confused. I’m pretty sure we had dedupe turned on so that may have factored into it. Anyway that’s the first and only time I’ve seen a file system munge up a regular file for no reason - and it was on ZFS.