3 ms·

>start making assumptions about it's performance on low memory systems. I've ran it for real. Once a few more background services start running, ZFS bogs down

by zaarn 8y ago

>start making assumptions about it's performance on low memory systems. I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive. >You actually can grow a ZFS vdev - exactly by replacing one disk at a time (albeit you don't get the gained capacity until all the disks in the vdev have been replaced). Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time. >The kind of storage arrays that grow by adding solitary disks work quite different to your typical RAID. They'll often have designated parity drives instead. See solutions like "unraid". Not only unraid, there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too. >surprisingly intuitive command line tools is what edges ZFS for me in terms of usability. I found the commands anything but intuitive though I guess that's a difference of a few years of experience, I'd agree that would be personal opinion if it's easier.

laumars 8y ago
> I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive. Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems). As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either. edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead. > Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time. ...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion. To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing. > Not only unraid I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there. > there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too. When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same? I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/ > I found the commands anything but intuitive though I guess that's a difference of a few years of experience Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :) Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS. In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.
- cyphar 8y ago
  > To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed. [1]: https://www.youtube.com/watch?v=Njt82e_3qVo https://www.youtube.com/watch?v=Njt82e_3qVo
  - laumars 8y ago
    I actually did acknowledged that limitation. Maybe I wasn’t all that clear but my actual point was that requirement is where most traditional raids often fall apart (even mdadm requires days of downtime to grow a vdev) so its usually better to grow the pool instead of the vdev anyway. Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
- zaarn 8y ago
  >Was that an off the shelf NAS? No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2. As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB. >...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. If you use mdadm and ext4 you can do it online. >To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions. The home desktop is possibly the largest filesystem market next to "disposable virtual machine". >and now you're saying your preferred solutions also does the same? They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage. On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs. >what is your experience of ZFS outside of the troubles you had on your NAS? I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.

by zaarn 8y ago

laumars 8y ago

> I've ran it for real. Once a few more background services start running, ZFS bogs down and keels over. I've had to reboot the NAS dozens of times because the entire filesystem had become totally unresponsive. Was that an off the shelf NAS? If so then I'm guessing it wasn't exactly high spec'ed even putting aside the RAM? eg did it have a 32bit CPU? I've heard that does a massive amount to dent ZFS's performance (I wouldn't know because I've only ever ran on it 64 bit systems). As I said, I've not had any issues with raidz2 on 2GB RAM. I've even ran virtual machines on one OpenSolaris box with ZFS and 2GB RAM so these weren't always dedicated storage servers either. edit: just read elsewhere you were running FreeNAS on a Pentium 4. Yeah that would do it. FreeNAS wasn't optimised for low end systems (in fact they outright tell you not to!) and the P4 is pretty low end CPU as well. I still think ZFS would have ran on that but you'd have been better off going with vanilla FreeBSD instead. > Which is not an accaptable answer to the situation I presented, you still have to buy multiple disks to grow the pool, you can't do it one at a time. ...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. That process can literally take several days on some storage servers. So ZFS might require you to add vdevs to the pool (tank) but at least it can all be done on live systems without any downtime. That in itself is a massive boon for ease of use in my opinion. To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. And if you are the type of person that would need to do that then you wouldn't go for a traditional raid5/6 style pool either. Thus the argument becomes a comparison of apples and oranges as you'e comparing completely different types of software RAIDing. > Not only unraid I didn't say "exclusively unraid". I said "solutions like unraid". I'm fully aware there are other solutions out there. > there are other systems that provide almost this. Syno and QNAP can both grow an array in storage capacity if you add a single new disk, even if it's larger and once you have added enough to provide parity, the additional capacity will become used too. When you said "there are other systems that provide almost this" and "once you have added enough [disks] to provide parity, the additional capacity will become used too." I really had to question what your actual point of this discussion was because just a paragraph earlier you were calling my points "unacceptable" because ZFS required you to add more than 1 extra disk to provide parity; and now you're saying your preferred solutions also does the same? I hope I'm just being presumptuous here but it's starting to feel like you're just arguing for the sake of arguing. :-/ > I found the commands anything but intuitive though I guess that's a difference of a few years of experience Maybe. I've been a Linux and UNIX sysadmin for a fair few years now so experience is not something I'm short of. :) Honest question: what is your experience of ZFS outside of the troubles you had on your NAS? I could totally see why the problems you described would taint your opinion of ZFS but I definitely wouldn't say what you experienced is typical for ZFS. In any case, I'm not going to argue that everyone should be running ZFS - it's just technology, you can run whatever you want to run.

cyphar 8y ago
> To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. OpenZFS is currently working on adding code to allow for vdev expansion[1]. I am a fan of ZFS as well, but I don't get not wanting to admit its limitations. Not being able to expand vdevs is a limitation and it's being addressed. [1]: https://www.youtube.com/watch?v=Njt82e_3qVo https://www.youtube.com/watch?v=Njt82e_3qVo
- laumars 8y ago
  I actually did acknowledged that limitation. Maybe I wasn’t all that clear but my actual point was that requirement is where most traditional raids often fall apart (even mdadm requires days of downtime to grow a vdev) so its usually better to grow the pool instead of the vdev anyway. Or in other words: if growing the vdev is a hard requirement then a traditional raid might not be the best option regardless of ZFS. It’s about picking the right tool for the job rather than moaning that ZFS doesn’t do every job perfectly (which seems to be the argument the GP was making).
zaarn 8y ago
>Was that an off the shelf NAS? No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2. As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB. >...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. If you use mdadm and ext4 you can do it online. >To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions. The home desktop is possibly the largest filesystem market next to "disposable virtual machine". >and now you're saying your preferred solutions also does the same? They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage. On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs. >what is your experience of ZFS outside of the troubles you had on your NAS? I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.