3 ms·

>Was that an off the shelf NAS? No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2. As you

by zaarn 8y ago

>Was that an off the shelf NAS? No it was a repurposed office computer running a 12 year old Pentium 4 Dual Core (about 2.8GHz per Core). 2GB of DDR2. As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB. >...but you can't do that with most traditional RAIDs either and the few you can do it on require you to take the whole storage pool offline while you grow the pool then resize the file system. If you use mdadm and ext4 you can do it online. >To be honest, it's pretty rare you want to upgrade a storage array by just one disk anyway. At least outside of home users anyway. There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions. The home desktop is possibly the largest filesystem market next to "disposable virtual machine". >and now you're saying your preferred solutions also does the same? They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage. On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs. >what is your experience of ZFS outside of the troubles you had on your NAS? I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since.

laumars 8y ago
> As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB. That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server. > If you use mdadm and ext4 you can do it online. Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though. > There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions. > The home desktop is possibly the largest filesystem market next to "disposable virtual machine". I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al? The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality. I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think? > They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage. No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air. But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank. The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me. > On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs. Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it). > I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since. Yeah, I don't much like how ZFS's cache isn't shared with the kernel as freeable RAM. I get why they did it that way and generally it's not an issue (aside monitoring / reporting tools) because the ZFS kernel module should deallocate memory itself when it detects system memory is getting low but clearly that didn't happen in your case (which is strange - sounds like you found a bug that was worth reporting). Frankly speaking, I'm not a massive fan of ZoL anyway. I've got it working before but honestly I much prefer running ZFS on FreeBSD or Solaris / OpenSolaris, With regards to your particular use case: I wouldn't have recommended running ZFS for your hypervisor backend anyway. You don't really want CoW for write-heavy processes - particularly ones that have their own journals and/or snapshotting systems such as databases and hypervisors would. Like I said before, it's about using the right tool for the job and XFS would be my preferred Linux file system in those particular instances.
- zaarn 8y ago
  >Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though. https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing ext4 online resize works by invoking resize2fs. >I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al? QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs). >That hardly seems fair, don't you think? It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings. >No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air. Reread what I wrote, I mentioned adding 2x2TB not 1x2TB. > in your situation you don't get redundancy so your data becomes at risk if you have a failure Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data. And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array. >So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it). Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
  - laumars 8y ago
    > ext4 online resize works by invoking resize2fs. That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job. > QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs). That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm? > Reread what I wrote, I mentioned adding 2x2TB not 1x2TB. You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix). > Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS! > And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array. Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though? I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis. To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.

by zaarn 8y ago

laumars 8y ago

> As you mention, FreeNAS isn't optimized for that but a plain BSD install with ZFS didn't fare much better either and required a rather inane amount of work compared to other solutions that worked OOTB. That's pure conjecture. Plus I actually have ran ZFS on a Pentium 4 on FreeBSD. For a while that was my personal backup server. > If you use mdadm and ext4 you can do it online. Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though. > There is a lot of home users. People will use RAID5s and 6s at home, they'd rather have no-though-just-works solutions. > The home desktop is possibly the largest filesystem market next to "disposable virtual machine". I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al? The issue I have with your complaints is that you're flip-flopping across a multitude of industries from desktop to enterprise, which seems grossly unfair for making a balanced opinion. I'm not going to deny that ZFS will lack some desirable features compared to some speciality tools but like I said before, it's about picking the right tool for the job rather than arguing that ZFS is garbage because it doesn't win every single argument in every specific speciality. I mean you're not even willing to take on board any different configurations of ZFS nor alternative platforms it ships on - instead only basing your biases on one failed attempt at running FreeNAS (something which openly states they need beefy hardware to run) on equipment that is several generations old while then comparing that one specific ZFS instance against every alternative tool you've read about. That hardly seems fair, don't you think? > They don't behave the same. In these solutions if you add 2TB to a 4x1TB RAID5 array, you get 4TB of storage. If you add another 2TB you get 7TB of storage. No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air. But to assume you meant 2TB of extra storage when you add a 2TB HDD: in your situation you don't get redundancy so your data becomes at risk if you have a failure. You can actually expand your ZFS tank the same way if you really wanted (but no-one who cares about their data would want to) by adding a HDD vdev (unraided) to an existing tank. The issue is when you want to expand an existing raidz while retaining redundancy. To do that on any of the existing FOSS tools I've used, you'd have to either buy more than one HDD or do an offline restripe (as is the case with mdadm). I guess at least mdadm allows you to do it - even if it takes the storage pool offline for several days. Personally I've always opted to buy multiple drives and expand the pool (tank) by adding an additional raidz vdev because that not only expands the storage array but also then gives me another failure option (eg I can then lose 2 disks simultaneously without losing data). As you start to expand the number of disks you have, you equally increase the likelihood that more than 1 disk might fail. So having that extra redundancy disk is desirable to me. > On ZFS with a RAIDZ1 you'd get 5TB of storage since you'd have to stand up two RAIDZ1 vdevs. Actually no. You have to stand up two vdevs but a vdev can be a raidz, mirror, a solitary HDD or even a file on an existing file system. So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it). > I've been running a few hypervisors using ZFS as storage backend, though at the moment I'm migrating to a LVM thinpool based backend. I've had a few share of problems on that as well, notably ZFS at some point consume almost all of the 32GB of memory and running the OOM killer in the linux kernel until everything had crashed and the host required a hard reboot. Though this hasn't happened since. Yeah, I don't much like how ZFS's cache isn't shared with the kernel as freeable RAM. I get why they did it that way and generally it's not an issue (aside monitoring / reporting tools) because the ZFS kernel module should deallocate memory itself when it detects system memory is getting low but clearly that didn't happen in your case (which is strange - sounds like you found a bug that was worth reporting). Frankly speaking, I'm not a massive fan of ZoL anyway. I've got it working before but honestly I much prefer running ZFS on FreeBSD or Solaris / OpenSolaris, With regards to your particular use case: I wouldn't have recommended running ZFS for your hypervisor backend anyway. You don't really want CoW for write-heavy processes - particularly ones that have their own journals and/or snapshotting systems such as databases and hypervisors would. Like I said before, it's about using the right tool for the job and XFS would be my preferred Linux file system in those particular instances.

zaarn 8y ago
>Not according to any of the docs I've read in the past. Happy to be proven wrong if you have a citation though. https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing ext4 online resize works by invoking resize2fs. >I don't disagree with that but how many of those users are running LVM + mdadm? And how many of those people are just using unraid (or similar purpose built solution for home users)? How many users are just using an off-the-shelf NAS like QNAP et al? QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs). >That hardly seems fair, don't you think? It's the real world experience I had with the software and I think it's totally fair to point out and complain about the shortcomings. >No raid would give you 3TB of extra storage when you add a 2TB HDD. That would be breaking the laws of physics because you're now asking your server to create 1TB from thin air. Reread what I wrote, I mentioned adding 2x2TB not 1x2TB. > in your situation you don't get redundancy so your data becomes at risk if you have a failure Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data. And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array. >So you can just add an additional HDD to a ZFS tank but you wouldn't get redundancy on that additional HDD (which is why nobody suggests you should do it). Again, what Syno offers (and what can be replicated on any Linux system) allows you to add a single drive to expand the storage without loosing parity and more drives to span any drive increases with parity. This allows adding a single drive at a time and at worst you loose any additional capacity without parity yet (in the 1TB/2TB example, that would be loosing 1TB of storage until the second drive is added)
- laumars 8y ago
  > ext4 online resize works by invoking resize2fs. That's resizing the filesystem not the pool (eg `mdadm --grow`). You're also missing the point about what I said. To run those commands you must first `umount` to pool. Hence it being an offline job. > QNAP and Synology don't do magic. They deploy mdadm, LVM and ext4 or btrfs (though btrfs only in single drive mode, they use LVM and mdadm to span the volume for btrfs). That's completely irrelevant to the point I was making. How many home users do you know who configure their own home raids using LVM and mdadm? > Reread what I wrote, I mentioned adding 2x2TB not 1x2TB. You actually said neither. You said, and I quote, "2TB" (ie without any accumulation prefix). > Still having redundancy is exactly the point. The resulting array in the example would span 6x1TB and 2x1TB in RAID5, giving me a single drive for parity in each span and resulting in an effective RAID5 for the entire RAID. I can loose any drive in that 4x1+2x2TB array without loosing data You can actually do that in ZFS. In fact what you've just described were is the exact solution that you called "unacceptable" in ZFS! > And it does that online without having to reboot or even halt any ongoing file operations. The volume will simply increase in size once it has finished formatting the new drive and integrated it into the array. Indeed. Lots of commercial tools do support that. I thought we were comparing ext4, LVM and mdadm to ZFS though? I'm guessing you just glossed other my post where I said you were flip-flopping all over the place? It's easy to pick any specific tool ("tool a")and compare it to a thousand other tools and argue that "tool a" is crap because it doesn't work as well as tool b, c and d in very specific edge cases but all you're doing is trying to win an argument via death from a thousand papercuts rather than forming a sane basis for analysis. To be honest I think it's probably best we give up on this discourse because your points are all over the place. This feels very much like you've got an axe to grind and no practical experience to back up your prejudices.