4 ms·

For the third time, my plan for growing the pool is based on being able to swap in new drives to trade them out with one existing drive in current mirror vdevs.

For the third time, my plan for growing the pool is based on being able to swap in new drives to trade them out with one existing drive in current mirror vdevs. As described previously, this allows me to grow two drives at a time and reduce risk of simultaneous failure in a vdev. As I mentioned before, this sort of swapping is not possible without degrading a raidz vdev. As you have brought up, this configuration optimizes resilvering speed, aka minimizing time spent in a degraded state. Resilvering a mirror is faster than resilvering a raidz, as it requires only sequential read from the good drive and sequential write to the new drive. Resilvering a raidz thrashes all drives in the vdev and requires re-calculating parity along the way. Is it more likely that one of three drives in the degraded raidz1 fails during resilvering or that the one drive in the degraded mirror fails? That is a tough question to answer. What if all the drives are the same age? Well then the likelihood is greater that a resilver will cause a cascading failure. Hence my thrice-repeated desire to use the ability to swap new drives into old vdevs to mix ages, and my mixed-brand vdevs, as described in the root of this thread. In short, resilvering is the highest-risk phase in a pool's lifecycle. I am paying an extra drive's worth of storage to optimize around this risk, and also to realize performance benefits we have discussed upthread. Again, I will state that objectively measurable improvements combined with subjective perception of better desktop performance is sufficient for me to claim that the performance improvement matters. My non-critical data is largely recoverable. Things like VM images and installers. These can be re-created and deployed from code that is in the critical portion, or downloaded from the internet. Indeed, my DR plan does not account for broad internet outages concurrent with a drive failure. A lot of the rest of the non-critical data is media that could be re-ripped from physical media (DVDs, Blu-rays). Being recoverable, it is not worth it to me to pay for cloud storage for the rest. Even "wasting" a drive's worth of storage, it is still a lot cheaper per gigabyte-year to ... you know, not pay for that. So, why do you care so much about my specific storage setup? Does it pass muster yet, or am I still someone who is wrong on the internet to you?

justsomehnguy 3y ago
> Is it more likely that one of three drives in the degraded raidz1 fails during resilvering or that the one drive in the degraded mirror fails? That is a tough question to answer Not that tough. If the load of 4 drives working bring them to malfunction then resilvering by pairs only prolongs the agony a bit and technically you are now 2 (or even 3 or 4) times more susceptible to a failure during rebuild (foreach stripe what would fail). > Hence my thrice-repeated desire to use the ability to swap new drives into old vdevs to mix ages, and my mixed-brand vdevs, as described in the root of this thread. LVM allows that without limits imposed by ZFS. Not applicable for your setup, sure. > , and also to realize performance benefits we have discussed upthread Depends on the workload. I had very good response from RAID0/10 SATA setups a decade ago (even from WD Greens), nowadays I find it inadequate for the current bloat. > A lot of the rest of the non-critical data is media that could be re-ripped from physical media (DVDs, Blu-rays). Understandable. > So, why do you care so much about my specific storage setup? Mostly idle curiosity. Especially with my first take (using LVM) is just inapplicable for you. > Does it pass muster yet, or am I still someone who is wrong on the internet to you? Harsh. But besides the idle curiosity I'm interested in other people setups and in giving a free, no strings attached, advice[0]. To sum up your setup from my POV: You are running a RAID10 setup to have the benefits of a slightly faster disk access and ease of resilvering. If there was a need to do something with this setup (and not limited to ZFS), I would consider the following: use a part of the disks as a RAID5 (3+1) for the WORM-like data (ripped DVDs, installers etc): greater SEQ and RND access, more usable space running RAID11 for the super critical data (so you can really lose 3 drives until you lose redundancy) there are also an option to run one mirror and another mirror or two simple volumes to have a protection against the logical failure, but you have ZFS snapshots which covers this use-case, for the most part Of course this is just fishes for though at best and again with ZFS (and [1]) there is not that much leeway, less the need to do anything. [0] heh [1] https://news.ycombinator.com/item?id=37190493 https://news.ycombinator.com/item?id=37190493
- greggyb 3y ago
  Thanks for the quiz and free advice. I'll ask you a few questions in return: does it seem like I have not put a fair bit of thought into my storage needs? Do I seem ignorant of disk topologies and their tradeoffs? In fact, I have run LVM as a storage manager in the past. There are many reasons I prefer ZFS. It is more rigid in some ways, and the benefits outweigh the bit of vdev inflexibility that I have to put up with. Perhaps understanding these preferences would be worthwhile before trying to tell me how to improve. Frankly, you come across as trying to convince me that I am wrong and that you know better. There were several points that I brought up repeatedly which you have dismissed out of hand. As just one example, you insist on dismissing the benefit of improved random IOPS, despite admitting that there is a measurable improvement when running a pair of mirrors; I stated several times that it is objectively better (which you never debated) and I experience a subjective improvement in responsiveness (which you repeatedly dismiss as trivial, negligible, or otherwise unimportant. Regardless of any difference of opinion, flatly dismissing what I have to say is not a great approach to educating or advising. Forcing me to repeat myself without actually addressing the point is rude at best. So, given that you clearly do not listen to me, I will treat your advice as worth exactly what I paid for it. And I hope you will forgive me if I do not consider repeating the same ideas and dismissing what I have to say as "idle curiosity". I do not say this out of malice. I have no reason to believe your intent is anything other than you say, but your intent is irrelevant; the effect of your communication is as I have described above. Hopefully someone else reading this thread will learn something.

justsomehnguy 3y ago

> Is it more likely that one of three drives in the degraded raidz1 fails during resilvering or that the one drive in the degraded mirror fails? That is a tough question to answer Not that tough. If the load of 4 drives working bring them to malfunction then resilvering by pairs only prolongs the agony a bit and technically you are now 2 (or even 3 or 4) times more susceptible to a failure during rebuild (foreach stripe what would fail). > Hence my thrice-repeated desire to use the ability to swap new drives into old vdevs to mix ages, and my mixed-brand vdevs, as described in the root of this thread. LVM allows that without limits imposed by ZFS. Not applicable for your setup, sure. > , and also to realize performance benefits we have discussed upthread Depends on the workload. I had very good response from RAID0/10 SATA setups a decade ago (even from WD Greens), nowadays I find it inadequate for the current bloat. > A lot of the rest of the non-critical data is media that could be re-ripped from physical media (DVDs, Blu-rays). Understandable. > So, why do you care so much about my specific storage setup? Mostly idle curiosity. Especially with my first take (using LVM) is just inapplicable for you. > Does it pass muster yet, or am I still someone who is wrong on the internet to you? Harsh. But besides the idle curiosity I'm interested in other people setups and in giving a free, no strings attached, advice[0]. To sum up your setup from my POV: You are running a RAID10 setup to have the benefits of a slightly faster disk access and ease of resilvering. If there was a need to do something with this setup (and not limited to ZFS), I would consider the following: use a part of the disks as a RAID5 (3+1) for the WORM-like data (ripped DVDs, installers etc): greater SEQ and RND access, more usable space running RAID11 for the super critical data (so you can really lose 3 drives until you lose redundancy) there are also an option to run one mirror and another mirror or two simple volumes to have a protection against the logical failure, but you have ZFS snapshots which covers this use-case, for the most part Of course this is just fishes for though at best and again with ZFS (and [1]) there is not that much leeway, less the need to do anything. [0] heh [1] https://news.ycombinator.com/item?id=37190493 https://news.ycombinator.com/item?id=37190493

greggyb 3y ago
Thanks for the quiz and free advice. I'll ask you a few questions in return: does it seem like I have not put a fair bit of thought into my storage needs? Do I seem ignorant of disk topologies and their tradeoffs? In fact, I have run LVM as a storage manager in the past. There are many reasons I prefer ZFS. It is more rigid in some ways, and the benefits outweigh the bit of vdev inflexibility that I have to put up with. Perhaps understanding these preferences would be worthwhile before trying to tell me how to improve. Frankly, you come across as trying to convince me that I am wrong and that you know better. There were several points that I brought up repeatedly which you have dismissed out of hand. As just one example, you insist on dismissing the benefit of improved random IOPS, despite admitting that there is a measurable improvement when running a pair of mirrors; I stated several times that it is objectively better (which you never debated) and I experience a subjective improvement in responsiveness (which you repeatedly dismiss as trivial, negligible, or otherwise unimportant. Regardless of any difference of opinion, flatly dismissing what I have to say is not a great approach to educating or advising. Forcing me to repeat myself without actually addressing the point is rude at best. So, given that you clearly do not listen to me, I will treat your advice as worth exactly what I paid for it. And I hope you will forgive me if I do not consider repeating the same ideas and dismissing what I have to say as "idle curiosity". I do not say this out of malice. I have no reason to believe your intent is anything other than you say, but your intent is irrelevant; the effect of your communication is as I have described above. Hopefully someone else reading this thread will learn something.