4 ms·

Can you educate me on the typical setup you would use to achieve that goal?

Nextgrid 5y ago
Either ZFS or some form of RAID with redundancy cranked up to the max, so that it can tolerate a high number of drives failing and still continue operating. ZFS configured in RAIDZ3 mode for example can tolerate 3 drives failing and still be able to operate (and rebuild itself once the failed drives are replaced). You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. Again, this is not an argument against backups (they're useful for other reasons), but with this amount of redundancy on the storage level, I'd expect something else in the system to die before the storage layer fails catastrophically.
- BiteCode_dev 5y ago
  Would you happen to be freelancing a bit? I'd like to hire somebody for a day to teach me those things. Remote is ok.
  - Nextgrid 5y ago
    Feel free to send me an email (address in my profile) and we can discuss. I don't think I'm the right person to teach system administration, but we can always have an informal conversation and I can hopefully point you to better resources.
  - RNCTX 5y ago
    Take a look at iocage and the FreeNAS project. They probably have the best implementation of all of this wrapped up into a cloud-like web interface.
- RNCTX 5y ago
  I would add to this that certain combinations of modern SSDs (like nvme) can put disk performance equal to or in excess of RAM performance, too, so there are other considerations besides resilience, too. I've seen people do this for DB-intenstive things like large search server deployments and map rendering. Doesn't take many nvme mirrors to get past DDR4 speed.
- toast0 5y ago
  > You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. You have to be careful. If you take three storage devices off the production line and use them as a mirrored array and one fails, there's a good chance they all fail, because you're not just dealing with storage hardware, but storage software too. HP had two batches of SSDs that failed when the power on time hit some rollover point. And I don't think they're the only one. Fabrication issues are also likely to turn up simultaneously if given equal write loads. If failures were independent events, you're spot on, but they may not be.
  - Nextgrid 5y ago
    Yes that is correct; I assumed it was somewhat common knowledge if you're going for (and have the budget for) such levels of redundancy.

Either ZFS or some form of RAID with redundancy cranked up to the max, so that it can tolerate a high number of drives failing and still continue operating. ZFS configured in RAIDZ3 mode for example can tolerate 3 drives failing and still be able to operate (and rebuild itself once the failed drives are replaced). You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. Again, this is not an argument against backups (they're useful for other reasons), but with this amount of redundancy on the storage level, I'd expect something else in the system to die before the storage layer fails catastrophically.

BiteCode_dev 5y ago
Would you happen to be freelancing a bit? I'd like to hire somebody for a day to teach me those things. Remote is ok.
- Nextgrid 5y ago
  Feel free to send me an email (address in my profile) and we can discuss. I don't think I'm the right person to teach system administration, but we can always have an informal conversation and I can hopefully point you to better resources.
- RNCTX 5y ago
  Take a look at iocage and the FreeNAS project. They probably have the best implementation of all of this wrapped up into a cloud-like web interface.
RNCTX 5y ago
I would add to this that certain combinations of modern SSDs (like nvme) can put disk performance equal to or in excess of RAM performance, too, so there are other considerations besides resilience, too. I've seen people do this for DB-intenstive things like large search server deployments and map rendering. Doesn't take many nvme mirrors to get past DDR4 speed.
toast0 5y ago
> You are very unlikely to have 3 drives fail at the same time, so you're pretty much immune to storage hardware failures. You have to be careful. If you take three storage devices off the production line and use them as a mirrored array and one fails, there's a good chance they all fail, because you're not just dealing with storage hardware, but storage software too. HP had two batches of SSDs that failed when the power on time hit some rollover point. And I don't think they're the only one. Fabrication issues are also likely to turn up simultaneously if given equal write loads. If failures were independent events, you're spot on, but they may not be.
- Nextgrid 5y ago
  Yes that is correct; I assumed it was somewhat common knowledge if you're going for (and have the budget for) such levels of redundancy.