3 ms·

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of rai

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be. Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

x0x0 3y ago
And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.) ps -- it's not about being bashing these folks. But these type of multiple operational incompetencies strongly call into question their ability to safeguard your bytes and produce them when you need them. It's kinda the whole point of the thing.
- wmf 3y ago
  It's a backup company. I can understand that they don't back up the backups.
  - colechristensen 3y ago
    It’s a backup company so I would expect the store 1 copy plus the minimum redundancy to make drive failures a predictable amount of unlikely to cause data loss. More than that would be additional features.
  - justinclift 3y ago
    If there's one thing you'd expect from "a backup company", it'd be that they're really good at doing backups. :) That does include their own of course.
viraptor 3y ago
Possible, but that's still guessing. Also possible: extra load while adding storage caused multiple drive failures, so suddenly they're without spares and want to finish the migration as soon as possible, so they stopped the customer traffic. "did not anticipate" can be interpreted in different ways too. It may be "didn't know it could happen", or just "estimated it to be so unlikely that emergency downtime was acceptable as a response". I've seen outside speculations as an inside person before and they're often very likely to be BS. Let's stick to facts and skip the "obviously" and "not possible" until we learn more. What's happening right now may be their prepared and tested procedure. (Or it may not)
mananaysiempre 3y ago
By the way, is this a plausible scenario? (1) Add more drives, start online rebuild (2) Have one drive fail/URE mid-rebuild (3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation If so, possibility (2) could in principle not be detectable in staging, as it could depend on the drive’s age, and having to reread everything—as a rebuild does—is a rather abnormal load. The failure could even predate (1), if it happened to data nobody’d looked at for a long time. This is less a question about these guys and more in general—is a RAID/ZFS/etc. array/pool/etc. in a vulnerable state after you’ve expanded it?
- wtallis 3y ago
  > and having to reread everything—as a rebuild does—is a rather abnormal load. Far from being an abnormal load, it should be a cron job. Backups need to be verified. Data at rest needs to be periodically checked for bitrot. Reading everything on the array and verifying parity and any checksums is important preventive maintenance and an early warning system.

x0x0 3y ago

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.) ps -- it's not about being bashing these folks. But these type of multiple operational incompetencies strongly call into question their ability to safeguard your bytes and produce them when you need them. It's kinda the whole point of the thing.

wmf 3y ago
It's a backup company. I can understand that they don't back up the backups.
- colechristensen 3y ago
  It’s a backup company so I would expect the store 1 copy plus the minimum redundancy to make drive failures a predictable amount of unlikely to cause data loss. More than that would be additional features.
- justinclift 3y ago
  If there's one thing you'd expect from "a backup company", it'd be that they're really good at doing backups. :) That does include their own of course.

viraptor 3y ago

Possible, but that's still guessing. Also possible: extra load while adding storage caused multiple drive failures, so suddenly they're without spares and want to finish the migration as soon as possible, so they stopped the customer traffic. "did not anticipate" can be interpreted in different ways too. It may be "didn't know it could happen", or just "estimated it to be so unlikely that emergency downtime was acceptable as a response". I've seen outside speculations as an inside person before and they're often very likely to be BS. Let's stick to facts and skip the "obviously" and "not possible" until we learn more. What's happening right now may be their prepared and tested procedure. (Or it may not)

mananaysiempre 3y ago

By the way, is this a plausible scenario? (1) Add more drives, start online rebuild (2) Have one drive fail/URE mid-rebuild (3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation If so, possibility (2) could in principle not be detectable in staging, as it could depend on the drive’s age, and having to reread everything—as a rebuild does—is a rather abnormal load. The failure could even predate (1), if it happened to data nobody’d looked at for a long time. This is less a question about these guys and more in general—is a RAID/ZFS/etc. array/pool/etc. in a vulnerable state after you’ve expanded it?

wtallis 3y ago
> and having to reread everything—as a rebuild does—is a rather abnormal load. Far from being an abnormal load, it should be a cron job. Backups need to be verified. Data at rest needs to be periodically checked for bitrot. Reading everything on the array and verifying parity and any checksums is important preventive maintenance and an early warning system.