6 ms·
Their current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable
by smcleod 3y ago
Their current ETA for service restoration is another 2-4 days.
Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%.
They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture.
Opshugs to the folks at Borgbase.
--
> box-us10 offline for storage expansion
> We are currently experiencing a temporary outage on our box-us10 server due to an unplanned expansion process. We added two new hard drives to the server in an effort to enhance capacity and accommodate future growth. However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.
> To prioritize the safety and integrity of your existing data, we have decided to keep the box-us10 server offline until the expansion is successfully completed. It's estimated that about a week of downtime will be needed for the expansion. We understand the inconvenience this may cause and sincerely apologize for any disruption to your services.
> Current progress: 26% (Sunday morning UTC)
> Date Created: 2023-08-11 16:17:36 (3 days ago)
> Last Updated: 2023-08-13 18:15:53 (15 hours ago)
- deleted 3y ago[deleted]
- CamperBob2 3y ago>unplanned expansion process Sounds like an Elon Musk euphemism after a bad day at SpaceX.
- bloopernova 3y agoSomeone added disk to an array without understanding what would happen next.
- trallnag 3y agoWhat happens next? Shouldn't the array continue working, albeit with significantly degraded performance?
- bloopernova 3y agoWell, in my sysadmin days we would have test/staging equipment that we'd make sure we knew what would happen when a change was made. But I don't know the details of what really happened, so I can't make a judgement really. If I could edit my previous comment I would, to be less emphatic that someone did something "wrong". It could have been that this particular disk array had different firmware or was faulty. All we know is that something unexpected happened.
- bloopernova 3y agoSorry, I didn't quite answer your question! Yes, the expected behaviour would be for the disks to be initialized. Beyond that, it'd be a configuration setting for what to do with new disks. It could be used to extend the existing LUNs, or added as a new pair in a RAID 10 style setup, or added as new members of a different level RAID. Then it would be up to the sysadmin to extend the LUN, or divide the newly added space among multiple LUNs. Different arrays have different behaviour with new disk. Some only add to offline LUNs. Some do everything online with lower performance, as you said. Even some super expensive enterprise kit will be very easy to break by adding new disk.
- trallnag 3y agoThanks for the insightful answer!
- x0x0 3y ago> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...
- dikei 3y agoYeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.
- x0x0 3y agoI suspect they can't. Abort and rollback, that is. Evidence: it's not rational to eat a week of downtime if you have an alternative. edit: Which tells you something about their backups, and it ain't great. Or they'd restore onto another box.
- TylerE 3y agoThis is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.
- istjohn 3y agoIf you have multiple backups, you don't have to trust anyone.
- greenie_beans 3y agoexcept yourself !
- crossroadsguy 3y agoActually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple). If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.
- shrubble 3y agoAre they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?
- colechristensen 3y agoIt would appear to be something like that. > However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in place to to dry runs in non production or any of the other things that would protect against something like this. Really questionable to trust an organization that makes a mistake like this.
- js4ever 3y agoIt's not an org but a one man show. I'm using borgbase myself, really happy with it until now and I have to admit I'm now reconsidering this choice...
- jskrablin 3y agoOne man show has a bus factor of 1. Which is kinda bad for anything critical...
- rsync 3y ago"This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category." You have no idea what happened. Protecting against "something like this" often introduces complexity and failure cascades that are much more harmful than a simple system being offline. In fact, I would consider it a feature if uptime was deliberately sacrificed for simplicity and data integrity. I wish Manu the very best and will consider it not a failure but a success if this array/subsystem/whatever comes through without data loss.