8 ms·
Borgbase backups have been unavailable for 3 days
- smcleod 3y agoTheir current ETA for service restoration is another 2-4 days. Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%. They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture. Opshugs to the folks at Borgbase. -- > box-us10 offline for storage expansion > We are currently experiencing a temporary outage on our box-us10 server due to an unplanned expansion process. We added two new hard drives to the server in an effort to enhance capacity and accommodate future growth. However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. > To prioritize the safety and integrity of your existing data, we have decided to keep the box-us10 server offline until the expansion is successfully completed. It's estimated that about a week of downtime will be needed for the expansion. We understand the inconvenience this may cause and sincerely apologize for any disruption to your services. > Current progress: 26% (Sunday morning UTC) > Date Created: 2023-08-11 16:17:36 (3 days ago) > Last Updated: 2023-08-13 18:15:53 (15 hours ago)
- deleted 3y ago[deleted]
- CamperBob2 3y ago>unplanned expansion process Sounds like an Elon Musk euphemism after a bad day at SpaceX.
- bloopernova 3y agoSomeone added disk to an array without understanding what would happen next.
- trallnag 3y agoWhat happens next? Shouldn't the array continue working, albeit with significantly degraded performance?
- bloopernova 3y agoWell, in my sysadmin days we would have test/staging equipment that we'd make sure we knew what would happen when a change was made. But I don't know the details of what really happened, so I can't make a judgement really. If I could edit my previous comment I would, to be less emphatic that someone did something "wrong". It could have been that this particular disk array had different firmware or was faulty. All we know is that something unexpected happened.
- bloopernova 3y agoSorry, I didn't quite answer your question! Yes, the expected behaviour would be for the disks to be initialized. Beyond that, it'd be a configuration setting for what to do with new disks. It could be used to extend the existing LUNs, or added as a new pair in a RAID 10 style setup, or added as new members of a different level RAID. Then it would be up to the sysadmin to extend the LUN, or divide the newly added space among multiple LUNs. Different arrays have different behaviour with new disk. Some only add to offline LUNs. Some do everything online with lower performance, as you said. Even some super expensive enterprise kit will be very easy to break by adding new disk.
- trallnag 3y agoThanks for the insightful answer!
- x0x0 3y ago> me wonder about their redundancy and recovery architecture You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...
- dikei 3y agoYeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.
- x0x0 3y agoI suspect they can't. Abort and rollback, that is. Evidence: it's not rational to eat a week of downtime if you have an alternative. edit: Which tells you something about their backups, and it ain't great. Or they'd restore onto another box.
- TylerE 3y agoThis is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.
- istjohn 3y agoIf you have multiple backups, you don't have to trust anyone.
- greenie_beans 3y agoexcept yourself !
- crossroadsguy 3y agoActually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple). If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.
- shrubble 3y agoAre they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?
- colechristensen 3y agoIt would appear to be something like that. > However, we did not anticipate that the expansion process would require the server storage to be temporarily offline. This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in place to to dry runs in non production or any of the other things that would protect against something like this. Really questionable to trust an organization that makes a mistake like this.
- js4ever 3y agoIt's not an org but a one man show. I'm using borgbase myself, really happy with it until now and I have to admit I'm now reconsidering this choice...
- jskrablin 3y agoOne man show has a bus factor of 1. Which is kinda bad for anything critical...
- rsync 3y ago"This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category." You have no idea what happened. Protecting against "something like this" often introduces complexity and failure cascades that are much more harmful than a simple system being offline. In fact, I would consider it a feature if uptime was deliberately sacrificed for simplicity and data integrity. I wish Manu the very best and will consider it not a failure but a success if this array/subsystem/whatever comes through without data loss.
- kiririn 3y agoA vague description or white lie might have been better here - a storage company with a fleet of servers shouldn’t ever be in this situation for the reason given
- smcleod 3y agoPersonally I appreciate the honesty and look for it in vendors, there are too many that cover up the truth (MS/Github etc...). However I agree that it doesn't look good for a storage company.
- TheNewsIsHere 3y agoI agree with you. I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation? I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming project.
- SteveNuts 3y agoI've been in this exact situation before in a storage expansion. With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards. It is dangerous because the rebuild stresses the drive which makes a failure more likely, and a failure during the rebuild process is not great, on top of that you've got a months long ETA... No fun
- noAnswer 3y ago> I've been in this exact situation before in a storage expansion. Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)
- ddtaylor 3y agoI guess now is a fun time to describe my cheap backup process. All of this is related to Linux and the backups are fully encrypted client-side. Under the hood `borgbackup` is used, but I use the PikaBackup app (see Flathub) to make it easier to interface with. First I have a local copy of my data living on my actual hard drive. Next I use PikaBackup (via borg) to encrypt and sync that to a cloud server I run that has about 200GB of storage for these backups for about $1/mo added cost. Next I use the Backblaze Cloud tool to synchronize those encrypted backups to B2 using `b2 sync --delete` command. It runs automatically via cron every night. The costs here are about $0.01 per month since I only get charged what I actually use. Backups are pruned and I can easily control the schedule, copies, etc. If I need to recover a file it mounts as a file system that I can easily navigate using any tools I want, including the command line. I can also mount older or different snapshots.
- willbush 3y agoI do something similar, but with restic. I wrote about it here: https://willbush.dev/blog/synology-nas-backup/ https://willbush.dev/blog/synology-nas-backup/ if anyone is interested.
- dikei 3y agoSimilarly, for my Linux devices I use `borg` to backup to a local NAS, and use `kopia` to have another backup to Google Cloud Storage, as B2 too slow where I live. I used to have back up on local external drives too, but stopped doing that, since the process was manual and I often forgot about it.
- rexreed 3y agoWhich cloud server do you use for your backups for the $1/mo in added costs?
- fjfuvucucuc 3y ago[dead]
- somishere 3y agoI've a similar local / S3 (compat) strategy but use Kopia. I currently only backup remotely to B2 (risk taker!), but could easily and cheaply add redundancy in here .. e.g S3 / R2 / random hosting. It's very cheap ~US$1. UI and default strategies are perfect for what's required. Completely automatic unless I need to browse some files in which case I just open the Kopia UI. I haven't used it, but understand Kopia also supports rsync. Edit: previous comment on same (not a shill, I swear!): https://news.ycombinator.com/item?id=34152369 https://news.ycombinator.com/item?id=34152369
- bloopernova 3y agoMy borgbase backups have been great, although according to my dashboard I'm on a different server instance. I don't envy their sysadmin team right now. Good luck, folks.
- bomewish 3y agoisn't it one guy??
- chanandler_bong 3y agoThe fact that as a customer that is impacted by this, I only found out about it when my backups and automated test restore failed is worrying. I get that things happen, but not realising that this was going to be a service impacting event does not inspire continued confidence. Sure, this situation probably won't happen again, but what else don't they understand about their infrastructure?
- exhilaration 3y ago> automated test restore failed You automatically test restore? That makes sense but I've never heard of that before, can you describe the process?
- semi-extrinsic 3y agoNot OP, but I would guess it's something like this: 1. Make a e.g. 30MB file of random data 2. Copy it to "_reference" file 3. Upload the file to backup service 4. Restore the file from backup service 5. Diff restored file against reference
- adobrawy 3y agoI don't use borg, but I used duplicity, which offers something like that. The verify operation simulates a backup restore to compare whether the restored file's checksum matches that expected from the metadata and optionally against the local file. I use this routinely, interesting to see that a local S3 provider can sometimes mess up your files silently.
- chanandler_bong 3y agoPretty simple, really. Pick a couple random files that should be in the repo, restore them from a random archive, check the md5sums against the source. If the md5sums don't match (or the file can't be found), something is wrong. I am mainly backing up RAW image files, so they should never change. Basically... $TEST_FILE=$(ls -p /source_dir | grep -v / | shuf -n1) $TEST_ARCHIVE=$(borgmatic -c config.file list | shuf -n1) borgmatic extract yada yada yada md5sum $TEST_FILE restored_file
- muppetman 3y agoI've loved BorgBase up until this point. I mean at least they emailed me quickly to tell me, but now I don't have backups. Well that's not true, I backup to a number of locations and BorgBase is just one of them. But it's a chink of my redundancy lost and a huge outage like this for a week makes me wonder what I'm paying for. What if BorgBase WAS my only backup solution and I needed to roll back? Everyone makes mistakes and I appreciate they were very upfront about it, but yea. Questioning if I'll be an ongoing customer now.
- countula 3y agoTIL Borgbase. The UI looks great for snapshot-oriented backups, and it's good to see them fund borgbackup development. rsync.net does not support append-only mode, despite advertising it: https://news.ycombinator.com/item?id=32756653 https://news.ycombinator.com/item?id=32756653 Unavailability is occasionally expected. Don't rely on a single provider to be your sole backup. Any one have any experience with Hetzner Storage Boxes for this?
- exitheone 3y agoBeen using hetzner storage boxes for borg backup for >5 years. Works without issue for multiple TB of backups. Every maintenance downtime so far has been announced in advance.
- AnonHP 3y agoThe main reason I’ve avoided Hetzner storage box is because they don’t accept loading money into the account as a prepaid credit for a longer period. It’s always a month-by-month payment. Ok, so they do allow it, but only through a bank transfer, which is either not easy or possible for those outside the EU. If they allowed this through credit/debit cards or PayPal or some other mechanism, I’d surely try it. Hetzner in general seems to be very conservative in handling (and avoiding) risks. I’d like to prepay for a year or longer for critical services so that in case something goes wrong (card expired, didn’t notice reminder emails, temporarily off the grid, temporarily incapacitated), things just don’t go poof.
- wiredfool 3y agoI’ve got an auto credit card setup on one of my hetzner accounts.
- lifeofguenter 3y agoHetzner Storage just lost a couple of snapshots a while ago… If you want “perfect architecture” and reliability/availability you just need to pay for it (aws/s3). Or make use of multiple providers.
- alex3305 3y agoI used their trial for a bit to test it out with Vorta [1] in a container. Vorta (and Borg) seemed to work fine, until I wanted to restore an archive and I noticed that my recent snapshots were completely empty. Probably because of a misconfiguration on my end though. But it made me look elsewhere. For me backups should be a fire, test and forget solution. Recently I made the switch to Kopia [2] which seems to have feature parity with Borg (and Restic [3]). It also has a web UI which is way easier to work with than Vorta. And I can easily view, extract and restore individual files or folders from there. This gave me way more confidence about this solution. The only thing I really miss is that I cannot chose different targets for different paths. For instance, with Borg I was able to backup a partial of my Docker appdata to an external source. And I haven't found a way to do this with Kopia. Besides that I'm pretty happy with this solution and I would recommend it. 1. https://vorta.borgbase.com/ https://vorta.borgbase.com/ 2. https://kopia.io/ https://kopia.io/ 3. https://restic.net/ https://restic.net/