4 ms·
Tarsnap outage 2023-07-02
Subject: Tarsnap outage 2023-07-02
From: Colin Percival
Date: Sun, 2 Jul 2023 10:13:26 -0700
The important information first: All data stored on Tarsnap is safe, but you
can't access it right now. I'm working to bring it back online.
The detailed story:
At approximately 2023-07-02 13:07 UTC the main Tarsnap server suffered a
"failed system status check" in Amazon Web Services vernacular. In plain
language, something broke -- either the server hardware, the power, or its
connection to the rest of the EC2 network (including network-attached disks).
At approximately 2023-07-02 13:52 UTC the server rebooted (probably running
on new hardware, considering that it's a virtual machine), but had filesystem
corruption affecting cached metadata. This cached metadata essentially tells
the Tarsnap server where to find user data in Amazon S3; without it, the
Tarsnap server cannot service requests. I *think* that the metadata present
is accurate as far as it goes, but it definitely lacks information about
archives stored in the last few seconds before the outage started.
While I could probably "replay" the most recently stored data from the backing
store in S3, the nature of filesystem corruption means that I can't guarantee
that this would work; since Tarsnap is a backup service I have to prioritize
safety, and so I've started the process of reconstructing the Tarsnap server
state from the "known good" data stored in Amazon S3.
If anyone needs data urgently, I can bring the Tarsnap server up in read-only
mode, subject to the caveat that the most recently stored data may not be
available. Please contact me if this would be useful to you.
It's too early in the state reconstruction process to provide an accurate ETA;
my best guess is somewhere between 12 and 24 hours.
As always, feel free to email me if you have any questions.
Sorry about the headaches,
- disadvantage 3y agoTarsnap just uses S3 for storage. Would it not be cheaper to just use that yourself, and ditch the abstraction layer of Tarsnap?
- voytec 3y agoThey use deduplication to upload differential backups for costs reduction, let you pay for months or years in advance, donate (iirc) 10% revenue to The FreeBSD Foundation and have more professional support.
- cperciva 3y agodonate (iirc) 10% revenue to The FreeBSD Foundation Tarsnap contributes an amount equal to its December operating profit to open source software. This largely takes the form of sponsoring conferences throughout the year and then the FreeBSD Foundation gets whatever is left in the budget (once I know how much profit Tarsnap made in December).
- _DMN_ 3y agoIs this an email you received? I (at work) did not get an email about this, tried a manual `tarsnap --list-archives` after seeing all the failed backup jobs, checked https://www.tarsnap.com/news.html https://www.tarsnap.com/news.html after that failed and found nothing. I had to Google "Tarsnap network outage" and find this post to have any idea something was actually wrong with the service, less than ideal.
- cperciva 3y agoYou should subscribe to the tarsnap-announce mailing list. That's where tarsnap announcements are emailed out. (And yes, the above text is what I sent to the list.)
- _DMN_ 3y agoThanks, I will do that. I do think that a backup service being down for 12-24 hours is pretty critical information that shouldn't require an opt-in though.