14 ms·
blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler
by cperciva 3y ago
blinks
Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.
- gfv 3y agoHow long do you keep the transaction logs before rewriting them? I too had a few EC2 instances go down with signs of being severed from the EBS in the recent couple of weeks; mine were in eu-west.
- cperciva 3y agoThere's a continual background cleaning process which depends on the amount of storage which can be reclaimed -- there's a tradeoff between cleaning too slowly (and paying for wasted storage) and cleaning too fast (and paying for lots of S3 operations). I think it averages a couple weeks right now.
- throwawaaarrgh 3y agoAre you gonna switch to us-east-2?
- stigz 3y agoWhy would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.
- deleted 3y ago[deleted]
- IntelMiner 3y agoI'm curious how the prices shake out against services like Wasabi, since it's just dumping to an AWS S3 bucket Wasabi does $7/TB with no ingress/egress fees. My NAS is set up to rclone to it about once a day and I've yet to have any problems
- crossroadsguy 3y agoI haven’t checked the pricing in a long time but you can use Tarsnap also if you have to backup only 7.3kb (okay I might ne exaggerating here but you get the drift) and pay for only that much. You can’t do that with Wasabi et al. Also it’s really simple and does what it says it does, nothing more, nothing less. In today’s everything convoluted and bloated world this is a luxury imho. The GUI app is also quite good and functional. Support is prompt (that is if you need it). You don’t have to worry about file being deleted just because your machine didn’t connect or backup for some time even if you keep paying (hello Backblaze) etc. I mean there’s no circus, melodrama , and cliffhangers involved. I personally would never use it backup my entire laptop, due to price alone. But I have a subset of VVI files and Tarsnap is one of more than one backups for those files. So for that use-case Tarsnap is perfect for me, so far.
- Aeolun 3y agoBackblaze has kept my ‘shutdown two years ago’ machine data without issue. What problems did you have with them (or did others have)?
- terinjokes 3y agoBackblaze has a policy of allowing backups of external disks, but the disks have to be connected at least once every 30 days, or they'll delete the backups. I understand they want to avoid abuse, but the lack of any grace period, or ability of support to ad an override, really soured the service for me.
- Aeolun 3y agoHuh, I had to start paying them $2 more for my nonexistent PC I think, but otherwise was fine. I have only 1 TB of total storage on that PC though, so maybe that’s the reason.
- throwaway290 3y agoWhat use is SLA? If a service goes down for too long, are you really going to hire a lawyer sue it over SLA or just... use another backup?
- abofh 3y agoNot even then - most SLAs say that if it's breached, you pay less. Not that you get money back
- AceJohnny2 3y agoIt's not about suing, but defining expectations about how you can rely on a service. For example, my team has people across the world for HW bringup, so we can't allow our code hosting or CI to be down for more than a few hours. Of course, backups have different uptime requirements, but as for everything, it's a tradeoff between features, of which an SLA is one. Tarsnap's features are granularity of cost, reliability of storage, and encryption, but not 99.999% uptime.
- playingalong 3y agoThen SLO (service level objectives) should be enough.
- AceJohnny2 3y agohuh, never heard of SLO before.
- toong 3y agoand now also go google what SLI means ;-)
- lifeisstillgood 3y agoVery roughly: SLI - Service Level Indicators - Metrics ie Latency of each request / response cycle SLO - Service Level Objective - What threshold we are aiming for - 10 ms from request to response averaged over 1 hour period. SLA - SL agreement - contract with custom yet what happens if we breach (credits given, put the CTO in stocks and throw eggs at him etc)
- crossroadsguy 3y agoYou really shouldn’t if that’s a major concern for you and that is a valid concern. For the same reason I’ll never use PurelyMail otherwise it’s perfect. I know you didn’t ask me — but I don’t think Colin can answer differently other than saying that he is training a family member or friend to take over if needed. Here’s more https://news.ycombinator.com/item?id=7514753 https://news.ycombinator.com/item?id=7514753 this is also linked there http://mail.tarsnap.com/tarsnap-users/msg00846.html http://mail.tarsnap.com/tarsnap-users/msg00846.html Very old threads but I am not sure much has changed there https://www.tarsnap.com/contact.html https://www.tarsnap.com/contact.html Why would you use it instead of restic? Well, for pricing in pico dollars ;-) and for it has a functional GUI with tiny system footprint and that there really aren’t many such solutions out there.
- k8sToGo 3y agoIf the data is super important you should be setting on two different providers anyways for backups.
- VWWHFSfQ 3y agoHonestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?
- ilyt 3y agoThe stuff I don't want to fuck around searching re-downloading from torrent for example.
- rocqua 3y agoMy nginx config is not super important. My old reports written for study are not super important. My pirated movie copies are not super important. These are examples of data that I could easily live without. Where losing it would either be a matter of re-doing old work, or just forgetting about old and minor things.
- ivanhoe 3y agoAnything that you stashed just for convenience, but you could re-download or re-generate it if really needed, or simply live without it... frankly, like 90% of stuff on my disks fall in the category "I'll read/view it one day", which in reality I'll probably never have time or patience to open ever again.
- nixpulvis 3y agoStrange, 90% of the things trapped in my flash memory are system files.
- vntok 3y agoYou should get another drive and reserve it to Data. The cost is negligible and it really makes everything much simpler. Optimizing your system or upgrading it just becomes a "trash boot drive and reinstall" operation, applied without a care in the world.
- amluto 3y agotarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult. (FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be far weaker.)
- e1g 3y agoYep, S3 is reasonably easy to configure for immutability. I personally use restic to send (encrypted) blobs to https://www.borgbase.com https://www.borgbase.com which has append-only mode and monitoring to warn me if some backups didn't happen.
- crossroadsguy 3y agoborgbase is another “little” service that I use and like just like tarsnap and to some extent rsync.net. And they also have an excellent gui app Vorta (it’s FOSS; borgbase dev is the maintainer).
- ivanhoe 3y agoIt's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 months ago and is more clueless on their system than myself...
- tomjakubowski 3y agoWhy the scare quotes? I would expect any well-experienced power user to know a complicated system better than a fresh engineer two months into working on it, with no previous experience on the system. Especially if the power user is an engineer themself.
- andrewmunsell 3y agoI hate to bring this up, but what about the bus factor? If Colin is physically unable to continue maintaining the service and something like this happens again, how will anyone be able to get their data out? It's not really a concern about the service Tarsnap provides today
- Alupis 3y agoThere's an old Sys Admin saying (perhaps from Allan Jude of ScaleEngine) that goes something like "if your data doesn't exist in at least three places, it doesn't actually exist at all..." That is to say, if Tarsnap is the only place you've keeping sensitive/important data, then you're "not doing it right" as a backup. Things happen... your hard drive can die suddenly, and a data center bursts into flames all on the same day.
- dryrun 3y agoI feel like ovh will never stop earing about this. This has been, frankly, a traumatic event for many sysadmins I believe, and one that was shared by many from the same source, which is quite different from the standard variation of "that time when I erased the production database" (looking at you gitlab, but also at myself!). I mean, at this point it's between a legend and a warning tale and I don't know what else to call it. A bad Wednesday probably.
- 88913527 3y agoEven large organizations can have fairly regular availability issues. I appreciate the noted flaws of "single point of failure", but I also see orgs where 100s of people have access to the infrastructure, make a change, and then it breaks something. I wouldn't do business with an org just because they have many people, that won't mean they're operationally sound, at least not to my expectations.
- twic 3y ago> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service Hence the toddler.
- hunter2_ 3y agoAre you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!
- molsongolden 3y agoRaising the toddler to have some help running the business.
- saurik 3y agoNot just help: there is now a clear heir to take over if (the gods forbid) cperciva ever succumbs to illness or is defeated in battle.
- cperciva 3y agoMight take a while. Tarsnap has never had an employee without a doctorate. She's a very bright girl but I'll be surprised if she gets her doctorate before 2040.
- mdda 3y agoSo you're the one in charge of the unix epoch rollover?
- devonkim 3y agoI am really confused by the communication thread and am interpreting that the toddler is somehow in charge of running the infrastructure as a joke. Yet I can’t see it as either a joke or serious. I’m a native English speaker but sometimes I swear I’m losing grasp on communication in the Internet age and am sincerely trying to understand this all.
- Mawr 3y agoUptime isn't an important property of a backup solution, so I'm not sure where the expectation comes from?
- vntok 3y agoIt sure should be up when you need it, exactly at the time you need it.
- mike_d 3y agoThis was an extremely well written and thoughtful postmortem, but I hope to never see one from you again. :)
- Tepix 3y agoIt was a postmortem without the mandatory "how can we prevent this in the future" steps…
- bluehatbrit 3y agoI think that's a little unfair given what was in the postmortem. It may not be a separate section with the key points, but the information is all there of what the issues were and what the solutions are. I think it's fair to assume they're actually acting on those without them needing to be reiterated at the bottom of the page.
- Tepix 3y agoWell, for sure he has fixed several bugs, but he didn't say that he would be testing his disaster recovery procedure every year in the future for example.
- cperciva 3y agoYes, rehearsing the process every year is the main lesson learned. Sorry, it was getting late and I wanted to get the email out so I cut it short.
- acedTrex 3y agoI agree, we don't really need a "key points/future actions" section that boils down to "The service will be geo redundant"
- tptacek 3y agoIn 15+ years of running this service, this is one of two (2) postmortems he's ever published, and the first in eleven (yes, 11) years.
- dharmapure 3y agoThank you for the post-mortem Colin and I hope you get some sleep!
- cperciva 3y agoThanks, I did! My long suffering wife was up at 3:30 though. :-(
- bombcar 3y agoTime to get your toddler providing round-the-clock support! ;) Have been having some luck reading https://www.amazon.com/No-Cry-Sleep-Solution-Toddlers-Preschoolers/dp/0071444912 https://www.amazon.com/No-Cry-Sleep-Solution-Toddlers-Presch... - available everywhere libraries (blockbuster for books!) are found.
- cperciva 3y agoShe's generally a wonderful girl. Right now she's dealing with her second molars coming out and just picked up a cold though, which is throwing off her sleep schedule.
- jacquesm 3y agoIn future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.
- idlewords 3y agoIt always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve. A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.
- bityard 3y agoThat's an overly cynical take, post-mortems are not for anyone's reassurance, they are a learning opportunity. The airline industry is as safe as it is because every accident gets thoroughly investigated with detailed reports ("post-mortems") including what to do differently going forward. These are taken as gospel among all players in the industry and as a result, you very rarely see two different accidents caused by the same thing anymore.
- jacquesm 3y agoThat was entirely not what I was getting at and is a cheap shot that is well beneath you, especially because I suspect that you know that that wasn't what I was getting at.
- idlewords 3y agoMy comment wasn't intended personally; your words about "will never recur" just reminded me of this peculiarity of software systems, where it's often error handling/monitoring/backups/etc. that cause cascading failures in the systems they're intended to safeguard. I'm sorry if I misconstrued your meaning, but I am flattered that you think there are things beneath me!
- 3y ago
- nodesocket 3y agoSome recommendations on the AWS front (not sure if some of these are already implemented since the postmortem does not go into AWS details). - Setup nightly automatic snapshots of EBS volumes (this is supported natively now in AWS under lifecycle manager). - Use EBS volumes of the new GP3 type, and perhaps use provisioned IOPS. - Setup a auto-scaling group with automatic failover. Of course increases cost, but should be able to automatically failover to a standby EC2 instance (assuming all the code works automatically which the blog post indicates is not currently the case).
- e63f67dd-065b 3y agoCan you say a bit more about the log-structured S3 filesystem? I wrote something very similar recently (https://github.com/isaackhor/objectfs https://github.com/isaackhor/objectfs) and I'm curious what made you settle on that architecture. The closest thing I know of that's similar is Nvidia's ProxyFS (https://github.com/NVIDIA/proxyfs https://github.com/NVIDIA/proxyfs)
- nextaccountic 3y ago> the central Tarsnap server (hosted in Amazon's EC2 us-east-1 region) What prevents you to distribute load among other regions? (Also: did you ever think about abandoning AWS?)
- rlt 3y agoNice write up. A couple questions: - The use of “I” begs the question: what’s the “bus factor” of Tarsnap? If you were unavailable, temporarily or permanently, what are the contingency plans? - Will you be making any other changes to improve the recovery time, or did the system mostly function as designed? For example having a hot spare central server?
- LinAGKar 3y agoWhat I'm wondering is, I had data on Tarsnap, why am I only hearing about this now?