13 ms·
About the Tailscale.com outage on March 7, 2024
- lmeyerov 3y agoExpiring certs strikes again! I'd recommend as part of the post mortem to move their install script off their marketing site or putting in some other fallback so marketing site activity is unrelated to customer operations critical path. They're almost there for maintaining that typical isolation, which helps bc this kind of thing is common. We track uptime of our various providers, and seeing bits like the GitHub or Zendesk sites go down is more common than we expected... and they're the good cases.
- GICodeWarrior 3y agoFurther, security of a marketing site tends to be lower priority than the product itself, and an install script should generally be secured similar to the product.
- bradfitz 3y agoYes. We're lamentably probably going to have to move it (the install script), even though it has a nice URL today. When we picked that URL, the marketing site was created and run by the same people who built the rest of the product, so it didn't seem like a concern at the time.
- oliviabenson 3y agoYou can achieve both. The only mistake you made was to half-bake the proxy (doing it for IPv6 only): proxy every http(s) request to tailscale.com. Vercel’s platform is valuable for a whole host of reasons, the networking side isn’t that important, your developers will greatly value the use of Vercel even if every request is being proxied through a web server hosting tailscale.com which responds to a request for /install.sh instead of passing it through to the marketing site. (In Google Cloud you could do it entirely with load balancing rules, no need to even run a web server)
- bradfitz 3y agoThat is exactly what I want us to do :)
- ShakataGaNai 3y agoThat's great that DevOps (or whatever their title) owns both product and marketing sites. Far too many companies (and DevOps teams) think the www site is "not important" or "not their core job" and outsource it to either a less qualified team, or out of the company altogether. From an external perspective no one cares if www going down isn't "your fault" or of "direct impact to the product". It's a corporate blackeye either way.
- wlonkly 3y agoAlas, the part you're describing as great is written in past tense.
- raffraffraff 3y ago"Far too many companies (and DevOps teams) think the www site is "not important" or "not their core job" and outsource it to either a less qualified team, or out of the company altogether" It's impossible to know because they won't admit it publicly. You are guessing based on some anecdotal experience. But then again... here's mine! I worked at a very successful SaaS that had (really not kidding) the most incompetent, lazy dope running the www site. He live-edited a "staging" version of the site on the fly (no, it wasn't private, you could access this thing from the internet, and he didn't know or care about that). When he was happy with his changes he'd destroy the live instances behind the load balancer and clone his staging instance without taking it down or running any extra checks. This staging instance was around for years and I don't think he ever bothered doing a system update. Since he didn't use git, I I'll bet that at least once he cloned a live instance back to staging to undo a bunch of bork. I lost count of the incidents. He never detected them himself, was never available to troubleshoot them and was generally a big "durrrr" when you'd finally get him on the call. Example: one time we had a "slow, intermittent errors" customer support ticket surfaced to us, not because it was our job, but because dopey was being an absolute ass to the helpdesk guys. He ran his crap in another AWS account we didn't have access to. About a day later the www site went down completely, so we got hold of the AWS account and dug in. All 5 of the instances behind the load balance were "unhealthy" for various reasons. Certs expired, disks full, apache stopped. We bounced them, restarted them and sshed in. They all had different versions of the site. It was a complete mess. Turns out dopey wasn't very good at killing the old instances and cloning staging. He was probably live-editing the instances for smaller changes if that seemed easier than a bunch of AWS console work. Unbelievably he wasn't fired and continued to mismanage the site, and we could do nothing because the head of marketing didn't listen to the head of engineering. They hated each other. The way Marketing saw it "your SRE guys couldn't fix it, they had to wait for <dopey> to get on the call". I'm not even kidding. Just more anecdotal evidence from me. You might be right.
- jonhohle 3y agoOrgitecture† strikes! † A systems architecture tightly coupled to the structure of the organization in which it was created.
- gregw2 3y agoMore widely called Conway’s law…
- hunter2_ 3y ago> it has a nice URL today `curl -fsSL https://install.tailscale.com https://install.tailscale.com | sh` wouldn't be any less nice. Append /sh if having something human-friendly at the root is desirable (SEO, etc.), and you're still at the same overall length as today.
- solatic 3y ago> Append /sh if having something human-friendly at the root is desirable Even this isn't really necessary; curl includes a default user agent header identifying the traffic coming from curl. It's simple enough to direct traffic with the curl user agent header to the script and all other traffic to a static website with directions for how to quick-install.
- hunter2_ 3y agoVery nifty, but I'd argue against it due to possible confusion in various atypical scenarios, such as: * The user wants to read the script before executing it, and their preferred reader (perhaps due to browser extension or something) is a standard browser. * The user has `curl` aliased to `curl-impersonate` in order to avoid things like Cloudflare's bot detection (a captcha that triggers on things beyond the HTTP request, like the less fancy TLS handshake of regular curl) -- https://github.com/lwthiker/curl-impersonate https://github.com/lwthiker/curl-impersonate * The user doesn't have curl installed, but has wget / lynx / some headless browser / etc. and expects any of those to work the same as curl. Not to mention, if a site encouraged users to execute an HTTP response by piping curl into sh, and the response for curl was different than the response otherwise, that just might make the top of HN for being sketchy as hell.
- solatic 3y ago> The user wants to read the script before executing it, and their preferred reader (perhaps due to browser extension or something) is a standard browser. I mean, the point of wanting to read the script before executing it is to try and protect yourself from malicious scripts that abuse the curl | sh pattern. So since it would be simple enough for a malicious actor to return something different when the user agent indicates the usage of curl, the only responsible thing to do, anyway, is to use curl to download the script to a file, read the file, then execute the file. > `curl` aliased to `curl-impersonate` So when the user uses a tool to impersonate a browser, they'll see exactly what they'll see in a browser... which are the quick-install instructions anyway, which can include a note about the user agent, if anyone actually hits this in the real world? > wget / lynx / some headless browser Which would provide the quick-install instructions to use curl :)
- j45 3y agoBrings to mind if there's a service that will monitor all the certs and their expiry. Cloudflare seems to handle a fair bit of this if you host your domain with them, but you have to use Cloudflare.
- cassianoleal 3y agoPush their expiry date as a UNIX timestamp to whatever TSDB you use and hook up an alert for when it gets close.
- nijave 3y agoThis is common in synthetic test software
- helloericsf 3y agoCloudflare has a dedicated team handling certificate-related engineering challenges. While it simplifies the process for your domain on Cloudflare, internally-facing domains remain a pain point for a lot engineering teams.
- Scubabear68 3y ago“That means the root issue with renewal is still a problem, and we plan to address it in the short term much like our ancestors did: multiple redundant calendar alerts and a designated window to manually renew the certificates ourselves”.
- lijok 3y agoThe other day I was looking for a system where we can track recurring yearly/monthly/etc tasks (such as cert rotation) and get alerted a week before and on the day. 2~ hours into my search, contemplating building my own, someone pointed out we can just use a shared gsuite calendar. How the mind overcomplicates things sometimes..
- rustcleaner 3y ago[flagged]
- nimih 3y ago> When the Third Millennium Stasis kick off, basilisks on leashes, those are the intelligence treasure troves they will stripmine your optionality with. I don't know what all this means, but generally I think it's probably not too hard to deduce an org's schedule for certificate rotation by just looking at the expiration dates on those same certificates.
- lijok 3y agoI understand and somewhat agree with what you're saying. But good luck justifying that to the business.
- _joel 3y agoI'd probably spend the effort adding ssl expiration to the monitoring system for all the certs in use. Trigger then a month/week whatever before they're due to expire.
- 3y ago
- nerdbaggy 3y agoI wonder what provider they use for their website. Sounds like a lot of hoops to jump through for IPV6 when just about any other provider has IPv6 support.
- p1mrx 3y ago$ host www.tailscale.com www.tailscale.com has address 76.76.21.21 # Vercel www.tailscale.com has IPv6 address 2600:9000:a51d:27c1:6748:d035:a989:fb3c # Amazon www.tailscale.com has IPv6 address 2600:9000:a602:b1e6:5b89:50a1:7cf7:67b8 # Amazon IPv4 uses a Let's Encrypt certificate, while IPv6 uses an Amazon certificate.
- deleted 3y ago[deleted]
- opheliate 3y agoThe Vercel IPv6 feature request & surrounding discussion makes for frustrating reading: https://github.com/orgs/vercel/discussions/47 https://github.com/orgs/vercel/discussions/47
- opello 3y agoOnly if you expand all the discussion they've hidden! Ignorance is bliss, right? > We are targeting to land support for IPv6 towards the beginning of next year. We will communicate updates on this issue. Was from 2023-10-01, I guess it's early until June 30.
- watermelon0 3y agoI'd say that the beginning of the year ends at the end of Q1, if not earlier.
- miyuru 3y agoVercel's VP of product has been asking for requirements for IPv6 there. This should be a good one. https://github.com/orgs/vercel/discussions/47#discussioncomment-4328175 https://github.com/orgs/vercel/discussions/47#discussioncomm... It painful to see tech providers go down this road, which is pretty similar to what's happening at Boeing. (Business taking over Engineering)
- PuffinBlue 3y agoMigrating to a host that doesn’t support IPv6 when it’s important to you seems…like a bad decision.
- bradfitz 3y agoSuffice it to say neither their lack of IPv6 nor its importance to us was evenly understood throughout the company.
- j45 3y agoThis doesn't seem uncommon. Why learn IPv6 until you need to. I know it has some great features.
- PuffinBlue 3y agoI very much enjoy the diplomatic phrasing of this statement :-)
- deleted 3y ago[deleted]
- speedgoose 3y agoIPv6 is important but step one is to remove the AAAA records.
- lanthade 3y agoMy thoughts exactly. If feature X is important enough to do all the silly workarounds they did then why would you in the first place choose a provider that didn’t support feature X? The choice of IPV4 + shenanigans vs IPV6 seems pretty straightforward.
- NelsonMinar 3y agoThe conclusion is hilarious: "we plan to address it in the short term much like our ancestors did: multiple redundant calendar alerts and a designated window to manually renew the certificates ourselves" Devops is so 2023. Back to ops!
- bradfitz 3y agoThat was mostly a joke. You know we're going to fix it properly, Nelson :) (But super short term, yes.)
- NelsonMinar 3y agoFair enough! But you might want to add this feed to your Google Reader, Brad :-) https://scrutineer.tech/monitor/cert/tailscale.com.rss https://scrutineer.tech/monitor/cert/tailscale.com.rss
- deleted 3y ago[deleted]
- j45 3y agoIt's a joke but also the least that should be in place while whatever fix is coming is put into place. A simple cronjob would look like it would handle it, but what usually ends up being needed with 10-15 of these types of tasks is a simple, independent bpm workflow platform that tracks whether it happened or not.. or anything else. Learned this the hard way and won't do it any other way.
- c6400sc 3y agoA calendar alert is an automated alert, is it not? </s>
- sowbug 3y agoTwo ideas for discussion. Certificate Transparency is used to account for maliciously or mistakenly issued certificates. Perhaps it could also be used to assert the unavailability of correctly issued but obsolete certificates that are believed to be purged but actually aren't. (Services like KeyChest might already do this.) Let's Encrypt is a miracle compared to the expensive pain of getting a cert 20 years ago. Rather than resting on laurels, would there be any benefit to renewing even more frequently, like daily? This might have confined the Tailscale incident to a quick "oops!" while the provider migration was still underway and being actively watched.
- striking 3y ago90 day renewal is frequent enough in my book. It's not so often as to be easy to miss, but often enough that the person setting it up can witness the first renewal cycle (if they so choose, which in this case they apparently did not).
- sowbug 3y agoRight. I was thinking of keeping the same 90-day validity but renewing much more frequently, rather than the 60-day period that LE recommends. But I can see my questions have irked other community members, so I'll leave it at that. :)
- starttoaster 3y agoI renew some of my LetsEncrypt certificates monthly, which should be plenty, in my opinion. Gets you about 2 buffer cycles to notice the certificate isn't updating and recognize an issue in your automation.
- pluto_modadic 3y agois this a good amount of not overwhelming the system? e.g. they recommend 60 days? https://letsencrypt.org/2015/11/09/why-90-days https://letsencrypt.org/2015/11/09/why-90-days - and folks say the bot process won't let you do it unless it's within 30 days of expiring? (but this might not be limited by the server-side API, because --force-renewal exists)
- agwa 3y agoWhy does the proxy need to terminate TLS? If it were just a TCP proxy, then at least the monitoring wouldn't have been fooled into thinking the certificate wasn't about to expire. Heck, a TCP proxy might even allow automatic renewal to work if the domain validation is being done using a TLS-ALPN challenge.
- fanf2 3y agoI think a TCP proxy would also work with http challenges.
- agwa 3y agoIt would, but so would an HTTP proxy. It makes make think the hosting provider doesn't use HTTP challenges.
- bastawhiz 3y agoNot that it's an amazing reason, but H3 doesn't run over TCP, and running a UDP proxy doesn't sound like a great time.
- ignoramous 3y ago> UDP proxy doesn't sound like a great time QUIC, in particular, is harder to proxy (if you're load balancing, say: https://quicwg.org/ops-drafts/draft-ietf-quic-manageability.html#name-specific-network-management https://quicwg.org/ops-drafts/draft-ietf-quic-manageability....). If it is point-to-point and you control both those points (forward A to B with ports open as approp), proxying any protocol should be straightforward, no?
- snapplebobapple 3y agoI really like these guys, I wish their pricings wasn't so ridiculous. proper access control shouldn't cost 18 bucks a month for a vpn, it's basically unsellable to management at that price and the lower tiers are unsellable without it.
- starttoaster 3y agoI'm really interested in what you're comparing Tailscale to internally, because it does way more than just VPN. What are the cheaper options, and do they also have an SSH feature, oauth authentication to the network for automation services, the ability to stand up VPN node loadbalancers in kubernetes clusters, and ACME certificate request automation through LetsEncrypt? Just to list a few features that I use from Tailscale's free tier that I don't normally think of as the job of a VPN service. And they're constantly adding new features that make it a really interesting and competitive choice, in my opinion. Honestly I'm mostly interested in this take because I'm shocked by how much they offer in the cheaper tiers.
- conception 3y agoBut if you’re comparing it to other VPNs and only need VPN the pricing is bonkers.
- starttoaster 3y agoThat's fair. I guess if you just need a VPN, it doesn't really make sense to consider a product that packs all of these VPN-adjacent features. But part of my point was how much you're able to do on Tailscale for free, so in that regard, the pricing really doesn't seem as bonkers to me, to be honest. The $6/month tier is also incredibly reasonable for unlimited users, but it is annoying, I'll grant, that they actually drop some ACL features from Free tier to Starter. I suppose that's how they funnel a large number of enterprises from considering Starter in favor of Premium. But if you actually make use of the feature set of the platform, which if you have a decent DevOps team I'm pretty sure there's tools in there that they'll love, then $18/User/month actually doesn't seem too outrageous to me.
- aktuel 3y agoThat's why I roll my VPN locally. One less party to worry about.
- johnnyAghands 3y agoWow, mad jelly their CI/CD and monitoring proceses are robust enough to trust a major rollout in December. That's a pretty badass eng culture That being said, still some unanswered questions: - If the issue was ipv6 configuration breaking automated cert renewals for ipv4, wouldn't they have hit this like.. a long time ago? Did I miss something here? - Why did this take 90 minutes to resolve? I know it's like a blog post and not a real post-mortem, but some kind of timeline would have been nice to include in the post. - Why not move to DNS provider that natively supports ipv6s? Also I'm curious if it's worth the overhead to have a dedicated domain for scripts/packages? Do other folks do this? (excluding third-parties like package repositories).
- Thorrez 3y ago>- If the issue was ipv6 configuration breaking automated cert renewals for ipv4, wouldn't they have hit this like.. a long time ago? Did I miss something her AIUI, they switched to their current setup 90 days prior to the outage. The initial cert they installed during their migration lasted 90 days. So 90 days after the migration, they had an outage.
- PokestarFan 3y agoThey're using Vercel, which lacks IPv6 support.
- smackeyacky 3y agoI've said it before and I'll say it again: expiring certs are the new DNS for outages. I still marvel at just how good Tailscale is. I'm a minor user really but I have two sites that I use tailscale to access: a couple of on-prem servers and my AWS production setup. I can literally work from anywhere - had an issue over the weekend where I was trying to deploy an ECS container but the local wifi was so slow that the deploy kept timing out. I simply SSH'd over to my on-prem development machine, did a git pull of the latest code and did the deploy from there. All while remaining secure with no open ports at all on my on-prem system and none in AWS. Can even do testing against the production Aurora database without any open ports on it, simply run a tailscale agent in AWS on a nano sized EC2. Got another developer you need to give access to your network to? Tailscale makes that trivial (as it does revoking them). Yeah, for that deployment I could just make a GitHub action or something and avoid the perils of terrible internet, but for this I like to do it manually and Tailscale lets me do just that.
- wonrax 3y agoTailscale remains useful when deploying with GitHub actions. Currently, I have my cloud VM open on an unconventional SSH port so that GHA workers can SSH into it and initiate the deployment. I plan to utilize their action [0] so that any GHA worker can access the deployment machine without exposing any ports. [0] https://github.com/tailscale/github-action https://github.com/tailscale/github-action
- nurettin 3y ago> simply SSH'd over to my on-prem development machine I use mosh and gnu screen for flaky connections. Works wonders even if you disconnect every 10 seconds.
- edvinbesic 3y agoMosh is indeed glorious, esp when tethering over a spotty mobile connection. I just wish they would support gpg forwarding, otherwise chefskiss.
- gigatexal 3y agoSurely they can automate the renewal? It seems their solution is a manual one. Am I being a simpleton?
- linsomniac 3y agoThe issue they mention is that renewals are automated, but the IPv4-hosting service noticed some extra IPv6 addresses and halted the renewals because of that. On top of that, their monitoring of cert expiry was checking the IPv6 proxy so it didn't notice the problem.
- gnat 3y agoThey made the same mistake we did at a former company — put a link to our webapp’s login page (app.foo.com) on the marketing site (www.foo.com) homepage. It wasn’t until our first marketing web site outage that we realised that our $40/mo hosting plan was not merely hosting a “marketing site” but rather critical infrastructure. That was a load-bearing $40 hosting plan. Our app wasn’t down but the users thought it was. I learned then that users follow the trails you make for them without realising there are others, and if you take one away then a segment of your user base be completely lost.
- PokestarFan 3y agoWhen I type "tailscale" into my browser, the first result is tailscale.com. I do not need to use the tailscale admin console often enough that I would go out of my way to memorize the different URL. My browser used to autofill dash.cloudflare.com when I typed in cloudflare. I visited the cloudflare.com website exactly once and now that's what shows up in the first result, and I find myself doing the same thing with Cloudflare.
- basch 3y agoTime for an industry standard subdomain. www for the website app, signin, login, entr, for the sign in page.
- MikusR 3y agoCouple years ago there was an outage at Cloudflare and their www.cloudflare.com started pointing to an unrelated 3rd party marketing website, while the dash.cloudflare.com was showing the dashboard.
- physicles 3y agoThey have monitoring for their infrastructure, right? Add 50 lines of code that connects to all public domains on ipv4 and ipv6 and alerts if the cert expires in under 19 days. Set automatic renewal to happen 20 days out. Done. I wrote this code years ago, after missing a couple ssl renewals in the early days of our small company. Haven’t had an ssl-related outage since. Edit: this is the only necessary fix, no need for calendar invites: > We also plan to update our prober infrastructure to check IPv4 and IPv6 endpoints separately.
- eacapeisfutuile 3y ago> That arrangement is deemed a “misconfiguration” by that provider, and we’ve been receiving alerts about it since rolling it out So 90 days of alerts about certs, and then certs fail?
- tczMUFlmoNk 3y agoIt sounds more like "90 days of alerts about DNS, and then certs fail". The fact that the presence of IPv6/AAAA DNS records causes Vercel to decline to auto-renew certificates seems to not have been known to the Tailscale team prior to the incident. (I haven't seen the alerts in question, so I don't know whether they made this fact clear.)
- benreesman 3y agoAnything even remotely security adjacent that TailScale as an institution even remotely fumbles even once is too dangerous for the merely mildly paranoid (like me for example). We need a better story on this.
- re5i5tor 3y ago“like our ancestors did: multiple redundant calendar alerts” Love that line