19 ms·
Auth0 Down
- rntrg 5y agoEven their status page was showing a Heroku down page for a bit.
- dualvectorfoil 5y agoI believe it still does: https://status.auth0.com/ https://status.auth0.com/
- webo 5y agoTheir own app is down too https://manage.auth0.com/ https://manage.auth0.com/
- rntrg 5y agoAnd their support center is down as well, so the offline ticket form is displaying, but prompts for login and fails on submit.
- dualvectorfoil 5y agoWhat alternatives to Auth0 are worth looking into? Between this P0 (with no ability to check the status or file a ticket) and the Okta acquisition, I hesitate to continue using Auth0 as the default when spinning up new web apps.
- aliswe 5y agoI dunno, Auth0 is still pretty stellar. Its not perfect, but the rate of disruptions isnt very significant IMO. (famous last words)
- sadturnip 5y agoAWS Cognito works but it is a far cry from how usable Auth0 is.
- zedpm 5y agoEvery year or so I check back in to see if Cognito has gotten any better. I hasn't. I'd love to use it, but some very basic things are just not done correctly. Yan Cui has a nice writeup on Cognito [0] that touches on some of its shortcomings. [0]: https://theburningmonk.com/2021/03/the-case-for-and-against-amazon-cognito/ https://theburningmonk.com/2021/03/the-case-for-and-against-...
- StevePerkins 5y agoMy blood pressure is still coming back down from AWS having an all-day outage during a holiday week last year, because Kinesis had an issue and it turns out their entire infrastructure depends on that.
- r3b00t 5y agoAgreed! just got done replacing Cognito across the board with Auth0, glad we hadn't pushed it to prod yet.
- SahAssar 5y agoIf you're okay with self-hosting then keycloak or ory.sh seems like viable alternatives.
- spacemanmatt 5y agoI'm just getting into prod with keycloak. It's pretty amazing for free.
- Nextgrid 5y agoWhat's the point of such services? Every web framework worth its salt supports all common authentication schemes and offers an easy interface to write your own.
- CA0DA 5y agoOkta?
- sk5t 5y agoOkta is in the process of acquiring Auth0.
- ycmimi 5y agoYou can try Azure AD B2C.
- thinkharderdev 5y agoKeycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally. Outside of that they Ory ecosystem is really nice. We user Hydra, which is not a drop-in Oauth2 server but requires you to write several of the components yourself.
- tyingq 5y ago"Keycloak is pretty battle-tested and rock solid although it is a bit of a behemoth operationally." Just looked. Java, Wildfly, Infinispan, Hibernate + A DB, network multicast if you want a cluster, your own separate load balancer, etc. So, a fairly large time investment if you aren't familiar with all of that.
- thinkharderdev 5y agoYeah, it's not the easiest thing in the world to get up and running but not quite as hard as it might seem at first look. If you are already using k8s then you can use https://github.com/codecentric/helm-charts/tree/master/charts/keycloak https://github.com/codecentric/helm-charts/tree/master/chart... to deploy Keycloak fairly easily. If you're not using k8s then it is probably more of an undertaking.
- aliswe 5y agoLogin works for us, it seems.
- nightowl_games 5y agoWow. The whole point of paying someone like Auth0 is to _not_ have this happen. This is basically their whole point, is it not? Really looking forward to the post-mortem, but I won't ever forget just how down this thing is right now.
- vinayp10 5y agototally
- TrogdorTheMan 5y agoYup, exactly. We're actively working on ripping them out of our software, this is ridiculous. I knew we should have taken the time to make a fail safe...sigh...
- Kiro 5y agoThe whole point for me is that I don't want to be responsible for user credentials. I don't trust the security of my app.
- SahAssar 5y agoIf you don't trust your app with credentials how can you trust your app with what those credentials are used to access?
- ProAm 5y ago
- vinayp10 5y agoStill down our whole app is down
- webo 5y agoThe Okta effect?
- salty_1 5y agoIf anything Okta would help them with this. Never experienced an Okta outage.
- jamwt 5y agoThis is crazy timing -- my co-host and I just released a Podcast episode yesterday sifting through the details about the Auth0 database-related outage from 2018 ( https://downtimeproject.com/ https://downtimeproject.com/ ). I'll be curious to see how much overlap or not there is with that previous outage. They wrote up a nice post-mortem back then, so hopefully we'll get another one this time.
- TrogdorTheMan 5y agoDo you have a link to that previous post-mortem?
- goshx 5y agohttps://cdn.auth0.com/blog/20181128-Incident-RCA.pdf https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf
- TrogdorTheMan 5y agoheh, smells like its similar. thanks for the link.
- twistedpair 5y ago> However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment They still don't know what's going on. This is concerning. Already down for two hours of prime time.
- stixx 5y ago> We have identified that this issue is coming from our database. However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment, and the full Auth0 team is engaged in resolving this.
- ramraj07 5y agoEvery SaaS application has probably blown past its SLA thanks to this. Not sure if anyone will trust this site anymore.
- TrogdorTheMan 5y agoThey're fast approaching the 95% availability mark for the month! Enterprise customers are already guaranteed 50% return on fees, I suspect they'll be giving out MUCH more than that.
- lukeramsden 5y agoDoesn't seem to be affecting us (UK company)
- politelemon 5y agoThe outage is affecting the US tenants, the EU ones are working fine.
- TrogdorTheMan 5y agoWe managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again". Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.
- mehphp 5y agoWasn't their previous major outage because of a bad migration?
- TrogdorTheMan 5y agoI don't think so, I think that it was a combo of malicious intent and some indexes that never got run. I guess you might call it a bad migration since indexes didn't get run, but that seems more like a catalyst than a root. https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf https://cdn.auth0.com/blog/20181128-Incident-RCA.pdf
- tyingq 5y ago"something to do with our DB" Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site? As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.
- TrogdorTheMan 5y agoFrom what I've seen on some internal email chains, there is a fail-over process for HA multi-region/site, but it didn't work right. Whomp whomp.
- energy_software 5y agotheir dev is up, but prod is down for several hours mid day, INSANE. My users are asking me to have a failover... hard to explain they ARE the redundancy plan! Anyboy else have ideas here on how to explain to users?
- SahAssar 5y agoDo you mean that your plan was to hope that they have a redundancy plan?
- jimschwoebel 5y agoWaaaaaaaa come on guys and gals!
- jimschwoebel 5y agoAnyone down to build Auth0.1?
- stixx 5y agoThis is feeling more and more like a malicious hack.
- iamleppert 5y agoAbsolutely insane. Already have a task in to move all our authentication away from them.
- wholien 5y agoWow, finding out just how many services use Auth0 because of this. Like I can't log into Segment (getting a 500). I guess Okta did buy something core to the Saas ecosystem
- TrogdorTheMan 5y agoOh lawd its starting to come back up! Pingdom monitor is flapping and we're starting to get some successful, albeit slow, logins!
- romanhotsiy 5y agoYes, looks like it started to work for us too a few minutes ago.
- zedpm 5y agoDon't forget to request your credit per their SLA[0]. You have 10 days to request your credit, which by my calculations should be 10% of this months' charge. Not a fair trade for leaving us dead in the water for 4 hours, but SLAs in general are worthless. [0]: https://auth0.com/docs/support/services-level-descriptions https://auth0.com/docs/support/services-level-descriptions
- TrogdorTheMan 5y agoI'm willing to bet we can get at least the 20% return, they're still "officially" down. I bet it'll be a few hours till they say we're good. Everyone make sure you've got detailed tickets into their service queue. Get your creds!
- deleted 5y ago[deleted]
- AgentK20 5y agoThis type of incident is exactly why I dislike identity federation as a service. Yes it's difficult to get right, and you open yourself up to additional risk and technical complexity to do the federation yourself, but simultaneously how many businesses are currently completely down and just sitting on their hands waiting for Auth0's engineers to fix their systems?
- wjertyj 5y agoBut how will you ever succeed if you don't outsource all but your core competencies while your core competencies simultaneously converge on banking/investment/midde-manning?
- sk5t 5y agoWhile this protracted outage has prompted some plans to get rolling with a self-hosted Auth0 substitute, bear in mind that this can't address users wanting to sign in with Google, Microsoft, LinkedIn, etc., where you'll still depend on their authentication service to work correctly, not ban your client ID, and so on.
- TrogdorTheMan 5y agoYup ^^ this. That and some dang help with managing security audits for our security team -.-. Just can't afford the man power to do it all.
- rntrg 5y agoManagement console went down for us again. 502 service unavailable.
- mattbnr32 5y agojust successfully authenticated a few times
- marcinzm 5y agoTheir status page is pretty depressing to read since all it really says, repeatedly, is "we're really sorry" and "we're working as hard as we can." You can just feel the abuse the person writing that must be getting from customers. I feel bad for them since they have no power to fix this and didn't cause the issue. No root cause or resolution yet and it's been 4 hours. Doesn't bode well for getting this resolved soon.
- thinkharderdev 5y agoHa, whenever some 3rd party SaaS tool is down and I am obsessively refreshing their status page I'm always enraged with the lack of updates. But then later I think about what if I were one of the engineers trying to bring things back up. There is nothing worse than being heads down trying to get a mission-critical system back online and being constantly distracted by demands for "status updates." What is the ETA on getting back online? Well, about 5 minutes longer than it would have been now that you pulled me away from what I was doing....
- quesera 5y agoIn my experience, Good organizations have scheduled updates for critical outages. No one pesters anyone, and everyone knows that a five minute update is expected periodically, e.g. at the top of every hour. Great organizations instantly spin up a response team which includes a communications liaison, who is technical enough to participate in the reponse, but is not presently leading it. They assist and communicate, either to the PM or to customers directly. ...and other organizations burn through humans until they get a handle on how to be better. That said, in these days of AWS and GCE, you can go a loong time without running into the kinds of failures that used to cause critical, extended outages.
- thinkharderdev 5y agoYep, there is definitely a way to do it well and having scheduled, regular updates (as opposed to ad-hoc requests) is key. But when "the big one" comes then it is hard to maintain the discipline in that process.
- twistedpair 5y agoFirst RCA is out: https://cdn.auth0.com/blog/Summary_Root_Cause_Analysis_(RCA)_04-2021.pdf https://cdn.auth0.com/blog/Summary_Root_Cause_Analysis_(RCA)... US-1 downtime: 4:20min. Cause: database query contention
- tedd4u 5y agoThis is the "quick" version. They write "we expect to provide our detailed RCA for this incident by 5:00pm PDT (24:00 UTC) on April 29th, 2021, or sooner."