11 ms·
Auth0 Has been down for almost 4 hours now
Seems I can't link to the incident (gets marked as a deadlink), but here it is: https://status.auth0.com/incidents/zvjzyc7912g5?u=3qykby4vypfp
- twistedpair 5y agoFinal RCA: https://cdn.auth0.com/blog/Detailed_Root_Cause_Analysis_(RCA)_4-2021.pdf https://cdn.auth0.com/blog/Detailed_Root_Cause_Analysis_(RCA... TL;DR feature flag service was to blame
- inssein 5y agoLink: https://status.auth0.com/incidents/zvjzyc7912g5?u=3qykby4vypfp https://status.auth0.com/incidents/zvjzyc7912g5?u=3qykby4vyp...
- romanhotsiy 5y agoPrevious discussion: https://news.ycombinator.com/item?id=26876287 https://news.ycombinator.com/item?id=26876287
- inssein 5y agoWeird I searched auth0 and got nothing, which I was surprised by.
- okhuman 5y agowrote https://github.com/pmprosociety/authcompanion https://github.com/pmprosociety/authcompanion to try and bring auth back on-prem.
- mattbnr32 5y agojust successfully authenticated a few times
- slackerIII 5y agoHuh, that's interesting timing. I co-host a podcast that walks through notable outages, and just yesterday we released an episode about Auth0's 2018 outage: https://downtimeproject.com/podcast/auth0-silently-loses-some-indexes/ https://downtimeproject.com/podcast/auth0-silently-loses-som... Last time was due to several factors, but initially because of silently losing some indexes during a migration. I'm very curious what happened this time -- we'll definitely do a followup episode if they publish a postmortem.
- dmlittle 5y agoOoh, this is a neat podcast niche that I'll probably enjoy! If you're taking suggestion of public postmortems to talk about I recommend Github's 2018 outage [1,2] caused by a network partition. [1] https://github.blog/2018-10-21-october21-incident-report/ https://github.blog/2018-10-21-october21-incident-report/ [2] https://github.blog/2018-10-30-oct21-post-incident-analysis/ https://github.blog/2018-10-30-oct21-post-incident-analysis/
- slackerIII 5y agoThat looks great! I'll put this on the list, thank you!
- znpy 5y agoIf I could make a suggestion, I'd advertise the RSS feed in a more clear way on your website. It's a very handy way to keep up with websites without having facebook/twitter/whatever in the middle. I had to go lookup the rss feed from the html source code... edit: aaand the rss is empty.
- slackerIII 5y agoAh, good idea. We just launched this last month and haven't given the website a lot of love yet. In the meantime, try https://downtimeproject.com/podcast/feed https://downtimeproject.com/podcast/feed
- znpy 5y agothat works, thank you!
- kawsper 5y agoHi, Really interesting project, I couldn't find your podcast using pocketcasts.com, so I added it through their form here: https://www.pocketcasts.com/submit/ https://www.pocketcasts.com/submit/ They mention a few errors in your feed: Problem 1: Your podcast doesn't seem to have an author Solution: Get some credit for your work by adding the following tag to your feed: <itunes:author>Author’s Name Goes Here</itunes:author> Problem 2: Your podcast doesn't seem to have a description Solution: Add a podcast description using one of the following tags: <description>Podcast description goes here.</description> <itunes:subtitle>Podcast description goes here.</itunes:subtitle> Problem 3: Some of your episodes are missing a file length Include the file length in bytes with each episode enclosure item: <enclosure url="http://www.yourhost.com/episode1.mp3 http://www.yourhost.com/episode1.mp3" length="25209836" type="audio/mpeg" />
- ryandvm 5y agoNot going to lie, of all the things to farm out to a 3rd party, auth/users always struck me as the dumbest.
- dmlittle 5y agoIt depends on your needs. What if you provide a SSO solution in your product, your customer is using Okta (or any other IdP) and that IdP goes down? There's nothing really you can do then unless you have other means of authentication.
- Raidion 5y agoI mean, it has the same benefit of other SaaS, you get to avoid building something and can spend that dev time on building something that solves a unique problem AND you have the benefit of knowing that you get to focus 100% on your app or site's problems and features, and that you have the entirety of Auth0 focusing on keeping your authentication working. I can promise Auth0 is better at building scalable, secure, and resilient authentication solutions than most dev teams, and I've been on a team that's built out a 1000s of logins/hour and 100k requests/hour enterprise grand IDAM solution. If it's data security or something else that's your concern, you can host the data in your own database with their enterprise package. General disclaimer: I'm a paying Auth0 customer but just use it for authentication, and it saved me a hundred hours of work for a pretty reasonable price.
- tluyben2 5y agoI guess it depends on your use case; I do not really find it reasonably priced but then again, I need neither the scalability nor all the features it offers. Gotrue or supertokens are fine for what we do.
- jjeaff 5y agoI've never really worked with a language that didn't have myriad options for open source, configurable, plug and play authentication. I can't imagine spending 100 hours doing authentication.
- streblo 5y ago
- gjsman-1000 5y agoI literally, today, had a demo of SSO for my organization and was panicking over what went wrong when it wasn't working, so I had to skip it.
- 1cvmask 5y agoOut of curiosity could you take a look at an alternative to Auth0 like acmelogin? I worked on the design of dashboard of it. Are there any features that are missing in it and that we should add? https://acmelogin.com/ https://acmelogin.com/
- Graffur 5y agoNext time, do a video recording of your demo
- f430 5y agoisn't the whole purpose of using Auth0 so that this stuff never happens?
- whydoineedthis 5y agono, it's that it happens less and when it does it's not so much your engineers problems as you have already paid someone to fix it. also, security practices are supposedly better and more robust there than at your average place. i think those two things are the value adds.
- f430 5y agoim not sure i see why i should use auth0 over AWS Cognito
- UglyToad 5y agoSo I've been mulling this stupid thought for a while (and disclaimer that it's extremely useful for these outage stories to make it to the front-page to help everyone who is getting paged with p1s out). But, does it really matter? I read people reacting strongly to these outages, suggesting that due dilligence wasn't done to use a 3rd party for this or that. Or that a system engineered to reach anything less than 100% uptime is professional negligence. However from the top of my head we've had AWS outages, Gmail outages, Azure outages, DNS outages, GitHub outages, whatever else. All these hugely profitable companies are messing this stuff up constantly. Why are any of us going to do any better and why does a few hours of downtime ultimately matter? I think it's partly living somewhere where a volcano the next island over can shut down connections to the outside world for almost a week. Life doesn't have an SLA, systems should aim for reasonable uptime but at the end of the day the systems come back online at some point and we all move on. Just catch up on emails or something. I dislike the culture of demanding hyper perfection and that we should be prepared to do unhealthy shift patterns to avoid a moment of downtime in UTC - 11 or something. My view is increasingly these outages are healthy since they force us to confront the fallibility of the systems we build and accept the chaos wins out in the end, even if just for a few hours.
- NicoJuicy 5y ago> why does a few hours of downtime ultimately matter? In our case ( Azure downtime), because none of our customer systems would work. This includes people on the road, that need to do something every 5 minutes on their PDA ( sometimes 100 people simultaneous in a big city) So yes, it matters.
- UglyToad 5y agoOut of interest, obviously you can't give too much away, what would happen if the users didn't/couldn't do that? The only situation that comes to mind is delivery drivers needing to get next destinations/mark deliveries completed but I'm maybe missing others. I'm just hoping the people building the ambulance dispatch networks aren't using Azure :laughing:.
- 5y ago
- pdx6 5y agoThe Auth0 team is probably distracted by their Okta onboarding. When I was onboarding at Okta after they bought the startup I was working at, I had to support both systems to bring myself up to speed fast -- and that caused some outages from double on call.
- Jack000 5y agoAuth0's pricing has always seemed really strange - 7000 active users for free but only 1000 on the lowest paid tier ($23/month). This means if you don't care about the extra features, once you exceed 7k you need to jump up to the $228/month plan.
- keithnz 5y agoOut of interest, what are peoples experience like with self hosted identity management options? I've been evaluating keycloak recently, and it seems pretty good.
- indiv0 5y agoKeycloak is pretty good in the average case, but when you get to esoteric use-cases like multi-thousand group/role setups it breaks down, performance-wise. Stuff like that isn’t common practice though.
- ravirajx7 5y agoHey! Corrrect me if I'm wrong But It seems using Azure's(or any third party) client credential flow is better (or say easier) option as it can be used for managing multiple microservices. However, I came across this specific need of implementing both Authorization and resource server on the same application and for that I'm planning to implement Authorization Server using Spring but I came to know that Spring have stopped active oauth project development and so I'm planning to use Keycloak for my application also I'm planning to store client id & client secret in mysql database. In authorization server I have to generate access token and then send it back to the client and verify when the api call is made with the same token. If you don't mind do you have any link or specific resources for the development which you did? I would love to see your project as well. Thanks.
- keithnz 5y agoI looked at azure a while back, funny thing is, like this incident, azure had an outage, I found keycloak pretty simple, you run it, you get a web front end, configure the bits, connect your app. I don't really have any resources at the moment, but I am going to do a github repo of example projects for connecting it to .NET stuff
- advaitruia 5y agoSince you're evaluating Keycloak, I'd recommend checking out Ory and SuperTokens.io as well (I am the cofounder of the latter)
- aleyan 5y agoIs it worthwhile to do authentication via SaaS instead of a local library? For password use case, it seems nice that you don't have to store client secrets (eg encrypted salted passwords) on your own infra. However now instead of authentication happening between your own servers and the users browser, there is an additional hop to the SaaS and now you need to learn about JWT etc. At my previous company, moving a Django monolith to do authentication via auth0 was a multi month project and a multi thousand line increase in code/complexity. And we weren't storing passwords to begin with because we were using onetime login emails links. Maybe SaaS platforms are worth it for social login? I haven't tried that, but I am not convinced that auth0 or some one else can help me connect with facebook/twitter/google better than a library can.
- TameAntelope 5y agoIt's terrifying to store credentials. I'll take 4 hours of downtime once in a blue moon over lost nights of sleep over potential security breaches. I just can't even imagine why you would these days, there are even "local" options that act as "local 3rd party auth providers".
- rozenmd 5y ago100% - for OnlineOrNot (https://onlineornot.com https://onlineornot.com) I only use passwordless auth (enter your email, get a magic link emailed) and Google via OAuth for this reason. Screw losing sleep over whether you're storing credentials correctly.
- 1cvmask 5y agoWhat happens when the emails fail (like spam folder)? I remeber a thread here on HN on a number of projects where they dumped email link sending as a login method for various reasons and complications. Have you face any challenges as well? If not what's your secret sauce? A better email provider? Would love to know.
- rozenmd 5y ago
- deleted 5y ago[deleted]
- coopreme 5y agoHow does Auth0 compare to keycloak? Is it similar?
- lawwantsin17 5y agoStop using PaaS crap like Auth0. It's a bullshit mess.
- trog 5y agoMy first Auth0 experience was a couple weeks ago when I had a quick crack at testing it out to see if it would be a suitable candidate to migrate a bunch of WordPress sites (currently all with their own separate, individual user accounts) onto. I didn't spend a lot of time on it but initially figured it would be easy because they had what seemed to be a well-written and comprehensive blog post[1] on the topic, as well as a native plugin. But I found a few small discrepancies with the blog post and the current state of the plugin (perhaps not too surprising; the blog post is 2 years old now and no doubt the plugin has gone through several updates). I found the auth0 control panel overwhelming at a glance and didn't want to spend the time to figure it all out - basically laziness won here, but I feel like they missed an opportunity to get a customer if they'd managed to make this much more low effort. I moved on to something else (had much better luck with OneLogin out of the box!), but then got six separate emails over the next couple weeks from a sales rep asking if I had any questions. I'm sure it's a neat piece of kit in the right hands or with a little more elbow grease but I was a bit disappointed with how much effort it was to get up and running for [what I thought was] a pretty basic use case. 1. https://auth0.com/blog/wordpress-sso-with-auth0/ https://auth0.com/blog/wordpress-sso-with-auth0/