9 ms·
Duo Outage
- billbrown 3y agoThey published the postmortem, and the penetration into colleges is hinted at as being causal. > Increased load on DUO1 due to significantly increased adoption and simultaneous peak usage across multiple larger customers led to authentication failures. https://status.duo.com/incidents/rw7g0q7ztj8f https://status.duo.com/incidents/rw7g0q7ztj8f
- f1refly 3y agoI bet both of duos users will be bummed to hear about this outage.
- Rebelgecko 3y agoIt's mandatory 2FA at the college I went to (where, coincidentally, classes are starting today)
- harrego 3y agoCollege-induced software!
- SoftTalker 3y agoIt's almost like all the college and university CIOs are choosing the same software platforms. Now why might that be?
- sbdaman 3y agoI work in post-sec and this is very common practice. There are few key players that tend to capture the majority of schools in the States/Canada for specific tech solutions. Blackboard/Canvas/D2L for LSMs, Shibboleth for SSO, Duo for 2FA, Cisco AnyConnect for VPNs. tech solutions in the field tend to be incredibly low risk given the size and make-up of the anticipated users (enterprise services with thousands of employees and tens of thousands of students). For public institutions, there's the added element of public sector risk avoidance.
- reaperman 3y agoTo be clear, Shibboleth is often self-hosted and usually the grey-beards understand how to maintain it. It's been around a long time and is very stable/robust and at least as unlikely to fail as Duo/Cisco (which are overall fairly robust with rare enough breaking failures). OTOH, rolling their own 2FA would likely create points of failure that rear their ugly head more often, not less often. Shibboleth is kind of an outlier here, due to its age/maturity and position as a very old-school piece of foundational tech that got implemented when academic IT salaries were quite a bit easier to live on than they are today. The disparity between tech salaries in academic institutions and FAANG/SaaS corps has grown immensely in the past 20 years. Most of the people who do the real work at academic institutions have been employed there for 25-40 years. Most of the young people can't stick around for long because they need to earn more money to build a stable life.
- sbdaman 3y agoI wasn't criticizing any of the companies I listed.
- harrego 3y agoMakes sense, but I still loathe the clunky stuff I end up using!
- mynameisvlad 3y agoYou make it sound like it’s some big conspiracy but the boring answer is that nobody gets fired for choosing an industry leader. Duo has positioned itself as an industry leader in MFA and is one of the safe bets when implementing that feature. This isn’t exclusive to schools, either.
- MattSteelblade 3y agoThis is Cisco's Duo used for MFA and SSO and not Google's discontinued video app.
- davewritescode 3y agoYep, and my company uses it as part of access to all of our internal applications including our AWS console. Not a super productive morning
- Gabrys1 3y agoI didn't realize this Duo was affiliated with Cisco. Now it makes sense it's so popular
- commandlinefan 3y agoIt's impacting me, but I was a little surprised to find out that Duo was "popular" enough for this outage to make it the front page of HN.
- davewritescode 3y agoI posted it and honestly I am too
- afavour 3y agoNever underestimate the number of college students hiding in the online shadows
- spike021 3y agoI've used it while working at two major companies everyone's heard of and used. Definitely seems to be popular.
- pavel_lishin 3y agoI thought this was one of Google's messaging apps.
- jolux 3y agoIt was a video call service: https://en.wikipedia.org/wiki/Google_Duo https://en.wikipedia.org/wiki/Google_Duo
- miked85 3y agoWow, never heard of this one - how many messaging/chat applications has Google created now?
- burkaman 3y ago~17, depending on how you count: https://www.theverge.com/2021/6/21/22538240/google-chat-allo-hangouts-talk-messaging-mess-timeline https://www.theverge.com/2021/6/21/22538240/google-chat-allo...
- silisili 3y agoDon't forget Google Photos...which also has chat for no discernable reason.
- gsa 3y agoIs it really discontinued yet? If anything it's an ongoing naming blunder with no end in sight. There are two products called Google Meet now. The web interface for the former Google Duo is duo.google.com, just rebranded to Google Meet. The former Google Meet still exists on meet.google.com. Both also have their own Android apps - one called Meet and other Meet (original). Both products have a different set of features and neither completely replaces the other.
- jolux 3y agoduo.google.com redirects to meet.google.com on my iPhone.
- stefanpie 3y agoThis is also an unfortunate coincidence, as today is Georgia Tech's first day of classes, and the institution's Single Sign-On (SSO) for almost all its services, such as the Canvas LMS, registration, and the Bursar, is based on Duo's service. Right now it seems to be having some issues and is timing out / not logging in. However, I'm not sure if GT's Duo service is self-hosted or is hooked into Duo's service "in the cloud".
- MattPerry 3y agoI am reading this, slacking off, because I couldn't get into Georgia Tech's Canvas. Judging by the status.gatech.edu message, it seems like changing the instance of Duo that responses is a quick fix.
- afavour 3y agoI wonder if it’s more causation than coincidence.
- steve1977 3y agoSingle Sign-On, single point of failure… maybe having lots of different logins wasn’t such a bad idea after all.
- kube-system 3y agoThere are a lot more other problems that happen when each service manages their own authentication. The move to SSO has been in response to problems that existed then.
- adbachman 3y agoSame for my kid picking up info for the beginning of their first year at Univ of Maryland. Millions of students coming online, textbooks are digital now, SSO all the things. Perfect storm.
- PAPPPmAc 3y agoUniversity of Kentucky too, first day of classes, no one can get into anything they aren't already logged in to with a valid cookie. I spent 20 minutes trying to figure out what new cookie I needed to grey-list for the half dozen redirections in the M365 auth flow to not bork before I thought to check if it was generally broken. Great success.
- harrego 3y agoI was always curious why Duo cloud 2FA was chosen over traditional offline 2FA codes in education.
- ihattendorf 3y agoUser experience is nicer: click confirm prompt from lockscreen vs. having to find and open app, select correct site, and enter code.
- lolinder 3y ago> click confirm prompt from lockscreen This is also a huge vulnerability that has been exploited. https://www.theregister.com/2022/11/03/mfa_fatigue_enterprise_threat/ https://www.theregister.com/2022/11/03/mfa_fatigue_enterpris...
- harrego 3y agoFrom the student side I end up carrying an extra 2FA app that only works with my college vs. Google Auth (and similar) that carry my other 15 sites.
- SkyPuncher 3y agoIt's simply so much easier. Okta offers a similar feature. So much easier to click a confirmation on my phone than to scroll through dozens of 2FA codes (some of which might be orphaned).
- cute_boi 3y agowell apps like 1password makes it easy to fill those 2fa codes from computer and phones easily.
- janejeon 3y agoAbsolutely. For me, I don't think I would use anything other than 1PW/Yubikey for 2FA codes. But not everyone uses such technologies, and a certain percentage of population is going to find the hurdle to adopt these technologies/apps too high. So, not for us, but I understand why they do it.
- dmattia 3y agoI love that the status page is being updated regularly. But I have no idea what the difference is between DUO1, DUO2, etc. through DUO73. I feel like they should have a better way to clarify which users are affected.
- electroly 3y agoThey're all identical deployments of their whole stack. They shard customers onto the deployments to reduce blast radius. In your Duo admin dashboard, look for "Deployment ID" in the left pane.
- sriske 3y agoFWIW I just logged into my university portal and Duo worked just fine. Perhaps it's not widespread?
- btbuildem 3y agoWe use Duo at work for 2fa. I am laughing my ass off, I literally can't do anything -- can't get on the VPN, can't get into my emails, can't access company services. They locked everything down so hard, they've literally chained themselves to a radiator and tossed the key out of reach. We didn't receive internal communications about the outage yet -- and my bet is it's because whoever's in charge of that, is locked out due to the outage.
- merb 3y agoI would never do duo 2fa only. It sucks if one of your factors does not work. Most of the time I always try to use two 2fa apps.
- Spooky23 3y agoTOTP tokens don’t meet alot of compliance requirements. You usually need a PIN or crypto device with a dedicated solution. Like the popular password vault says in the name… 1Password for everything. If everything can stand alone in 1Password, it ain’t MFA.
- gchallen 3y agoAnd the proud Illinois tradition of some mission-critical service crashing on the first day of class continues. In this case, it is an external service. However, I also suspect that the Duo outage is probably shielding other on-campus services from load surges that would probably be causing them to get crashy. I guess I don't know how we could ever prevent such incidents. Given that the first day of classes is a well-kept secret /s.
- senectus1 3y agono issue here in australia