7 ms·
Gitlab password reset bug leaves more than 5.3K servers up for grabs
- tptacek 3y agoI don't know the history of this bug but just want to chime in with a word about how absolutely terrifying the "associate email address with account" feature in account-based web apps is. Which, I guess that's my word: terrifying, one of the things pentesters make a beeline to mess with, with a vulnerability history stretching all the way back to the early 2000s when these features were often implemented on standard Unix MTAs that could be tricked into sending password resets to multiple addresses at once, an attack featureful web frameworks seem to have resurrected in Gitlab. If you're a normal HN reader that found themselves interested in this story, go check your password reset feature, specifically the email association logic! Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid.
- j-bos 3y agoHow does this exploit work? Or do you have a link to a rundown of it?
- tptacek 3y agoI'm guessing here, because I only read a high-level description of it, but I think it's a password reset flow endpoint that takes the email address to look up and send a reset to, and the framework will accept an array instead of a simple string; the endpoint looks up the first address, but the variable used to determine who to send the reset mail to is the array. Again: just a guess as to the underlying bug (I've seen that specific bug before is why I guessed).
- nightpool 3y agoThis is correct: The vulnerability lies in the management of emails when resetting passwords. An attacker can provide 2 emails and the reset code will be sent to both. It is therefore possible to provide the e-mail address of the target account as well as that of the attacker, and to reset the administrator password. Here's an example payload: user[email][]=my.target@example.com&user[email][]=hacker@evil.com (per this POC github: https://github.com/Vozec/CVE-2023-7028 https://github.com/Vozec/CVE-2023-7028)
- tptacek 3y agoYep, so it's the classic Rails bug pattern.
- chatmasta 3y agoThis is such a rarely used feature that I wonder if it would be helpful to have a CSP or preflight header that restricts the browser from sending multiple values for the same parameter.
- janpieterz 3y agoIf you'd want to configure this you'd probably want it as some kind of configuration in the backend that disables it.
- 5- 3y agoa browser is typically not used to exploit this class of bugs.
- chatmasta 3y agoThere is precedent for deputizing the browser to stop this kind of bug with Access-Control-Allow-Headers. If the backend wants to default to ignoring multiple GET/POST parameters with the same name, then the browser could helpfully fail to make a request that attempts to send them.
- talkin 3y agoThe attacker doesnt’t use a compliant browser to make the request. User agent protections only help in situations where a regular user (or their software) is being tricked
- tptacek 3y agoIt's pretty commonly used in Rails form handling logic though, isn't it?
- tzs 3y agoI think I'd count that as a bug and a design error. The bug is accepting an array when it should only take a scalar. The design error is that the endpoint should not be taking email addresses at all. It should take account IDs. Even if a system uses email addresses as account IDs they are conceptually not the same and the code should not muddle them. Keep them separate and then even if you get an "allows an array where it should have been a scalar" bug the result should be either just the first account in the array gets a reset email or all the accounts in the array that are existing accounts get reset emails for their accounts.
- tptacek 3y agoUsers hate this, because it requires them to do two email round trips (the first to recover the account ID).
- tzs 3y agoI think it is OK to call the account ID "Email Address" on everything the user sees, and make it the same string as the user's email address. As far as the user is concerned they login with email address and password. I'm just saying that in the software and in the database store account ID and email separately. Treat the fact that the account ID column matches the email address column as just a coincidence that you do not take advantage of. I'd enforce not taking advantage of it by having employee accounts actually use an account ID that does not match their email address, such as their name, so that if we accidentally leave out a call to EmailFromAccountID(...) somewhere and try to use an account ID directly as an email address it will break employee accounts. Also, it is not clear to me that even with user visible account ID that is not the same as email address that it would take two email rounds trips. The reset page could take email address, not account ID. The reset endpoint could then look up the account ID from the email address, and initiate the reset, calling the SendEmailToAccount service with the account ID to send the email. That service would look up the email address for the account.
- tptacek 3y agoOh, sure. Storing the email address in a canonical column and using the provided address as a key helps. But I think the underlying bug is still there, because the email code will still _accept_ the user input if you feed that to it.
- Sleaker 3y agoIt doesn't sound like they have a great security team, they added the "Associate a secondary email address" feature recently. This isn't something that has always been in the software. It seems more like they were cutting corners and not properly testing through ways to exploit their own new feature when it was related to account security. On top of that it looks like they had a 9.6 CVE that allowed integrations to perform commands as other users... From the outside it looks like they are trying to ship features faster than they can keep them safe and tested. Perhaps because they are having an incredibly difficult time monetizing well? It makes sense from business standpoint in some respects, but also the security stuff could just absolutely tank the business when the whole point of a (self)hosted git solution is essentially just account management.
- tptacek 3y agoI don't put much stock in this "9.6" stuff; CVSS is a ouija board that will say whatever people want it to say. But regardless: the best security teams in the world still see critical vulnerabilities in their software, because software is all garbage.
- EdwardDiego 3y agoI've recently stopped two design choices in external facing resources that posed significant security risks. One of which was around credentials resetting to emails that aren't stored in the API auth system itself, but rather come into Salesforce as a support case. "Don't worry, a support team member has to action the request" was meant to be reassuring, until I explained that this translated to "the only mechanism in place to prevent credentials being stolen comes with a massive social engineering vulnerability". But it's the previous choices I haven't come across yet that worry me.
- blibble 3y ago> From the outside it looks like they are trying to ship features faster than they can keep them safe and tested. this sums up their entire product every feature you could possibly imagine, somewhat working
- amatecha 3y agoYeah, every single time I get a spurious password reset email (presumably from someone trying to hijack my account), I'm worried they've somehow managed to add an unauthorized recovery email address outside of my control. It hasn't yet happened to me, but as we can see from this story, it is absolutely possible unfortunately.
- andix 3y agoI can't remember getting something like that. Maybe because I use a different e-mail address for signing up to services than for regular communication. I just had an idea, maybe using a + alias (yourname+some-alias-address@example.com, made famous by gmail) could help against attackers. Even if they find out your email they will never guess the part after the plus. If you forget it though then you can't reset your password anymore either.
- deleted 3y ago[deleted]
- starttoaster 3y ago> If you forget it though then you can't reset your password anymore either. If you struggle with memorizing your username/email, there's a near zero chance you're using a password manager, which also means there's a near zero chance you're using decent passwords for your logins, in my experience.
- andix 3y agoIt happened more than once that I didn't store a login to the password manager correctly. Either mixed up something while editing, or forgot to save it at all, or accidentally deleted an entry, and only noticed years later (long after old backups were overwritten). So nothing ever is 100%.
- hansvm 3y agoOr it could open you up to other attacks when the service coalesces all those emails to one canonical example (to uniquely identify you or whatever -- note that almost no online service recognizes the importance of caps in the email address you use as a username for example, where the underlying email provider sometimes does) but does so differently than the actual email service, allowing anyone and their dog to create an email which will collide with yours.
- lossolo 3y agoWhat a strange perspective. If not e-mail, then what would you associate it with? I've been running a site for over 20 years with a large user base. We initially used usernames, but it was disastrous. Everyone knew each other's usernames, making it easy to attempt brute force attacks on passwords, reset them, etc. Using e-mails isn't the problem, the issue is overcomplicating the login and password recovery logic, tons of abstraction everywhere, overengineering, people rushing to push code without proper checks in security sensitive parts etc. > Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid. You should examine the history of security issues on GitLab. There are critical exploits multiple times a year, requiring an urgent upgrade of your GitLab distribution. Gitlab is the worst product I've used security wise.
- SlightlyLeftPad 3y agoI tend to agree. Regarding Gitlab, there’s a bit of a dichotomy here. On one hand it’s good that they’re diligently catching and patching these things quickly and effectively notifying with transparency, that’s a great thing. On the other hand, it means Gitlab is an absolute nightmare to maintain, the process to upgrade it is not always trivial and to add to that, consistently, the day after a Gitlab upgrade a critical vulnerability is found and patched. Every product has security issues and what should worry you more, things that never see security patches or something that does?
- lossolo 3y agoGitlab upgrades, omg that was a nightmare, so many broken upgrades, there are literally thousands of issues on their issue tracker only about problems with upgrades.
- tptacek 3y agoYou took the wrong thing away from the comment. I'm not saying you shouldn't do email password resets. We do. Everybody does. I'm saying: be ultra careful with that code. Gitlab is both open source and has an on-prem product, so my guess is that you're simply hearing about more of the Gitlab bugs than you would with a comparably sized competitor.
- ndriscoll 3y ago> Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid. Based on the other comment that describes how this bug works, it is completely trivial to avoid. If you use a statically typed language, you'd have to go out of your way to create this bug, and it'd stand out like a sore thumb in code review, to the extent that if I saw that I might wonder whether my coworker is actively trying to create a backdoor.
- tptacek 3y ago"This bug is completely trivial to avoid. Just rewrite your whole product in a different language".
- ndriscoll 3y agoThe suggestion would more be to choose the correct technology from the start. Something like Scala is 20 years old, has a massive library ecosystem since it integrates with Java, and has type inference so it basically feels like a dynamically typed language, except bugs like this can't happen. It's also high performance thanks to the JVM. I don't know what it was like when Gitlab launched in 2014, but I know it was very solid in 2017 when I started using it. Or if that feels too hipster/niche, just use Java. It's not my cup of tea, but it's boring, practical, and the developer pool is infinite.
- simonw 3y agoYou understand that's not exactly actionable or helpful advice, right?
- ndriscoll 3y agoIt's not actionable for Gitlab, but it's actionable for anyone weighing what technologies they should use for future projects. Use one that doesn't open you to this kind of mistake, and ignore people who talk about "productivity" because that's going to be people arguing about their gut feelings anyway, and everyone will say their chosen stack is more productive. Something like Java is going to have the largest hiring pool, the largest ecosystem, and support from every major vendor. I don't particularly like it and think that Scala is pretty much a straight upgrade, but Java's a smart choice.
- Rapzid 3y ago> Gitlab has, as I understand it, a pretty excellent security team, which gives some sense of how hard this bug class is to avoid. I disagree. It's a bonehead mistake to send password resets out to tainted email addresses. As this was an authentication change it should have received extra scrutiny and so have been even harder to introduce. Something is wrong with their engineering culture that needs correcting.
- tptacek 3y agoEvery time we get a story about a vulnerability, we get comments about how they're indications that engineering cultures need correcting. All I'll say is, the impression one gets is that every engineering culture needs correcting. I buy it!
- Rapzid 3y agoSure, but some corrections are more urgent than others. Such as that time they deleted the prod database.
- TylerE 3y agoFrom my experience people say “engineering culture” when what they actually mean management culture. It’s not typically the engineers that are the problem.
- Rapzid 3y agoWhat's the difference. That's a statement.
- andix 3y agoE-mail password reset is a security nightmare even when implemented correctly. And the worst part: On most services you can't even disable it, the only way around is often only Enterprise SSO. On some services you can set up a phone number for SMS token instead. But I've never seen the possibility to require both. Password reset only with e-mail AND SMS token.
- admax88qqq 3y agoWhat makes it a security nightmare in your opinion?
- jchw 3y agoThere's really nothing about the e-mail system that makes it particularly good for anything secure at all. - Easy to accidentally forward confidential tokens - Validation of email sender authenticity is still piecemeal, and there are relatively frequently ways to bypass or work around validation. - Mail is not end-to-end encrypted. I hope that it's at least encrypted with TLS, but last time I actually messed with talking directly to an MX, it seemed like TLS was still limited to smart relays, and actual mail delivery always went to port 25 as usual... - The cardinality of e-mail addresses is not really defined anywhere. Gmail has plus addressing and ignores periods for the purposes of delivering an e-mail. Meanwhile, Google Workspace defaults to not ignoring the periods. Pretty sure some mailbox providers are case-sensitive and others are not. This means you can't know if an e-mail address at a given provider is unique. Best bet is to treat the entire damn address as a bag of bytes, but that opens up room for UX issues of all sorts, so it's hard to balance. - It's bad that if your e-mail address gets compromised, any account you have that doesn't have some kind of secure 2FA is literally seconds away from being compromised, too. Obviously, this is not a limitation that is limited to e-mail addresses, people wrongly treat SMS as secure, too. The thing with e-mail addresses though is that they're easier to treat as secure because it is somehow still less of a crapshoot than cell carrier security, and also because literally the majority of online accounts can be stolen with just access to the user's email address, and it can be done quickly and easily, and in many cases it can be hard to convince support that you are the correct owner afterwards.
- 3y ago
- throwaway63467 3y agoHonestly I would never put any kind of internal server on the public Internet, make it reachable only via VPN to have a second line of defense.
- andix 3y agoEspecially something like Gitlab might benefit a lot from external integrations that need to call the Gitlab API. It might be possible to whitelist exactly those requests, but also a bit cumbersome.
- dijit 3y agoGitlab is my favourite for running a codeforge: git.drk.sc For a highly secure environment, I agree to practice even more defensible tactics, but I think software out to be designed to survive the open web.
- pritambarhate 3y agoyes, especially if you are using self hosted gitlab for your company, you should always put it behind corporate VPN.
- alexpls 3y agoFor folks who wanna see what led to this exploit in a Rails codebase, here’s the commit where it’s fixed: https://gitlab.com/gitlab-org/gitlab/-/commit/c571840ba2f0e91ca7ec3c436f796532dbb3c550 https://gitlab.com/gitlab-org/gitlab/-/commit/c571840ba2f0e9...
- sethammons 3y agoas a non-rubiest, can you point to the error?
- Rapzid 3y agoRuby. I kid, but also I don't. Initially a single email could be passed into the API/form call and they would look it up. If found they would send a recovery to that email but it was the email the user supplied not what was in the DB. Oh, no problem we looked it up so they are the same! But then the ability to look up accounts from a list of emails was added. If any email matches the account lookup would succeed. Then they sent the reset link to that same user supplied value but OH NOEHS IT'S AN ARRAY NOW AND SOME MIGHT NOT HAVE MATCHED ACCOUNT EMAILS! So they ended up sending out reset links to a tainted list of emails. Rails "concerns" are the worst IMHO anyway, but looks like they aren't using strong params here either which is even worse. Also someone thought it was more elegant to reuse the tainted value which is par for the RoR course.
- szundi 3y agoOmg is this fix had to be this bloated?
- thedanbob 3y agoThis is actually a follow-up refactor, the fix is here: https://gitlab.com/gitlab-org/gitlab/-/commit/abe79e4ec437988cf16534a9dbba81b98a2e7f18 https://gitlab.com/gitlab-org/gitlab/-/commit/abe79e4ec43798...
- taspeotis 3y ago# Concern that overrides the Devise methods # to send reset password instructions to any verified user email module RecoverableByAnyEmail So it was a feature?? Anyway, in the fixed version it's still called RecoverableByAnyEmail. Do people not read the code around what they are changing??
- bluedino 3y agoReminds me of a bug where you could brute force an account by putting an array of passwords in the login form. It was some junky web interface to a spam appliance of all things, I'm not sure if it was intentional, or just some php rookie wrote the code. One of our users discovered it when they had a (rare at the time) special character in their password.
- PufPufPuf 3y agoRuby on Rails accepts arrays as parameters to the ORM's ".where(...)", which means "OR" between the array values. So if the code does something like "User.where(name: name, password: password)", I could totally see this happening.
- dijit 3y agoI got hit by this, we also noticed it being used with a second “feature” that exposed us a bit more than it should have. Basically a requirement for this attack is to know the email of the user you want to reset, but, there is a hidden email address that is tied to your gitlab userid (a number incrementing from 1). Since its a safe bet that ID 1 or 2 is an admin: thats a good target. the email is something like 1-user@mail.noreply.<gitlabhost>. Really bad, seemed like it was automated. 2FA saved us here.
- xyst 3y ago[flagged]
- mooreds 3y agoWhat is your alternative? I don't love passwords. I think that magic links are a decent alternative, as are passkeys (for web) or similar (for non-web). SSO is fine, but just centralizes the problem (like magic links do). I just see adoption of any of these taking a long time.
- solatic 3y agoSSO centralizes the problem to somewhere that can require hardware-based auth, e.g. passkey, Yubikey, etc. Passwords are fine if they're combined with hardware-based auth. Expecting every vendor to implement passkeys etc. natively isn't going to happen.
- frenchman99 3y agoGood reminder to always run internal services such as Gitlab behind a VPN which only trusted users have access to.
- kdtsh 3y agoI really don’t understand why anyone would have their internal VC and CI/CD on the public Internet. This is exactly what VPNs are for.
- INTPenis 3y agoYes that's what saved us, and a few other things. I work for a huge government owned telco and our networking guys are the best. They keep us server guys in line. So even though they did expose our Gitlab to an extent, for certain external projects and consultants, you still can't visit it from the internet freely. And also we manage users in AD so there is no SMTP connection to even do password resets. But we really need to enforce more 2FA, we've left it up to each project to enforce their own rules on 2FA.
- TobyTheDog123 3y agoCan we please stop pretending that Ruby/Rails is in any way a good choice for software that needs to be safe? I do understand that it is what it is and GitLab has to deal with it, but going forward, can we stop pretending a language and framework that prioritizes cleverness and hidden control flow is better than something more boring? If I sound overly-annoyed it's because I have to work on a production Ruby codebase where I can absolutely see a scenario in which we have similar issues just waiting to be exploited, because someone thought seventeen layers of abstraction made the code super extensible.
- dpkirchner 3y agoAny language or framework that lets the caller specify if a parameter may be a string or an array of strings should probably be avoided, IMO. The cost of this one error likely outweighs the total value realized by use of the feature.
- Helmut10001 3y agoSorry, it is really easy to automate Gitlab updates. Just one option is to use Gitlab in Docker+Compose, which works rock-solid, and have Watchtower (e.g.) do the updates daily. I have two Gitlab servers that do this since 7+ years and no issue whatsoever. Looking around, I see so many outdated Gitlabs - what are the administrators doing?
- glub103011 3y ago[dead]
- Too 3y agoAnother reminder to alway use SSO and to always use 2FA.