27 ms·
Heroku resetting passwords for “a portion” of their users immediately
Just received this email from Heroku. Given the timeline it seems like the breach is pretty serious.
"As part of our efforts to enhance our security and in response to an incident published on status.heroku.com, we wanted to inform you that we will begin resetting user account passwords on May 4, 2022. We recommend that you reset your user account password in advance here and follow the best practices below:
Minimum of 16 characters
Minimum complexity of 3 out of 4: Uppercase, Lowercase, Numeric, Symbol
Don't just add a letter or a 1 digit number to the existing password while changing
Passwords may not be duplicated across accounts
If you do not reset your password and your user account password is reset by Heroku on May 4, 2022, your existing password will no longer work. To log in to Heroku, you must reset your password by accessing the Heroku login page and clicking the "Forgot your password?" link . Please be aware that you may be required to reset your passwords again in the future. "
- jmbpiano 4y ago"Passwords may not be duplicated across accounts" This concerns me. How are they checking that no other account has the same password? Wouldn't that imply a strong possibility they're either hashing with the same salt across all accounts or not hashing at all? Hopefully some of the smarter folks here can tell me why I'm way off base...
- why-el 4y agomultiple salts can be stored, no? and they can still figure out the same password. Google does the same, even going back multiple passwords in the past ("this password was used previously X years ago").
- vorador 4y agoHow would you check that across accounts if you had per-user salts though?
- why-el 4y agoAh I see, that I don't know (and I misread the original, quite correctly concerned post, my apologies). I am a customer and I will follow up with them about this. If I were to guess, they mean same accounts within the same organization, in case account setup has been automated.
- vorador 4y agoYeah, that makes sense!
- why-el 4y agoThis is the response I got: > Sorry, that part of the email wasn't clear. We don't compare passwords between Heroku accounts, this was more of encouragement so customers will hopefully practice good password hygiene.
- throwawayboise 4y agoThey could in theory hash the new password with every salt on every other account and see if there is a match... but that seems unrealistic, and if they can do it, it would imply that they are not using a slow hash algorithm such as bcrypt.
- adrr 4y agoIt is easy to determine previous password. Just store the previous hashes with the salt and check it when the user changes the password. The too similar checks are easy when you require the previous password. You can compare the new and old passwords together.
- paxys 4y agoWhat does that line even mean? Duplicated across what accounts? Obviously they aren't checking that every password across Heroku is unique.
- addandsubtract 4y agoThat's how I read it. How else would the sentence make sense? Maybe they're saving a separate list of all passwords, saved with the same salt, but not associated with any account? Not sure how compatible that would be with the rest of their "security measures", but then again, their security policy is all over the place already.
- amluto 4y agoPerhaps they assume that all of their users’ passwords are pwned essentially immediately, so they can check against haveibeenpwned.com. More seriously, it ought to be possible to create a data structure that, with some degree of security, will efficiently state either that a password is probably in a set or that it’s probably not. A bloom filter can do this, for example. I would not do anything of this sort in production, since, even if it has the best possible level of cryptographic security, any such capability could be abused to precompute a dictionary containing at least a massively higher than average density of Heroku passwords. IOW, the mere fact that Heroku can tell if a password is or was in use with decent accuracy is, in and of itself, a dangerous oracle.
- avalys 4y ago“Sorry, this password is already being used by joseph.h.blow@gmail.com, please choose another.”
- genewitch 4y agoevery so often i'll find a website that has logins that will say "email address / username not found" instead of "username or password incorrect"; or, when clicking "i forgot my password" it will say it doesn't recognize the email address instead of "check your email inbox, if you have an account here, you will receive instructions via email". I don't even
- latch 4y agoYou can remove username enumeration from login and password reset, but it's much harder to remove it from signup. And this is pretty much an all or nothing, there's no point giving cryptic error messages (in the case of forgot password) to users when hackers will just use the signup form anyways. Want a good example? Hacker News: "That username is taken. Please choose another." I just tried on github, generic error message on login, explicit "email in use" error on signup. Amazon? Same thing. I'd wager that 99% of website allow username enumeration. The very few cases that don't, typically shield their signup with additional steps (captchas, mobile OTP (e.g. paypal, ...)). None of which are foolproof and all of which are a compromise not just from a security point of view but also from a usability point of view. Until someone gives me a great way to do this with signups, I'm going to stick with the much more usable forgot password error message.
- na85 4y agoCouldn't you freeze the current, compromised database and when a user tries to set their password, try to "log them in" to the frozen db?
- deleted 4y ago[deleted]
- philliphaydon 4y agoAll you do is keep a history of Password + Salt. When you change the password you hash the password against all historical salts, if one of those matches then you know the password has been re-used.
- fourstar 4y agonot that they're using it, but this is something that a zk proof would be a good solution.
- slenk 4y agoPerhaps the haveibeenpwned.com API? Otherwise that does seem odd https://haveibeenpwned.com/API/v3 https://haveibeenpwned.com/API/v3
- bibinou 4y agoI'm pretty sure they don't check. "May" now means "legal told us to ask you to PLEASE don't do this". Related, I often see people trip on the sense of MAY in RFCs (https://www.ietf.org/rfc/rfc2119.txt https://www.ietf.org/rfc/rfc2119.txt), because in everyday use it has become "passive-agressive must". edit: I'm ESL so might be reading way to much into this.
- genewitch 4y ago"may not" is kind of weird, and probably was a legal team decision. "passwords can not be reused" would imply security issues that an A+ certified fresh-faced intern would prickle at, but "should not be reused" is too mealy and "Do not feed the ducks"-ish. Also while "MAY" in RFCs per that document mean something, press releases aren't RFCs and there may be significant amounts of money riding on the exact interpretation of what those words mean. And "reading way too much into" is the bread and butter of lawyers, technologists, and mathematicians. Don't worry about it!
- deleted 4y ago[deleted]
- mgomez 4y agoJust got this email as well. They ask us to "reset" our password through a link inside the email itself which I won't do out of habit since this could be a phishing attempt for all I know. So could I just change my password on the site itself? Would that be the same thing as the "reset" that they're requesting of us or does the link inside the email set a special flag of some sort? The link is wrapped in some type of redirect so I can't really tell.
- ChrisArchitect 4y agoIt's the normal password reset page
- jarcoal 4y agoThis is turning into a complete train wreck and a case study on how not to communicate with your customers. For those of you that haven't been following, Heroku has been adding non-update updates to this security thread over the last couple of weeks, which began with the announcement that some (or maybe all) of their GitHub granted access tokens had been compromised: https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413 Now, weeks later, we're hearing that all account passwords are being reset, and for some reason if you have been using an HTTPS-style log drain that you should reset any secrets related to it as well. Heroku needs to come out and clearly state what they know about this situation, and more importantly what they don't know -- which is starting to sound like the answer is "a lot". It's not even clear they know how this all happened -- whatever door was left open might still be open. So if you've gone and rotated all of your application secrets (which you probably should do), be prepared to rotate them again when this is all over.
- heartbreak 4y agoThey sent out a notification about cycling shared dataclip slugs today too.
- jrochkind1 4y agoI found the email I received about the logdrain today to be particularly confusing. "any secrets related to it" indeeed. The specific wording in the email was... "We recommend updating and refreshing the credentials used with those log drains as soon as possible." I'm still not entirely sure if I've reset/rotated everything I need to, what is "any credentials used with"? Neither the email nor any docs it linked to was clear about exactly what they are suggesting be rotated. That message wasn't very specific, while also they're not providing the context about the breach that one could use to fill in the gaps. At this point it kinds of sounds like... everything there is was compromised?
- jarcoal 4y agoVery confusing! To clarify for others, what they mean is that if there was a secret embedded in the log drain URL, rotate it. This is often the case for HTTPS log drains. Example: https://datadog.com/logs?api_key=abc123 https://datadog.com/logs?api_key=abc123
- vr46 4y agoThis email arrived in the evening but this morning I received the hard word to switch to AWS - NOW! - instead of wasting time with Heroku. Several orders of magnitude more complex but probably the right answer.
- adrr 4y agoWe had some legacy applications running on Heroku. We're porting it over to our EKS infra over the next couple weeks. There has been no explanation how the Github keys got leaked and I can only assume the worst.
- dimgl 4y agoNowadays there are many solutions that aren't AWS. Render, Porter, Fly.io to name a few.
- ergocoder 4y agoHeroku github connection has been broken for >14 days. And heroku doesn't seem to acknowledge that at all.
- djs070 4y agoThat's not true. https://status.heroku.com https://status.heroku.com shows that the incident is ongoing, that they've disabled the GitHub integration, and don't have plans to reenable it until full assessment is complete
- strix_varius 4y agoUnless it's changed in the past hour, status.heroku.com shows a green checkbox and "No known issues at this time."
- xeromal 4y agoIt's under tools. Click on that and you'll find that they turned off github.
- bigDinosaur 4y agoYou have to actually click on the general advisory of the email from Salesforce (which isn't obviously actually useful to click, there's no arrows indicating it'll expand or anything) and then you have to scroll down to an entry which isn't signposted. Additionally, in two browsers, if I clicked to expand it, then clicked the 'back to current status' button, it errored out with the console full of errors and couldn't be clicked again, which doesn't fill me with too much confidence. What a mess. I have fond memories of Heroku (or rather, how little I had to think about anything deployed to it. Liberating.)
- ergocoder 4y agoThat didn't even mention GitHub. It was a generic message about security
- thomgo 4y agoClosed my account, I hope my password didn't get leaked
- xyst 4y agoDoes it matter if the actual password is leaked in this day and age? I use a password manager and generate 64+ character passwords for each service.
- judge2020 4y agoBiggest problem is the API keys that were potentially/allegedly visible to a third-party or leaked. Depending on how far that goes back, if you don't have cold log storage you might not be aware of which exact keys are out there.
- genewitch 4y agoin my personal opinion, 64 character passwords are overkill, and i absolutely refuse to put a password manager on a cellphone. That's just asking for trouble should i ever have my phone confiscated for any reason. I use a private matrix server to send passwords from my laptop/desktop password managers to myself in E2EE channels, then delete the message. On occasion, if a password is only 18 or 20 characters (like an admin account or a single user service account) i'll type it by hand. And i use all lowercase "easy to say" passwords whenever possible. I do not believe (and never have) that adding an ampersand and a 7 to a password makes it more secure than adding "rb" to the end of a password.
- ChrisArchitect 4y ago(1) pretty crazy that that same incident is still ongoing/unresolved. (2) just saw that the password reset will invalidate all API tokens. Arrrrggh. Annoying!
- teaearlgraycold 4y agoHeroku competitors are getting endless Christmas.
- ryanSrich 4y agoIf only any of them were even slightly competitive.
- shaunrussell 4y agofly.io feature set and dx is great
- danjac 4y agofly.io looks OK feature-wise, but requires a credit card to do anything and there's no indication of how you can accidently exceed the free tier. That's a dealbreaker for me - I want to evaluate some PAAS offerings for a company, but it's a hassle to get authorization to use a company credit card, especially when I just want to kick the tyres. Heroku, Render, Railway and most others let you at least run a toy app without needing a credit card, fly.io is doing themselves a disservice here.
- aaaaaaaaata 4y agoOr they're separating themselves with their value proposition.
- danjac 4y agoMaybe, but still it means I'll just pass on fly.io and go with render. Their feature set is not that compelling that I'll want to jump through hoops to get company authorization or have to deal with expenses to use my own card.
- dimgl 4y agoHow are they not competitive?
- ezekg 4y agoI didn’t use the GitHub integration and still received this email. That’s concerning, and to me, indicates a larger breach than they’ve communicated thus far.
- martinald 4y agoYes, same here. Looking like a really big breach is my guess given the strange wording and downplaying of the issue, combined with what we actually know (Heroku source code accessed; Github tokens leaked; some other tokens leaked and now mandatory password resets).
- 27182818284 4y agoJust adding a "me too" to this thread. I just saw in my inbox my warning. I haven't used Heroku in roughly a decade. My last activity was circa early 2014, but mostly 2013.
- collectedparts 4y agoI have 3 different Heroku accounts of different vintages, one of which I haven't logged into in ages. Sure enough, each got the notice. So yeah, ready to call BS on this being "a portion" of users.
- tomglynch 4y agoI didn't get the notice - heroku account from August 2019 that I haven't used since then.
- lazyresearcher 4y agoMe neither. I never actually published anything, so that might be the reason.
- tomglynch 4y agoUPDATE: Got the email 2 hours ago
- jerpint 4y agoI'm relatively new to Heroku (about 2 months) and haven't received a notification
- lalopalota 4y agoI have 2 Heroku accounts. 1 received this email. The other has not.
- MobileVet 4y agoIf we haven’t gotten an email, does that mean our account wasn’t part of the leak? Was this a system wide event? We have been on Heroku for ~5 years and the last 18 months have been far less reliable. Sad.
- Ancient 4y ago
- alephnan 4y agoIt feels a by developers for developers platform should have be competent with these matters. Hobbyists and Developers who are looking such a platform should expect they are least technical to solving harder software problems then they are, and that’s worth paying the premium for. May the 4th be with you.
- ab-dm 4y agoI was wondering if that email was going to make it to HN. This whole things has been an absolute s*tshow. Now I'm worried that there is a much bigger issue at play, and we might actually be in a bit of trouble.
- ryanSrich 4y agoThis has gone from bad to worse very quickly. The initial issue was supposedly a breach of customer repositories. Which sounds bad, but if you’re not storing credentials in code, then the worst case is that a potential hacker had access to download your code. Not great, but certainly not as catastrophic as some breaches have been. Since then, Heroku has been acting beyond strange. Everyday they update the incident with essentially the same non-update, but written differently, with vague references to the same information they’ve sent 14 days in a row. Now, they send another very ominous and strange update about “some” customers having their passwords reset. However, based on this thread and my own experience, it seems like every customer is getting this message. What does this have to do with the initial issue? Were actual Heroku accounts compromised? This behavior is either extreme incompetence wrt customer communication, or they’re preparing to announce a truly insane breach that may include everyone that has ever used Heroku. They need to get their shit together and quickly.
- thematrixturtle 4y ago+1. If you need another data point, the entirety of my Heroku presence is a single free plan toy app that has literally been untouched for many years at this point, and I got the password reset notification.
- radlad 4y agoJust FWIW... I have an account I created in 2013 (based on a registration email.) It appears in my 1Password, imported from LastPass, so I probably logged in around/after 2016. No later than Nov 2018. I don't have the password reset email, and logging in just asks me to add MFA, and then to accept 2020 terms.
- cmg 4y agoFor another data point, I got an email yesterday from "Salesforce" about a Heroku dataclip. I had recently deleted it, so maybe the email was queued up a couple days before. The email says, in part, "We recommend that you update and refresh the unique URL used with dataclips using Shareable Links as soon as possible."
- 4y ago
- dml2135 4y agoThe fact that I haven't heard more about this Heroku breach makes me think that maybe fewer startups than I thought are actually using it. I'm at a startup that's been on Heroku since its inception and this breach has got my team thinking about moving to AWS for the first time, even if our scale doesn't necessarily demand it yet. Does anyone that has done this transition in the past have any advice?
- ab-dm 4y agoWe're also starting to look into this. We're already spending more than we need to, but depending on what this breach turns out to be, we very well may need to move to AWS to keep our enterprise customers happy.
- multiplegeorges 4y agoYeah, same here. I have accelerated my plans to move to either GCP or AWS. I am evaluating both and plan to move to Kubernetes... even though I'd been avoiding it as long as possible.
- redleather 4y agoCheck out Render
- multiplegeorges 4y agoI'd already be on either fly.io or Render, but my company has a hard requirement for SOC2 and ISO27001 certification. Heroku has it, Render is working on it, but not there yet.
- 0des 4y agoThere was a time when everyone I knew had a rails app on heroku, and I mean everyone. Now it's just one guy left. I don't know if this translates outside of my small social circle, but they all seemed to fade hand in hand, rails, and heroku in my local group.
- jamil7 4y ago
- onphonenow 4y agoMinimum of 16 characters?? Do they not have any fraud screening or rate limiting on these things, 2FA on unknown devices or something? I've had my same password forever (I'm old) on google without issue) that is scary short. However, when I login from a new device or every 30 days or go to an admin / security page it asks me to stick my yubico key into my computer. Works fine. Is this same password used for high volume API stuff without anything else? Normally that's with an access token type system (at least on AWS). Those, yes, should be highly complex. The AWS access / secret key combo is ridiculous frankly in length (mixed case + numbers + symbols) * 40.
- smt88 4y agoMinimum of 16 characters is great, especially for a service with technical users. "Scary short" and "same password" are terrifying to read. Get a password manager (I recommend 1Password) and use it. My dad is 78 and uses a password manager -- so can you. Regardless, stop reusing your password please! This is the most basic security measure possible.
- onphonenow 4y agoScary short is fine. 6 characters (case sensitive + 10 digits alone) = plenty of complexity if you are doing ANY type of intrusion detection. Coupled with a Yubico and I feel pretty darn secure. I don't re-use the password, because it doesn't rotate I can also remember it.
- stevenicr 4y agomay not be related - and def a tiny little data point - I sent heroku an email about a week or bit more ago - complaining that I had gotten a very well crafted phishing email - which was seriously very clever in it's timing (or just blind lucky for them, as I was changing servers and dns and things within 48 hours of this) - that made it look I needed to login and do something with server credentials or something similar.. and the link to click to fill info went to a heroku app with a name attached to it.. So I emailed them and asked them to look into it to see if it was overly obvious that user XY was getting clicks and if they had set up some sort of official looking form to phish server accounts or something like that - memory a bit foggy in the midst of so many changes past couple weeks. Never heard back from them, but maybe my email helped them realize some things were not being used as they should of been authorized to be used for. Maybe not.
- Mandatum 4y agoAccount since 2021. No notice. Transparent as a brick wall, in true Salesforce fashion.
- Mandatum 4y agoUPDATE: Just received a notice. Only spun up a single dyno to see what it was like for about 10 minutes in 2021. Never logged in again. No integration with GH or anythig. Assume they've been pwned end-to-end based on this alert.
- eastern 4y agoThe password reset mail is yet another train wreck. I received this email. The reset password link in it is NOT https. If I manually change the http to https it turns out that the server, click.msg.salesforce.com, is returning a certificate that is only valid for click.s10.exacttarget.com
- 0des 4y agoI assumed that was a phishing email and didn't proceed.
- sjhuang26 4y agoIf it isn't a phishing attempt it would be really surprising to me that they would use http. BTW, https://exacttarget.com https://exacttarget.com redirects to an official Salesforce product page, but that doesn't guarantee it to not be phishing.
- infamouscow 4y ago> If it isn't a phishing attempt it would be really surprising to me that they would use http. Why? Technical incompetence is a systematic problem that comes from the top down. This is what happens when mediocrity becomes acceptable and takes priority over correctness. I would suggest moving all of your business from Heroku ASAP. These kind of mistakes strongly suggest nobody capable of intelligent thought makes decisions at Heroku, nor has for awhile.
- memorable 4y agoThe website seems to be down at least for me.
- zeepzeep 4y ago> If it isn't a phishing attempt it would be really surprising to me that they would use http Spammers are the first to adopt https and all the mail and dns security stuff. It never was a sign of legitimacy
- 4y ago
- janejeon 4y agoOh wow, I'm definitely alarmed by this whole situation. I've went ahead and just deleted the whole account - it's at the point where I don't want to do business with this company, full stop.
- antoniuschan99 4y agoIn the email Heroku also says any API access tokens will be invalidated upon password reset so those have to be updated too! Not sure how much effort that requires but it doesn’t seem like it’s as easy as just changing a and having to think of a 16 minimum character password ：$ NOTE: A password reset will also invalidate your API access tokens. As a result, any automations you’ve built to integrate with the Heroku Platform API that use these tokens may result in 403 forbidden errors . To avoid downtime you will need to re-enable direct authorizations by following the instructions here and update your integrations to use your newly generated token.
- urthor 4y agoWithout reference to Heroku's current incompetence. As a general thing, I like the idea of a "swift, well written" email to your entire userbase saying "please reset your password, now." That seems like a perfectly reasonably response, and request of your user-base.
- tialaramex 4y agoAgain, stop it with the passwords. Because passwords are shared secrets, the relying party (here Heroku) can lose them and then will predictably act like somehow this is your problem when it happens, see also "Identity theft" aka "We're incompetent and our money got stolen but it's convenient to pretend that's actually your money and ask when you'll fix it". Use WebAuthn. If (when) outfits like Heroku incompetently lose the credentials nothing interesting happens.
- 8organicbits 4y agoI'm wondering at what point we'll see more sites transition to server generated passwords. At login time, the UX of a server generated password is about the same for anyone who saves their password in a password manager, i.e. it autocompletes. But at password generation time, it's much easier. You don't need to tell the user all these rules about length, complexity, reuse, etc. The server generates the password in a way that's sufficient, and the user just saves it. It also avoids the mess of password manager generated passwords not meeting some criteria, or passwords meeting client side checks but failing server side checks. When you've got so many sites asking for passwords >=10, >=16 characters in length, is anyone actually able to memorize a truely unique password per-site any more? Besides, a server generated password has known entropy, so you can model the security much more accurately. This push for really long passwords is to guard against low entropy passwords that users create, although users seem to respond predictably: [password] too short? I'll try [password]123 or [password][password].
- sofixa 4y agoThat's all fine until there's a bug or attack and now your server generated password is logged somewhere.
- 8organicbits 4y agoAgreed, you don't want to log passwords. But that's not unique to server generated passwords, you also need to avoid logging user generated passwords as well. I'd recommend storing server generated passwords in the same way a user generated password should be stored on the backend: bcrypt or scrypt.
- deleted 4y ago[deleted]
- YATA1 4y ago
- NoXero 4y agoWell, this is just a big turn-off. GitLab did this to me recently, apparently for security purposes. However, someone changing my password for me is a mega breach in my security. GitLab lost my business because of this "security feature" or whatever. GitHub if you do this, I will self-host my code, and stop using your service. Other companies who market to Software Engineers, and security folk, should take heed.
- k4ch0w 4y agoI do want to say there are legitimate reasons to have your users have a forced reset. For example, you’re upgrading your encryption, you migrated to a new IAM system, you’re handling of UTF8 could have been wrong or your meeting a new compliance standard that requires stricter passwords. I don’t know what happened with GitLab, but a telltale sign it’s bad is if they do it for you and not on your next login.
- Aabhii 4y ago[dead]
- dz0ny 4y agoThey are now also resetting integrations. I got notice that I need to re-add Slack integration.
- User123987 4y ago"Passwords may not be duplicated across accounts" hmm, how do they know if those passwords are same across accounts? Either they save them in plaintext or they can't check if I have same password as some other account.
- deleted 4y ago[deleted]
- tosh 4y agoEarlier today I thought: ok, apparently I am not part of the "portion of users" but fast forward to a few hours later and I also got the email.
- bpeebles 4y agoNew update at https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413 as of MAY 5, 2022 02:21 UTC the core paragraphs are > On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code. ... > Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.