14 ms·

Heroku Security Notification

424 points by peterwallhead 4y ago

bpeebles 4y ago
I had posted the most relevant paragraphs in https://news.ycombinator.com/item?id=31255450 https://news.ycombinator.com/item?id=31255450 but it probably should be here too: >On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code. ... > Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.
VWWHFSfQ 4y ago
> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else? I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.
- majormajor 4y ago
  The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
  - throwawayboise 4y ago
    Yep. Same thinking lead to the split of /etc/shadow from /etc/passwd.
  - gabereiser 4y ago
    This is also why folks advocate for a separate auth system (keycloak was just mentioned here on HN) since they are different types of information. >I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up. I feel like they are heading this route as well. Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.
    - huijzer 4y ago
      > Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine. Trust takes long to build and is easy to break. For anyone able to convert the Heroku buildpack to a Docker and able to move the database, moving away from Heroku shouldn’t be too hard. There are multiple similar services nowadays.
  - donatj 4y ago
    We moved basically everything but username into an entirely different db, went so far as to hash the username column so we don’t even know what’s it is until you log in.
    - 3np 4y ago
      > hash the username column Is this giving you any real security benefit? (I'd assume the usernames are indexed elsewhere and that it's a reasonable assumption that whoever gains access to this hashed data has access to the username list as well, making a lookup trivial - or are these not safe assumptions?)
      - couchand 4y ago
- glenngillen 4y ago
  Yeah, the comms around this has been very concerning. Do I need to rotate every config var on all of my apps? Re-install every add-on? While the nature of what limited things they had disclosed to date pointed to this situation part of me wanted to believe it wasn't as bad as I was assuming. And now the trendline on this suggests I should have already done everything I've outlined above. And I'm low confidence anybody is going to be proactive in telling me until it's absolutely obvious that these things have been compromised and exploited.
  - xena 4y ago
    IIRC the environment variable settings are encrypted in a physically separate database. However it may be a good idea to rotate your secrets anyways. My hunch would be that there are so many "juicy" targets on Heroku that you probably don't need to worry too much right now unless you are or work for a "juicy" target. This is gonna suck.
    - t_sawyer 4y ago
      I posted a similar worry about the ENV's. Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.
- sofixa 4y ago
  > I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up They can't, they surely have EU customers so have to follow GDPR disclosure rules, which they might already be afoul of.
- benevol 4y ago
  > What else was in this database? Typically the password field is stored alongside the rest of the user record. That is where you are supposed to have encrypted as many data fields of the record as possible - in addition to the conventional database encryption which encrypts the database as a whole.
- jeremymcanally 4y ago
  The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase. If that's the case...yikes.
  - jeremyjh 4y ago
    They didn’t say that happened. I’m reading it as their DB was compromised and it’s contents included GH auth tokens.
    - jeremymcanally 4y ago
      How do you read this as the database had tokens in it? > Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database... EDIT: Ah yep you're right. Two tokens in play there: one Heroku API token, one GitHub token. Phew.
onphonenow 4y ago
Ugh - I got the password reset email w silly password complexity - never a good sign.
- DethNinja 4y ago
  That email made it clear that Heroku lacks fundamental knowledge about security. I’m sure they lost some enterprise customers, I know I don’t open accounts on websites with silly password complexity requirements.
  - onphonenow 4y ago
    Google allowed 6 character passwords for a while, and didn't expire them when they increased minimum to 8 for google workspace accounts. This has been fantastic, as users can remember their password forever even if its higher complexity (google does a password strength eval). No rotations either. I'm pretty confident google will pick-up someone trying to brute force a 6 character password. That google will notice connections from new / different IPs or browsers. That's because google asks for my 2FA in various situations but doesn't annoy me by asking for 2FA all the time. I use one govt system that has something like a 14 character password requirement. For even more security if you don't log in for 90 days your account goes inactive and the password EXPIRES! Very secure you say? Well, to regain access you have to provide the answer to a security question - favorite pet! That's a 5 letter word that doesn't change (and is probably pretty guessable). Here is another example: "(b) Information systems must be designed to require passwords to be changed not less frequently than every sixty (60) days." - SBA IT Security Policy - 90 47 4
    - vel0city 4y ago
      For Workspace accounts, an admin can choose to enforce complexity requirements on next login after making changes to the complexity requirements.
      - onphonenow 4y ago
        Right, we've found actually simple passwords but with the mandatory 2FA turned on works really well. The 2FA google uses is a gentle touch in most cases (can persist on a device for 30 days). Google has nice 2FA controls. In a workspace setup you can actually tweak them to match your needs because the lockout / reset path (was) pretty reasonable (when it was onsite). Ie, we could disable certain methods and for some higher security groups you can provide hardware keys and then turn that group up a bit. Never had to rotate passwords and users are glad for that I think. I do wish google offered "Cloud Chrome" for admin staff to open email / click on links etc. Basically a remote VM with chrome but no file access directly.
productceo 4y ago
What are some good Heroku alternatives these days?
- kenrose 4y ago
  Render.com
  - henryaj 4y ago
    Stuff like this worries me - https://community.render.com/t/genuine-question-why-does-render-feel-so-unreliable/3907 https://community.render.com/t/genuine-question-why-does-ren...
    - anurag 4y ago
      (Render founder) Stuff like ^ worries me even more. What keeps me sane is our engineering team obsessing over reliability, and learning from every single incident, no matter how small. We're improving every day.
- oxff 4y ago
  I've seen fly/io and render/com mentioned in these Heroku discussions, but I have yet to test either of them.
- SkyPuncher 4y ago
  Aptible (though geared more towards Healthcare/Compliance space). ^ Note: I use to work for Aptible. Great company. Great people. Now working for one of their spin outs.
  - btoro 4y ago
    This is super cool. I think you just saved me a massive headache, as I plan to roll out a healthcare app with PHI in the next few months.
    - 4y ago
mac-chaffee 4y ago
I really feel like people don't value intrusion detection enough. Why was Heroku's intrusion detection system "rely on Github's intrusion detection system"?
- jlmorton 4y ago
  As described, the attacker quietly retrieved OAuth tokens from a single Heroku database, and then very loudly scraped GitHub. I'm not saying it would have been impossible to detect the first action, but it's substantially easier to detect the second.
oxff 4y ago
Why am I hearing about this on checks the fucking date on May 5th instead of, like, month ago.
- VWWHFSfQ 4y ago
  My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all. Who knows how long Heroku's internal systems were compromised.
  - cmeacham98 4y ago
    Even if that's the case, it's still way after April 13th.
    - chii 4y ago
      it might be that it took this amount of time to establish the facts of the events. If they recounted an incorrect version early, it might do more damage than not telling it. I dont know if the github disclosure "includes" heroku's disclosure : https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u... - but it was at least april 15th - close-ish to when the event occurred.
      - bigDinosaur 4y ago
        Is it ever true that earlier indications that credentials should be rotated leads to worse outcomes, though, as just one example? I'm sure I've received emails of the form: we suspect there may have been a breach, so we're forcing password resets, and have always taken that fine.
    - dev_tty01 4y ago
      Heroku reported it on 4/15. Read the beginning of the string of updates on the notification page posted here. Also, https://news.ycombinator.com/item?id=31048646 https://news.ycombinator.com/item?id=31048646
rapfaria 4y ago
It is weird to start the day with a Salesforce email in your inbox - had to go checking if it was valid.
koolba 4y ago
> Access to the environment was gained by leveraging a compromised token for a Heroku machine account. Any idea if this involves AWS EC2 Instance Roles? It’s incredibly convenient, but has got to be the scariest feature to enable on a platform that allows arbitrary user code to execute.
darig 4y ago
completelylegit 4y ago
Pretty good alternative to “We and our customer data got hacked”.
craigkerstiens 4y ago
"a Heroku database" was what was known as core-db internally for the longest time. I'm not sure if still the case or not today. But at one point was the source of everything for Heroku. Over time things were moved out, so this isn't an everything that exists has been leaked, but it is not a guarantee that attacker didn't move from one area to another. As someone with some apps on Heroku, having worked there, but no knowledge of the details of the incident more than others... I would: 1. Rotate all creds 2. Ensure logging all connections to the DB (I can't recall how much you can do this on Heroku) 3. Extra heavily audit Github commits and Heroku releases 4. Maybe keep rotating all creds?
- srinathkrishna 4y ago
  I feel for the team working on this at this time. I hope this doesn't end up accelerating the culling off of Heroku by Salesforce. One of the smartest and nicest bunch of folks I've worked with.
  - craigkerstiens 4y ago
    Completely agree, heart goes out to the team. No harsh judgement of all the engineering team there, not a fun situation and hope they know a lot of folks in their corner.
  - jeromegv 4y ago
    Heroku is heavily connected with Salesforce now with Heroku Connect, I doubt this is part of the plan.
    - bobx11 4y ago
      There was already a plan by salesforce to kill it and make it into salesforce functions. Someone else here called it project periwinkle.
      - traderj0e 4y ago
        That's frustrating. Heroku is one of the nicest cloud services I've ever used, and am currently using.
ngcc_hk 4y ago
Why not just say hacker ? Know we are on hacker news and hence it is semantically not exactly right. But threat actor … sound more like threaten actor. Just a movie star or drama queen. If one say Heroku was hacked or just sales force … I know one want to manage but somehow the title is not exact right. Too pr.
- tempfs 4y ago
  Threat actor is industry standard terminology for a malevolent or blackhat brand of hacker. Not all hackers are threat actors. https://en.wikipedia.org/wiki/Hacker https://en.wikipedia.org/wiki/Hacker
owlninja 4y ago
Always interesting to me with events like this that the actions or intentions of the "threat actor" are never discussed. The conversation is always finding tiny holes in the victim's systems and admonishing them for not being prepared.
- subroutine 4y ago
  Actions other than... (1) obtained access to a Heroku database, (2) downloaded customer GitHub integration OAuth tokens, (3) enumerated metadata on customer repos with the OAuth tokens, (4) downloaded some Heroku private GitHub repos containing source code, and (5) exfiltrated customer hashed and salted passwords?
woodrow 4y ago
I reset my password and then closed my Heroku account today. What a sad end for something that was once a model for software deployment and developer experience.
Mandatum 4y ago
Really, really bad form from the Salesforce Trust team here. Hopefully the Slack acquisition means better, quicker communications. On a scale of Slack to Oracle on breach notifications, this was definitely closer to Oracle.
- curuinor 4y ago
  rule of thumb: company A acquires company B, then company A does not change to become like company B, company B changes to become like company A. there are exceptions, but few. I think Slack's communications are going to become more crap, personally.
  - JorgeGT 4y ago
    Boeing's acquisition of McDonnell Douglas being a sad exception to that rule of thumb.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
lovelearning 4y ago
> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetorically or in bad faith. It's a genuine question I have based on a project I did. I researched cybersec tech like SOAR, XDR, security logging, and SIEM in depth. On paper, the marketing for such tech gives the impression that by using them, such breaches can be detected and prevented in real-time. But there seems to be a mismatch between the claims and ground realities. If so, why?
- cube00 4y ago
  > GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. The three days after being notified to actually revoke the tokens isn't ideal either. Surely if GitHub comes to you and warns you of suspected unauthorised access you'd spend a very limited amount of time and then revoke the credentials to be on the safe side.
  - yuliyp 4y ago
    I think (based on what I've read on the linked page) the notice was telling Heroku that "hey, someone used a compromised OAuth token to download your source code" not that "the tokens that you are using to read Github repositories of your users are compromised". Both are Github OAuth tokens, but playing different roles. Presumably the compromise of source code might have been used to help get access to the database that had the Github integration OAuth tokens, and realizing that might indeed have taken a couple days.
- Mandatum 4y ago
  Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding). In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "security" ends looking the same. They don't pay well, executives have pulled talks and fired speakers who do things they disagree with (the same executives are promoted and remain there - no accountability), they've got a pretty bad wrap within the industry.
abrkn 4y ago
I received an email yesterday asking me to change my password. I did, and updated our services with the regenerated Heroku API key. This morning, I was unable to log into my account and had to reset again. And update our services again.
- ryannevius 4y ago
  The email I received clearly stated the following: > Due to the nature of this issue, you may be required to reset your passwords again in the future.
peterwallhead 4y ago
Cheers to admin for changing the title, but old title better reflected the last update (and I used it because Heroku doesn't have a permalink to each new update).
tossaway2134 4y ago
This isn't the first time Salesforce Cybersecurity has left us in the lurch while they perform damage control. On 17 May 2019, Salesforce performed maintenance on their databases that clear permission sets for users. My team was able to piece together that the incident happened at about 0200 CDT, and Salesforce didn't take ANY noticeable action for at least 9 hours when they locked all customers out of the platform. Salesforce "fixed" the issue, which meant our Admins had to go in and reapply a bunch of profile settings...no big deal, right? Just a little bit of work for everyone to fix their own accounts. Salesforce acted like it wasn't a big deal. Wrong. If you were a Salesforce customer that built a tool using the Portal or Community tools Salesforce provides for external users, there was a 9 hour window when a customer could log in and instead of seeing the data you were sharing with them, they would see all data for all users. The permissions that indicated that a user should only be able to see their own data was gone. The only reason we knew about this was because we were paying extra for advanced logging. We were able to see a few of our users logged in during this time and looked at customer records they should not have had access to. Salesforce stood fast that exposing data through their Community and Portal tools this did not constitute a breach or even a violation of their SOC-2 Type II compliance. We were lucky that the only people that had access at the time were licensed partners. Nevertheless, our users lost their jobs and were stripped of their licenses. Anyone that was using those tools at the time for any sort of direct customer interaction that shared order history, customer engagement, referral programs, etc. was not so lucky; doubly so if they weren't paying for advanced logging and/or didn't know what to look for. Salesforce was more concerned about covering up their mistakes than they were about telling their customers that there was a problem. Seeing the Heroku notification page gives me PTSD. This looks all-too-familiar to me and I sympathize with those affected by this. I still feel like they were negligent back then, and I wish I knew who to tell to warn others.
- stevebmark 4y ago
  Salesforce’s monitoring, availability, incident handling, transparency, and accountability, are abysmal. We too we’re affected by the incident. I sincerely hope they lose some of their compliance certifications, because their behavior is unacceptable.
photon12 4y ago
There's going to be a question about the expected probability of this across cloud service providers. I've done security work for multiple cloud service providers and know a lot of people in the industry. I'm not really privy to give details. I can say: dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, security operations teams have a lot of data points to look at, and in theory even changes in personnel in a couple of teams can have an impact on the threat posture for a given set of a company. People are trying. But if you do the math for introduction of attack surface over time versus risk mitigation effort over time for that attack surface you can derive some estimates for likelihood of attack. If your cloud provider isn't providing you that data as a customer, it's not simple to make a determination about likelihood of risk introduction and breach of your cloud provider. This hasn't really been something people talk about because there was a tacit assumption that the biggest companies are mostly getting this right. I'm not trying to say the cloud is inherently broken, I run a lot of workloads in the cloud and trust sensitive data to the cloud. But I do wish there were better ways for customers to have data points with which they could evaluate platforms besides extrapolating the history of public breach reports.
- pm90 4y ago
  Agreed. cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open”. Its so easy to make a resource accessible to anyone, or an IP accessible from anywhere in the cloud without proper restrictions by internal teams in the company; and often the default is to give teams superadmin to “unblock their time sensitive project” rather than maintaining principle of least access which requires more discipline (and thus effort). Either cloud providers need to assume more responsibility for security or a Federal Agency like the FBI or NIST need to be more proactively engaged in improving the security posture of cloud hosted US corps.
  - hnbad 4y ago
    > cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open” So no different from every VC funded startup (or startup seeking VC funding) then? The sentiment of imposing tighter regulations around data security feels counter to the general idea that the lack of regulations around data security (e.g. strong data protection laws) are what allows the US tech industry to dominate compared to its EU equivalents. I'm not disagreeing with this, I'm just pointing out the contradiction and wondering how those who believe the latter would reconcile that belief with demanding the former.
stephenanand 4y ago
[flagged]
samwillis 4y ago
Well, it only seems to be getting worse on this one. I’m keen to get off Heroku, but waiting for one of the newer alternatives (Render/Fly+others) to implement WAL point in time restore for Postgres. It’s the only thing keeping me on Heroku now, but is indispensable. Anyone here from them have any update on when we could see that feature made available?
- ryanSrich 4y ago
  I’m also interested in this. I have 4 decent sized applications I want to migrate, but I don’t have the resources to self host on GCP or AWS. K8s is nice in theory, but I know from experience it requires too much hand holding (let alone other managed services like a database and cache). From what I’ve seen in the market, fly, render, and railway are the only real Heroku competitors. All seem to be missing a few critical pieces of functionality that is preventing me from migrating. Railway doesn’t appear to allow me to remotely run Python commands, they also don’t have automated nightly backups. Fly and Render respectively have a handful of missing features. It would be amazing if one of these services came out with 1:1 feature parity with Heroku. I’d move in a heartbeat.
  - gizzlon 4y ago
    Google Cloud Run or App Engine Standard might be what you want: https://cloud.google.com/run/docs/quickstarts https://cloud.google.com/run/docs/quickstarts https://cloud.google.com/appengine/docs/standard https://cloud.google.com/appengine/docs/standard Unfortunately some things are more difficult than necessary =/ OTOH the GCP ecosystem, network, and data-center presence are vast.
  - martinald 4y ago
    Azure web apps is pretty similar to heroku and I find it very easy to use. It has had a Linux version (instead of Windows) for quite a while now and it's pretty solid. Haven't had many real issues with it in many years of use. They also have managed postgres db with very powerful tools.
- zoomzoom 4y ago
henryaj 4y ago
Obviously Heroku have handled this horribly - but are any small startups out there considering replatforming? Still seems like a lot of hassle and the competition I've tried (Cloud66, excid3's thing) haven't been as good.
- staindk 4y ago
  I'd maybe use this as an opportunity to prioritise moving everything over to AWS. I'm sure for some it may be an unwieldy amount of work but for others (depending on tech stack etc.) it ought to be fairly doable. In the long run it'll save money too.
  - ojame 4y ago
    I’m interested in peoples experience with this and if it’s relatively true. We (like many others I assume) pay more for Heroku than AWS as it allows us to “outsource” our dev ops. We are a small team (sub 15) with a decent sized, decade old app. We’ve had it on AWS before (and used platforms like BuildKite) but both required much more overhead (in terms of employee salary). Anecdotally I’ve heard the same from friends, though I understand AWS works well and is cheaper if you know AWS well.
    - Dave3of5 4y ago
      > I’m interested in peoples experience with this and if it’s relatively true. My personal experience is that AWS is always more expensive. You don't use AWS because of the cost saving you use it because: * Top of the line h/w * Always the first to bring out new features * Very high availability of resource, like seriously I've never had a time even during the pandemic where they struggled with resource availability * Resilience of AWS systems are very high * Very good support There was an article a few months back where a teams tried moving from Heroku to AWS to save money and ended up spending nearly 3x the amount. Heroku give more resource than you actually pay for by default and it turns out they were using the extra resource during normal operation. When they done their calcs for a switch to AWS they used the quoted resource that Heroku say you get and there system died due to being under resourced. They had to up the resource which pushed them well over their budget. I'll try and find the article. Please Note: Heroku servers are on AWS already
boesboes 4y ago
Seems like they don't have much of their notifications in order either. I haven't heard _anything_ from Heroku on this, my colleague has been getting updates since this started... We are both admins of our companies account.
- Narkov 4y ago
  How does a company like Salesforce mess this up for such an extended period of time? I understand that companies can make mistakes early on in a critical incident but this has been going on for weeks?!
  - xena 4y ago
    Root cause analysis must have failed pretty bad. I worked at Heroku and with how much headcount they've lost my Salesforce starving the beast over the years, it's not surprising that it would take this long for them to react like this. When you scare away the best people with Salesforce policies, their Jira clone built on Salesforce, and overall being difficult to impossible to get more people on the team; yeah people will pack their shit up and leave. Given how much of a web of interdependent and undocumented pain heroku is, I'm not surprised it's taken this long. A security team without any context to Heroku must have had to trace through everything system by system. Especially if core-db got popped.
kaushikt 4y ago
I feel saddened to see this as a heroku side-project user since 2011. We are a small team and were hoping to migrate all our infra to Heroku in the upcoming quarter.
wiredfool 4y ago
TIL that I have a heroku account.
flas9sd 4y ago
last I've seen the Oauth permissions for the Heroku Dashboard given by Github are excessive and include write access to all public repos - as read-only is not an option if I recall correctly, see https://github.com/dear-github/dear-github/issues/113#issuecomment-906779498 https://github.com/dear-github/dear-github/issues/113#issuec... Newer integrations like Github Apps are more granular and can restrict the scope , also ssh deploy keys are an option for other purposes, but specifically the tokens issued for the Heroku Dashboard can write to the public repos of a user or org.
TobiK2020 4y ago
I'm really wondering right now how hard or easy it was to suddenly change all of their customers' passwords in the system.
t_sawyer 4y ago
I'm interested if application ENVs were stolen. If they stored Github APIs in plain text I'm sure our ENVs were too. I went ahead and reset my planetscale passwords and moved my app out of Heroku. But I only had two apps hosted there I feel for anyone with a large number of apps on there.
anyfactor 4y ago
I have seen nothing but bad news about Heroku recently. Albeit nothing provides that much ease to use on a free or affordable tier when you have minimum computing demands. I use it as my fundamental cloud provider for proof of concept stuff, so does many people. Salesforce is doing a terrible job managing this lucrative platform. I have no idea why muck up a good service like that. They have some plugins and a postgres connector but the drive to innovate, the drive to even care stops there. All these news act as a reminder that I should move my code to a "real" cloud provider.
bspear 4y ago
The comms for this security breach is a textbook case study of what NOT to do
dataspun 4y ago
It seems as though enterprise customers of Heroku that leverage data products tied to their Salesforce instances, such as Heroku Connect and Salesforce Connect, are at risk of serious data breach and possibly haven’t realized it. Nearly a month since the initial 2413 incident, we’re still learning of its full scope. If customers haven’t rotated credentials — mistakenly thinking the incident is isolated to web apps and GitHub — the risk is still present. Can it go any more sideways? Yet the communications from Heroku and Salesforce are disjointed, even amateur.

424 points by peterwallhead 4y ago

I had posted the most relevant paragraphs in https://news.ycombinator.com/item?id=31255450 https://news.ycombinator.com/item?id=31255450 but it probably should be here too: >On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code. ... > Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.

VWWHFSfQ 4y ago

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else? I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.

majormajor 4y ago
The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
- throwawayboise 4y ago
  Yep. Same thinking lead to the split of /etc/shadow from /etc/passwd.
- gabereiser 4y ago
  This is also why folks advocate for a separate auth system (keycloak was just mentioned here on HN) since they are different types of information. >I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up. I feel like they are heading this route as well. Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.
  - huijzer 4y ago
    > Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine. Trust takes long to build and is easy to break. For anyone able to convert the Heroku buildpack to a Docker and able to move the database, moving away from Heroku shouldn’t be too hard. There are multiple similar services nowadays.
- donatj 4y ago
  We moved basically everything but username into an entirely different db, went so far as to hash the username column so we don’t even know what’s it is until you log in.
  - 3np 4y ago
    > hash the username column Is this giving you any real security benefit? (I'd assume the usernames are indexed elsewhere and that it's a reasonable assumption that whoever gains access to this hashed data has access to the username list as well, making a lookup trivial - or are these not safe assumptions?)
    - couchand 4y ago
glenngillen 4y ago
Yeah, the comms around this has been very concerning. Do I need to rotate every config var on all of my apps? Re-install every add-on? While the nature of what limited things they had disclosed to date pointed to this situation part of me wanted to believe it wasn't as bad as I was assuming. And now the trendline on this suggests I should have already done everything I've outlined above. And I'm low confidence anybody is going to be proactive in telling me until it's absolutely obvious that these things have been compromised and exploited.
- xena 4y ago
  IIRC the environment variable settings are encrypted in a physically separate database. However it may be a good idea to rotate your secrets anyways. My hunch would be that there are so many "juicy" targets on Heroku that you probably don't need to worry too much right now unless you are or work for a "juicy" target. This is gonna suck.
  - t_sawyer 4y ago
    I posted a similar worry about the ENV's. Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.
sofixa 4y ago
> I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up They can't, they surely have EU customers so have to follow GDPR disclosure rules, which they might already be afoul of.
benevol 4y ago
> What else was in this database? Typically the password field is stored alongside the rest of the user record. That is where you are supposed to have encrypted as many data fields of the record as possible - in addition to the conventional database encryption which encrypts the database as a whole.
jeremymcanally 4y ago
The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase. If that's the case...yikes.
- jeremyjh 4y ago
  They didn’t say that happened. I’m reading it as their DB was compromised and it’s contents included GH auth tokens.
  - jeremymcanally 4y ago
    How do you read this as the database had tokens in it? > Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database... EDIT: Ah yep you're right. Two tokens in play there: one Heroku API token, one GitHub token. Phew.

onphonenow 4y ago

Ugh - I got the password reset email w silly password complexity - never a good sign.

DethNinja 4y ago
That email made it clear that Heroku lacks fundamental knowledge about security. I’m sure they lost some enterprise customers, I know I don’t open accounts on websites with silly password complexity requirements.
- onphonenow 4y ago
  Google allowed 6 character passwords for a while, and didn't expire them when they increased minimum to 8 for google workspace accounts. This has been fantastic, as users can remember their password forever even if its higher complexity (google does a password strength eval). No rotations either. I'm pretty confident google will pick-up someone trying to brute force a 6 character password. That google will notice connections from new / different IPs or browsers. That's because google asks for my 2FA in various situations but doesn't annoy me by asking for 2FA all the time. I use one govt system that has something like a 14 character password requirement. For even more security if you don't log in for 90 days your account goes inactive and the password EXPIRES! Very secure you say? Well, to regain access you have to provide the answer to a security question - favorite pet! That's a 5 letter word that doesn't change (and is probably pretty guessable). Here is another example: "(b) Information systems must be designed to require passwords to be changed not less frequently than every sixty (60) days." - SBA IT Security Policy - 90 47 4
  - vel0city 4y ago
    For Workspace accounts, an admin can choose to enforce complexity requirements on next login after making changes to the complexity requirements.
    - onphonenow 4y ago
      Right, we've found actually simple passwords but with the mandatory 2FA turned on works really well. The 2FA google uses is a gentle touch in most cases (can persist on a device for 30 days). Google has nice 2FA controls. In a workspace setup you can actually tweak them to match your needs because the lockout / reset path (was) pretty reasonable (when it was onsite). Ie, we could disable certain methods and for some higher security groups you can provide hardware keys and then turn that group up a bit. Never had to rotate passwords and users are glad for that I think. I do wish google offered "Cloud Chrome" for admin staff to open email / click on links etc. Basically a remote VM with chrome but no file access directly.

productceo 4y ago

What are some good Heroku alternatives these days?

kenrose 4y ago
Render.com
- henryaj 4y ago
  Stuff like this worries me - https://community.render.com/t/genuine-question-why-does-render-feel-so-unreliable/3907 https://community.render.com/t/genuine-question-why-does-ren...
  - anurag 4y ago
    (Render founder) Stuff like ^ worries me even more. What keeps me sane is our engineering team obsessing over reliability, and learning from every single incident, no matter how small. We're improving every day.
oxff 4y ago
I've seen fly/io and render/com mentioned in these Heroku discussions, but I have yet to test either of them.
SkyPuncher 4y ago
Aptible (though geared more towards Healthcare/Compliance space). ^ Note: I use to work for Aptible. Great company. Great people. Now working for one of their spin outs.
- btoro 4y ago
  This is super cool. I think you just saved me a massive headache, as I plan to roll out a healthcare app with PHI in the next few months.
  - 4y ago

mac-chaffee 4y ago

I really feel like people don't value intrusion detection enough. Why was Heroku's intrusion detection system "rely on Github's intrusion detection system"?

jlmorton 4y ago
As described, the attacker quietly retrieved OAuth tokens from a single Heroku database, and then very loudly scraped GitHub. I'm not saying it would have been impossible to detect the first action, but it's substantially easier to detect the second.

oxff 4y ago

Why am I hearing about this on checks the fucking date on May 5th instead of, like, month ago.

VWWHFSfQ 4y ago
My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all. Who knows how long Heroku's internal systems were compromised.
- cmeacham98 4y ago
  Even if that's the case, it's still way after April 13th.
  - chii 4y ago
    it might be that it took this amount of time to establish the facts of the events. If they recounted an incorrect version early, it might do more damage than not telling it. I dont know if the github disclosure "includes" heroku's disclosure : https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/ https://github.blog/2022-04-15-security-alert-stolen-oauth-u... - but it was at least april 15th - close-ish to when the event occurred.
    - bigDinosaur 4y ago
      Is it ever true that earlier indications that credentials should be rotated leads to worse outcomes, though, as just one example? I'm sure I've received emails of the form: we suspect there may have been a breach, so we're forcing password resets, and have always taken that fine.
  - dev_tty01 4y ago
    Heroku reported it on 4/15. Read the beginning of the string of updates on the notification page posted here. Also, https://news.ycombinator.com/item?id=31048646 https://news.ycombinator.com/item?id=31048646

rapfaria 4y ago

It is weird to start the day with a Salesforce email in your inbox - had to go checking if it was valid.

koolba 4y ago

> Access to the environment was gained by leveraging a compromised token for a Heroku machine account. Any idea if this involves AWS EC2 Instance Roles? It’s incredibly convenient, but has got to be the scariest feature to enable on a platform that allows arbitrary user code to execute.

darig 4y ago

completelylegit 4y ago

Pretty good alternative to “We and our customer data got hacked”.

craigkerstiens 4y ago

"a Heroku database" was what was known as core-db internally for the longest time. I'm not sure if still the case or not today. But at one point was the source of everything for Heroku. Over time things were moved out, so this isn't an everything that exists has been leaked, but it is not a guarantee that attacker didn't move from one area to another. As someone with some apps on Heroku, having worked there, but no knowledge of the details of the incident more than others... I would: 1. Rotate all creds 2. Ensure logging all connections to the DB (I can't recall how much you can do this on Heroku) 3. Extra heavily audit Github commits and Heroku releases 4. Maybe keep rotating all creds?

srinathkrishna 4y ago
I feel for the team working on this at this time. I hope this doesn't end up accelerating the culling off of Heroku by Salesforce. One of the smartest and nicest bunch of folks I've worked with.
- craigkerstiens 4y ago
  Completely agree, heart goes out to the team. No harsh judgement of all the engineering team there, not a fun situation and hope they know a lot of folks in their corner.
- jeromegv 4y ago
  Heroku is heavily connected with Salesforce now with Heroku Connect, I doubt this is part of the plan.
  - bobx11 4y ago
    There was already a plan by salesforce to kill it and make it into salesforce functions. Someone else here called it project periwinkle.
    - traderj0e 4y ago
      That's frustrating. Heroku is one of the nicest cloud services I've ever used, and am currently using.

ngcc_hk 4y ago

Why not just say hacker ? Know we are on hacker news and hence it is semantically not exactly right. But threat actor … sound more like threaten actor. Just a movie star or drama queen. If one say Heroku was hacked or just sales force … I know one want to manage but somehow the title is not exact right. Too pr.

tempfs 4y ago
Threat actor is industry standard terminology for a malevolent or blackhat brand of hacker. Not all hackers are threat actors. https://en.wikipedia.org/wiki/Hacker https://en.wikipedia.org/wiki/Hacker

owlninja 4y ago

Always interesting to me with events like this that the actions or intentions of the "threat actor" are never discussed. The conversation is always finding tiny holes in the victim's systems and admonishing them for not being prepared.

subroutine 4y ago
Actions other than... (1) obtained access to a Heroku database, (2) downloaded customer GitHub integration OAuth tokens, (3) enumerated metadata on customer repos with the OAuth tokens, (4) downloaded some Heroku private GitHub repos containing source code, and (5) exfiltrated customer hashed and salted passwords?

woodrow 4y ago

I reset my password and then closed my Heroku account today. What a sad end for something that was once a model for software deployment and developer experience.

Mandatum 4y ago

Really, really bad form from the Salesforce Trust team here. Hopefully the Slack acquisition means better, quicker communications. On a scale of Slack to Oracle on breach notifications, this was definitely closer to Oracle.

curuinor 4y ago
rule of thumb: company A acquires company B, then company A does not change to become like company B, company B changes to become like company A. there are exceptions, but few. I think Slack's communications are going to become more crap, personally.
- JorgeGT 4y ago
  Boeing's acquisition of McDonnell Douglas being a sad exception to that rule of thumb.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

lovelearning 4y ago

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetorically or in bad faith. It's a genuine question I have based on a project I did. I researched cybersec tech like SOAR, XDR, security logging, and SIEM in depth. On paper, the marketing for such tech gives the impression that by using them, such breaches can be detected and prevented in real-time. But there seems to be a mismatch between the claims and ground realities. If so, why?

cube00 4y ago
> GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. The three days after being notified to actually revoke the tokens isn't ideal either. Surely if GitHub comes to you and warns you of suspected unauthorised access you'd spend a very limited amount of time and then revoke the credentials to be on the safe side.
- yuliyp 4y ago
  I think (based on what I've read on the linked page) the notice was telling Heroku that "hey, someone used a compromised OAuth token to download your source code" not that "the tokens that you are using to read Github repositories of your users are compromised". Both are Github OAuth tokens, but playing different roles. Presumably the compromise of source code might have been used to help get access to the database that had the Github integration OAuth tokens, and realizing that might indeed have taken a couple days.
Mandatum 4y ago
Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding). In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "security" ends looking the same. They don't pay well, executives have pulled talks and fired speakers who do things they disagree with (the same executives are promoted and remain there - no accountability), they've got a pretty bad wrap within the industry.

abrkn 4y ago

I received an email yesterday asking me to change my password. I did, and updated our services with the regenerated Heroku API key. This morning, I was unable to log into my account and had to reset again. And update our services again.

ryannevius 4y ago
The email I received clearly stated the following: > Due to the nature of this issue, you may be required to reset your passwords again in the future.

peterwallhead 4y ago

Cheers to admin for changing the title, but old title better reflected the last update (and I used it because Heroku doesn't have a permalink to each new update).

tossaway2134 4y ago

This isn't the first time Salesforce Cybersecurity has left us in the lurch while they perform damage control. On 17 May 2019, Salesforce performed maintenance on their databases that clear permission sets for users. My team was able to piece together that the incident happened at about 0200 CDT, and Salesforce didn't take ANY noticeable action for at least 9 hours when they locked all customers out of the platform. Salesforce "fixed" the issue, which meant our Admins had to go in and reapply a bunch of profile settings...no big deal, right? Just a little bit of work for everyone to fix their own accounts. Salesforce acted like it wasn't a big deal. Wrong. If you were a Salesforce customer that built a tool using the Portal or Community tools Salesforce provides for external users, there was a 9 hour window when a customer could log in and instead of seeing the data you were sharing with them, they would see all data for all users. The permissions that indicated that a user should only be able to see their own data was gone. The only reason we knew about this was because we were paying extra for advanced logging. We were able to see a few of our users logged in during this time and looked at customer records they should not have had access to. Salesforce stood fast that exposing data through their Community and Portal tools this did not constitute a breach or even a violation of their SOC-2 Type II compliance. We were lucky that the only people that had access at the time were licensed partners. Nevertheless, our users lost their jobs and were stripped of their licenses. Anyone that was using those tools at the time for any sort of direct customer interaction that shared order history, customer engagement, referral programs, etc. was not so lucky; doubly so if they weren't paying for advanced logging and/or didn't know what to look for. Salesforce was more concerned about covering up their mistakes than they were about telling their customers that there was a problem. Seeing the Heroku notification page gives me PTSD. This looks all-too-familiar to me and I sympathize with those affected by this. I still feel like they were negligent back then, and I wish I knew who to tell to warn others.

stevebmark 4y ago
Salesforce’s monitoring, availability, incident handling, transparency, and accountability, are abysmal. We too we’re affected by the incident. I sincerely hope they lose some of their compliance certifications, because their behavior is unacceptable.

photon12 4y ago

There's going to be a question about the expected probability of this across cloud service providers. I've done security work for multiple cloud service providers and know a lot of people in the industry. I'm not really privy to give details. I can say: dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, security operations teams have a lot of data points to look at, and in theory even changes in personnel in a couple of teams can have an impact on the threat posture for a given set of a company. People are trying. But if you do the math for introduction of attack surface over time versus risk mitigation effort over time for that attack surface you can derive some estimates for likelihood of attack. If your cloud provider isn't providing you that data as a customer, it's not simple to make a determination about likelihood of risk introduction and breach of your cloud provider. This hasn't really been something people talk about because there was a tacit assumption that the biggest companies are mostly getting this right. I'm not trying to say the cloud is inherently broken, I run a lot of workloads in the cloud and trust sensitive data to the cloud. But I do wish there were better ways for customers to have data points with which they could evaluate platforms besides extrapolating the history of public breach reports.

pm90 4y ago
Agreed. cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open”. Its so easy to make a resource accessible to anyone, or an IP accessible from anywhere in the cloud without proper restrictions by internal teams in the company; and often the default is to give teams superadmin to “unblock their time sensitive project” rather than maintaining principle of least access which requires more discipline (and thus effort). Either cloud providers need to assume more responsibility for security or a Federal Agency like the FBI or NIST need to be more proactively engaged in improving the security posture of cloud hosted US corps.
- hnbad 4y ago
  > cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open” So no different from every VC funded startup (or startup seeking VC funding) then? The sentiment of imposing tighter regulations around data security feels counter to the general idea that the lack of regulations around data security (e.g. strong data protection laws) are what allows the US tech industry to dominate compared to its EU equivalents. I'm not disagreeing with this, I'm just pointing out the contradiction and wondering how those who believe the latter would reconcile that belief with demanding the former.

stephenanand 4y ago

[flagged]

samwillis 4y ago

Well, it only seems to be getting worse on this one. I’m keen to get off Heroku, but waiting for one of the newer alternatives (Render/Fly+others) to implement WAL point in time restore for Postgres. It’s the only thing keeping me on Heroku now, but is indispensable. Anyone here from them have any update on when we could see that feature made available?

ryanSrich 4y ago
I’m also interested in this. I have 4 decent sized applications I want to migrate, but I don’t have the resources to self host on GCP or AWS. K8s is nice in theory, but I know from experience it requires too much hand holding (let alone other managed services like a database and cache). From what I’ve seen in the market, fly, render, and railway are the only real Heroku competitors. All seem to be missing a few critical pieces of functionality that is preventing me from migrating. Railway doesn’t appear to allow me to remotely run Python commands, they also don’t have automated nightly backups. Fly and Render respectively have a handful of missing features. It would be amazing if one of these services came out with 1:1 feature parity with Heroku. I’d move in a heartbeat.
- gizzlon 4y ago
  Google Cloud Run or App Engine Standard might be what you want: https://cloud.google.com/run/docs/quickstarts https://cloud.google.com/run/docs/quickstarts https://cloud.google.com/appengine/docs/standard https://cloud.google.com/appengine/docs/standard Unfortunately some things are more difficult than necessary =/ OTOH the GCP ecosystem, network, and data-center presence are vast.
- martinald 4y ago
  Azure web apps is pretty similar to heroku and I find it very easy to use. It has had a Linux version (instead of Windows) for quite a while now and it's pretty solid. Haven't had many real issues with it in many years of use. They also have managed postgres db with very powerful tools.
zoomzoom 4y ago

henryaj 4y ago

Obviously Heroku have handled this horribly - but are any small startups out there considering replatforming? Still seems like a lot of hassle and the competition I've tried (Cloud66, excid3's thing) haven't been as good.

staindk 4y ago
I'd maybe use this as an opportunity to prioritise moving everything over to AWS. I'm sure for some it may be an unwieldy amount of work but for others (depending on tech stack etc.) it ought to be fairly doable. In the long run it'll save money too.
- ojame 4y ago
  I’m interested in peoples experience with this and if it’s relatively true. We (like many others I assume) pay more for Heroku than AWS as it allows us to “outsource” our dev ops. We are a small team (sub 15) with a decent sized, decade old app. We’ve had it on AWS before (and used platforms like BuildKite) but both required much more overhead (in terms of employee salary). Anecdotally I’ve heard the same from friends, though I understand AWS works well and is cheaper if you know AWS well.
  - Dave3of5 4y ago
    > I’m interested in peoples experience with this and if it’s relatively true. My personal experience is that AWS is always more expensive. You don't use AWS because of the cost saving you use it because: * Top of the line h/w * Always the first to bring out new features * Very high availability of resource, like seriously I've never had a time even during the pandemic where they struggled with resource availability * Resilience of AWS systems are very high * Very good support There was an article a few months back where a teams tried moving from Heroku to AWS to save money and ended up spending nearly 3x the amount. Heroku give more resource than you actually pay for by default and it turns out they were using the extra resource during normal operation. When they done their calcs for a switch to AWS they used the quoted resource that Heroku say you get and there system died due to being under resourced. They had to up the resource which pushed them well over their budget. I'll try and find the article. Please Note: Heroku servers are on AWS already

boesboes 4y ago

Seems like they don't have much of their notifications in order either. I haven't heard _anything_ from Heroku on this, my colleague has been getting updates since this started... We are both admins of our companies account.

Narkov 4y ago
How does a company like Salesforce mess this up for such an extended period of time? I understand that companies can make mistakes early on in a critical incident but this has been going on for weeks?!
- xena 4y ago
  Root cause analysis must have failed pretty bad. I worked at Heroku and with how much headcount they've lost my Salesforce starving the beast over the years, it's not surprising that it would take this long for them to react like this. When you scare away the best people with Salesforce policies, their Jira clone built on Salesforce, and overall being difficult to impossible to get more people on the team; yeah people will pack their shit up and leave. Given how much of a web of interdependent and undocumented pain heroku is, I'm not surprised it's taken this long. A security team without any context to Heroku must have had to trace through everything system by system. Especially if core-db got popped.

kaushikt 4y ago

I feel saddened to see this as a heroku side-project user since 2011. We are a small team and were hoping to migrate all our infra to Heroku in the upcoming quarter.

wiredfool 4y ago

TIL that I have a heroku account.

flas9sd 4y ago

last I've seen the Oauth permissions for the Heroku Dashboard given by Github are excessive and include write access to all public repos - as read-only is not an option if I recall correctly, see https://github.com/dear-github/dear-github/issues/113#issuecomment-906779498 https://github.com/dear-github/dear-github/issues/113#issuec... Newer integrations like Github Apps are more granular and can restrict the scope , also ssh deploy keys are an option for other purposes, but specifically the tokens issued for the Heroku Dashboard can write to the public repos of a user or org.

TobiK2020 4y ago

I'm really wondering right now how hard or easy it was to suddenly change all of their customers' passwords in the system.

t_sawyer 4y ago

I'm interested if application ENVs were stolen. If they stored Github APIs in plain text I'm sure our ENVs were too. I went ahead and reset my planetscale passwords and moved my app out of Heroku. But I only had two apps hosted there I feel for anyone with a large number of apps on there.

anyfactor 4y ago

I have seen nothing but bad news about Heroku recently. Albeit nothing provides that much ease to use on a free or affordable tier when you have minimum computing demands. I use it as my fundamental cloud provider for proof of concept stuff, so does many people. Salesforce is doing a terrible job managing this lucrative platform. I have no idea why muck up a good service like that. They have some plugins and a postgres connector but the drive to innovate, the drive to even care stops there. All these news act as a reminder that I should move my code to a "real" cloud provider.

bspear 4y ago

The comms for this security breach is a textbook case study of what NOT to do

dataspun 4y ago

It seems as though enterprise customers of Heroku that leverage data products tied to their Salesforce instances, such as Heroku Connect and Salesforce Connect, are at risk of serious data breach and possibly haven’t realized it. Nearly a month since the initial 2413 incident, we’re still learning of its full scope. If customers haven’t rotated credentials — mistakenly thinking the incident is isolated to web apps and GitHub — the risk is still present. Can it go any more sideways? Yet the communications from Heroku and Salesforce are disjointed, even amateur.