6 ms·

This is turning into a complete train wreck and a case study on how not to communicate with your customers. For those of you that haven't been following, Herok

This is turning into a complete train wreck and a case study on how not to communicate with your customers. For those of you that haven't been following, Heroku has been adding non-update updates to this security thread over the last couple of weeks, which began with the announcement that some (or maybe all) of their GitHub granted access tokens had been compromised: https://status.heroku.com/incidents/2413 https://status.heroku.com/incidents/2413 Now, weeks later, we're hearing that all account passwords are being reset, and for some reason if you have been using an HTTPS-style log drain that you should reset any secrets related to it as well. Heroku needs to come out and clearly state what they know about this situation, and more importantly what they don't know -- which is starting to sound like the answer is "a lot". It's not even clear they know how this all happened -- whatever door was left open might still be open. So if you've gone and rotated all of your application secrets (which you probably should do), be prepared to rotate them again when this is all over.

heartbreak 4y ago
They sent out a notification about cycling shared dataclip slugs today too.
jrochkind1 4y ago
I found the email I received about the logdrain today to be particularly confusing. "any secrets related to it" indeeed. The specific wording in the email was... "We recommend updating and refreshing the credentials used with those log drains as soon as possible." I'm still not entirely sure if I've reset/rotated everything I need to, what is "any credentials used with"? Neither the email nor any docs it linked to was clear about exactly what they are suggesting be rotated. That message wasn't very specific, while also they're not providing the context about the breach that one could use to fill in the gaps. At this point it kinds of sounds like... everything there is was compromised?
- jarcoal 4y ago
  Very confusing! To clarify for others, what they mean is that if there was a secret embedded in the log drain URL, rotate it. This is often the case for HTTPS log drains. Example: https://datadog.com/logs?api_key=abc123 https://datadog.com/logs?api_key=abc123
metadat 4y ago
Heroku used to be freaking awesome, back in 2012. Ever since Craig Kersteins left circa 2015-ish, the UX, QoS, and platform really seem to have taken a dive into the complexity and nonsensical deep end. It's what happens when the product visionaries get bored and leave. Such a shame.
- devin 4y ago
  You're being downvoted because having a "product visionary" does not in any way protect your company from being owned. It happens. If your point is that previous leadership would have communicated the situation more clearly, then perhaps you have a point, but even then it's purely speculation and not particularly useful to the current discussion.
  - metadat 4y ago
    It's okay, thanks for the feedback. Downvotes don't bother me, they are a welcome signal.
  - genewitch 4y ago
    in this context "pwned" is probably a better fit; natch?
- craigkerstiens 4y ago
  Don't want to comment too much on this thread as I don't know the exact details of the incident and pulling for the team that is still around as I know this can't be an easy time for them. It does seem like the issue is not good and may continue to be trickle of updates like this for a while. Hugops to the team. A wow, and thanks for the praise but the credit goes to way more than me. The team around in the early days was unbelievable, I learned a ton on building products and developer experience from James/Adam (founders) in particular, though Heroku wouldn't have gotten there without Orion (the other founder) as well. Byron, PvH, Mark, Noah, Morten, Oren were absolutely huge in so many ways to the leadership and direction of Heroku. And I'm sure I'm going to get messages from 50 others there in the early days that I didn't name drop them, the collective team was an awesome team and pushed each other really well. At around 2015 it did feel like there was attrition and the technical leadership and vision started to fade. It wasn't me, it was a lot of us moved on to the next thing. At the time it wasn't Salesforce taking control or one person, we'd all put a lot into it and various folks moved on. Adam/Orion/James gave an incredibly amount and were understandably ready to recharge. Still very proud of what we created at that time, what it did for developer experience, and personally (along with that original Heroku Postgres team) trying to do what I describe as unfinished business for creating the amazing developer experience of Postgres.
Exuma 4y ago
Is papertrail a log drain?
- teaearlgraycold 4y ago
  No
  - glenngillen 4y ago
    How do you know that? And that it doesn’t need to also be rotated?
    - all-the-lights 4y ago
      It does. My thinking is that Heroku rotated everything they could do on their own, and need users to handle custom drains. However, looking at the json output, updated_at dates for the heroku-managed drains are old. I have a question back them about that. Related, my hope is that the drains, passwords, etc. are efforts towards doing literally everything possible before re-enabling integrations so they can do so with more confidence (vs breach damage control). Fingers crossed.
- javawizard 4y ago
  Yes it is. Note that OP's comment only applies to those who connected Papertrail to their Heroku apps manually instead of using the Heroku addon to do it.
Ozzie_osman 4y ago
So the original issue was described as a leak of github oauth tokens, and made it sound like the risk would be someone using oauth tokens to access github repos. Resetting passwords implies something else may have been compromised (passwords, either hopefully encrypted), but is a pretty scary ask for them to make without providing more context here. Trainwreck indeed.
- andreareina 4y ago
  I certainly hope that passwords aren't encrypted but run through an appropriately-expensive password hash.
  - Ozzie_osman 4y ago
    Yes, I should have said hashed not encrypted.
    - zeepzeep 4y ago
      hashing is one-way encryption tho
      - Tiddles-the2nd 4y ago
        That's the point, take a look into salting + hashing passwords
- matthewcford 4y ago
  Resetting the password resets your API key, which is different from the Oauth tokens.
jarcoal 4y ago
I just received an email back from an Incident Handler at Salesforce. I wrote: > A statement that confirms whether or not config variables and secrets were accessed, or that you're not sure, needs to be sent out. To which they replied: > We currently have no evidence that Heroku customers’ secrets stored in config Var were accessed. If we find any evidence of unauthorized access to customer secrets, we will notify affected customers without undue delay. Take that as you will, but it doesn't fill me with confidence.
- emilsedgh 4y ago
  Why? I think it's pretty evident that there's no reason to believe there's been a security breach there as far as they understand as of now.
  - macNchz 4y ago
    There was certainly a breach three weeks ago that they seem to have been investigating since. I am, like the commenter above, not filled with confidence about their statement, mostly because of the total lack of transparency so far. The fact they’re only now sending additional notifications to rotate creds hints at something bigger than they initially announced, but really we have no idea since they never gave much detail in the first place.
  - adrr 4y ago
    Github tokens were accessed and used to attack things outside of heroku. Something was breached. We don't know what, when it occurred and extent of the breach and I bet heroku doesn't know either.
  - wlll 4y ago
    There was a security breach. They don't know what the breach was, they don't know if the attacker has access to other systems, they just don't know. No evidence of something is not particularly useful information if you weren't even looking out for it in the first place.
- endgame 4y ago
nomilk 4y ago
It's a small comms 101 thing, but the email is from "Salesforce Incident Alerts". Since the email's communicating to Heroku customers about a Heroku incident, the email should be from "Heroku Incident Alerts". I know it's small, but some will skip the email because they don't use Salesforce software directly and wouldn't anticipate emails from a parent company.
- wildrhythms 4y ago
  Yep, I would have 100% skimmed past that.
- nanoservices 4y ago
  Its small and deliberate. Setting the mental state of the customer to obfuscate the responsible party by throwing in Salesforce. A deliberate dark pattern.
- johnmw 4y ago
  Another small point that contributes to the poor communication - as of 4th May, the option to connect to Github is still there and the documentation hasn't been updated [1]. If you try to connect you get an obscure error that tells you nothing about the situation. [1] https://devcenter.heroku.com/articles/github-integration https://devcenter.heroku.com/articles/github-integration
snowwolf 4y ago
>If you used your previous password on any other sites, we highly recommend you also change your password on those sites. This is the most concerning part of that email, as it implies more than an "out of an abundance of caution", but rather that they suspect their password DB has been compromised. Thinking about it, it does sound the most likely as they were probably the same DB the customer oAuth tokens were stored in that were used to access Github repositories. But if they already knew the data was stored together why wait till now to reset passwords?
- m12k 4y ago
  Just to verify - having TOTP-based 2FA enabled doesn't help in case of a password DB breach, right? Since the protocol is based on a shared password, which means an attacker would be able to generate valid tokens using the secret they got from the breach. (looks like there's work underway to make a breach-resistant alternative to TOTP[1]) This means that assuming the DB is using proper salt+hash, the main differentiator is the strength of your password. If it's a relatively short one that can be brute-forced/found via dictionary+small mutation, then attackers could possibly log in as you. If it's a strong password from a password manager, then that will likely have kept them from being able to crack your password. Of course all this only has value if we assume that only the password db was breached. If they managed to access the place your env-var/secrets are stored, then all bets are off. [1] https://www.mdpi.com/1424-8220/20/20/5735/htm https://www.mdpi.com/1424-8220/20/20/5735/htm
  - jrochkind1 4y ago
    I don't understand enough about the protocol to judge your claim, but it's challenging my assumptions about what 2FA is for. If 2FA with TOTP does not protect you in cases where the attacker knows your password... what is it for? I thought that's what it was for.
    - bigDinosaur 4y ago
      The server also contains the secret, so if that secret is leaked then the attacker can generate new tokens. It protects you in the case that your password was stolen, but nothing else e.g. via phishing.
Mo3 4y ago
Fwiw, I skimmed over all emails I got until I saw this thread.
- pelagicAustral 4y ago
  You and me both, buddy

heartbreak 4y ago

They sent out a notification about cycling shared dataclip slugs today too.

jrochkind1 4y ago

I found the email I received about the logdrain today to be particularly confusing. "any secrets related to it" indeeed. The specific wording in the email was... "We recommend updating and refreshing the credentials used with those log drains as soon as possible." I'm still not entirely sure if I've reset/rotated everything I need to, what is "any credentials used with"? Neither the email nor any docs it linked to was clear about exactly what they are suggesting be rotated. That message wasn't very specific, while also they're not providing the context about the breach that one could use to fill in the gaps. At this point it kinds of sounds like... everything there is was compromised?

jarcoal 4y ago
Very confusing! To clarify for others, what they mean is that if there was a secret embedded in the log drain URL, rotate it. This is often the case for HTTPS log drains. Example: https://datadog.com/logs?api_key=abc123 https://datadog.com/logs?api_key=abc123

metadat 4y ago

Heroku used to be freaking awesome, back in 2012. Ever since Craig Kersteins left circa 2015-ish, the UX, QoS, and platform really seem to have taken a dive into the complexity and nonsensical deep end. It's what happens when the product visionaries get bored and leave. Such a shame.

devin 4y ago
You're being downvoted because having a "product visionary" does not in any way protect your company from being owned. It happens. If your point is that previous leadership would have communicated the situation more clearly, then perhaps you have a point, but even then it's purely speculation and not particularly useful to the current discussion.
- metadat 4y ago
  It's okay, thanks for the feedback. Downvotes don't bother me, they are a welcome signal.
- genewitch 4y ago
  in this context "pwned" is probably a better fit; natch?
craigkerstiens 4y ago
Don't want to comment too much on this thread as I don't know the exact details of the incident and pulling for the team that is still around as I know this can't be an easy time for them. It does seem like the issue is not good and may continue to be trickle of updates like this for a while. Hugops to the team. A wow, and thanks for the praise but the credit goes to way more than me. The team around in the early days was unbelievable, I learned a ton on building products and developer experience from James/Adam (founders) in particular, though Heroku wouldn't have gotten there without Orion (the other founder) as well. Byron, PvH, Mark, Noah, Morten, Oren were absolutely huge in so many ways to the leadership and direction of Heroku. And I'm sure I'm going to get messages from 50 others there in the early days that I didn't name drop them, the collective team was an awesome team and pushed each other really well. At around 2015 it did feel like there was attrition and the technical leadership and vision started to fade. It wasn't me, it was a lot of us moved on to the next thing. At the time it wasn't Salesforce taking control or one person, we'd all put a lot into it and various folks moved on. Adam/Orion/James gave an incredibly amount and were understandably ready to recharge. Still very proud of what we created at that time, what it did for developer experience, and personally (along with that original Heroku Postgres team) trying to do what I describe as unfinished business for creating the amazing developer experience of Postgres.

Exuma 4y ago

Is papertrail a log drain?

teaearlgraycold 4y ago
No
- glenngillen 4y ago
  How do you know that? And that it doesn’t need to also be rotated?
  - all-the-lights 4y ago
    It does. My thinking is that Heroku rotated everything they could do on their own, and need users to handle custom drains. However, looking at the json output, updated_at dates for the heroku-managed drains are old. I have a question back them about that. Related, my hope is that the drains, passwords, etc. are efforts towards doing literally everything possible before re-enabling integrations so they can do so with more confidence (vs breach damage control). Fingers crossed.
javawizard 4y ago
Yes it is. Note that OP's comment only applies to those who connected Papertrail to their Heroku apps manually instead of using the Heroku addon to do it.

Ozzie_osman 4y ago

So the original issue was described as a leak of github oauth tokens, and made it sound like the risk would be someone using oauth tokens to access github repos. Resetting passwords implies something else may have been compromised (passwords, either hopefully encrypted), but is a pretty scary ask for them to make without providing more context here. Trainwreck indeed.

andreareina 4y ago
I certainly hope that passwords aren't encrypted but run through an appropriately-expensive password hash.
- Ozzie_osman 4y ago
  Yes, I should have said hashed not encrypted.
  - zeepzeep 4y ago
    hashing is one-way encryption tho
    - Tiddles-the2nd 4y ago
      That's the point, take a look into salting + hashing passwords
matthewcford 4y ago
Resetting the password resets your API key, which is different from the Oauth tokens.

jarcoal 4y ago

I just received an email back from an Incident Handler at Salesforce. I wrote: > A statement that confirms whether or not config variables and secrets were accessed, or that you're not sure, needs to be sent out. To which they replied: > We currently have no evidence that Heroku customers’ secrets stored in config Var were accessed. If we find any evidence of unauthorized access to customer secrets, we will notify affected customers without undue delay. Take that as you will, but it doesn't fill me with confidence.

emilsedgh 4y ago
Why? I think it's pretty evident that there's no reason to believe there's been a security breach there as far as they understand as of now.
- macNchz 4y ago
  There was certainly a breach three weeks ago that they seem to have been investigating since. I am, like the commenter above, not filled with confidence about their statement, mostly because of the total lack of transparency so far. The fact they’re only now sending additional notifications to rotate creds hints at something bigger than they initially announced, but really we have no idea since they never gave much detail in the first place.
- adrr 4y ago
  Github tokens were accessed and used to attack things outside of heroku. Something was breached. We don't know what, when it occurred and extent of the breach and I bet heroku doesn't know either.
- wlll 4y ago
  There was a security breach. They don't know what the breach was, they don't know if the attacker has access to other systems, they just don't know. No evidence of something is not particularly useful information if you weren't even looking out for it in the first place.
endgame 4y ago

nomilk 4y ago

It's a small comms 101 thing, but the email is from "Salesforce Incident Alerts". Since the email's communicating to Heroku customers about a Heroku incident, the email should be from "Heroku Incident Alerts". I know it's small, but some will skip the email because they don't use Salesforce software directly and wouldn't anticipate emails from a parent company.

wildrhythms 4y ago
Yep, I would have 100% skimmed past that.
nanoservices 4y ago
Its small and deliberate. Setting the mental state of the customer to obfuscate the responsible party by throwing in Salesforce. A deliberate dark pattern.
johnmw 4y ago
Another small point that contributes to the poor communication - as of 4th May, the option to connect to Github is still there and the documentation hasn't been updated [1]. If you try to connect you get an obscure error that tells you nothing about the situation. [1] https://devcenter.heroku.com/articles/github-integration https://devcenter.heroku.com/articles/github-integration

snowwolf 4y ago

>If you used your previous password on any other sites, we highly recommend you also change your password on those sites. This is the most concerning part of that email, as it implies more than an "out of an abundance of caution", but rather that they suspect their password DB has been compromised. Thinking about it, it does sound the most likely as they were probably the same DB the customer oAuth tokens were stored in that were used to access Github repositories. But if they already knew the data was stored together why wait till now to reset passwords?

m12k 4y ago
Just to verify - having TOTP-based 2FA enabled doesn't help in case of a password DB breach, right? Since the protocol is based on a shared password, which means an attacker would be able to generate valid tokens using the secret they got from the breach. (looks like there's work underway to make a breach-resistant alternative to TOTP[1]) This means that assuming the DB is using proper salt+hash, the main differentiator is the strength of your password. If it's a relatively short one that can be brute-forced/found via dictionary+small mutation, then attackers could possibly log in as you. If it's a strong password from a password manager, then that will likely have kept them from being able to crack your password. Of course all this only has value if we assume that only the password db was breached. If they managed to access the place your env-var/secrets are stored, then all bets are off. [1] https://www.mdpi.com/1424-8220/20/20/5735/htm https://www.mdpi.com/1424-8220/20/20/5735/htm
- jrochkind1 4y ago
  I don't understand enough about the protocol to judge your claim, but it's challenging my assumptions about what 2FA is for. If 2FA with TOTP does not protect you in cases where the attacker knows your password... what is it for? I thought that's what it was for.
  - bigDinosaur 4y ago
    The server also contains the secret, so if that secret is leaked then the attacker can generate new tokens. It protects you in the case that your password was stolen, but nothing else e.g. via phishing.

Mo3 4y ago

Fwiw, I skimmed over all emails I got until I saw this thread.

pelagicAustral 4y ago
You and me both, buddy