3 ms·

It sounds they have an idea how it could have happened — the Lish vulnerability — but they don't know if that's how it actually did happen or if there's another

by chc 11y ago

It sounds they have an idea how it could have happened — the Lish vulnerability — but they don't know if that's how it actually did happen or if there's another undiscovered vulnerability lurking.

tptacek 11y ago
Ok, now help me understand how that can be true given this quote from the advisory: The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access. Did they get compromised or not?
- deleted 11y ago
  [deleted]
- craig131 11y ago
  "no evidence" or insufficient logging. The wording is spun to their favor throughout the entire advisory. Like "Security Investigation Retrospective" making it sound like they have the whole thing wrapped up. Also that everyone will be fine once their passwords are reset, when it sounds like they still don't really have any solid idea if they fixed the problem or not.
- chc 11y ago
  My reading was that: 1. Their security partner (whoever that is) didn't see the Lish vulnerability, either because Linode's security team had already fixed it or because they just missed it. 2. Their records weren't sufficient to show the breach itself.
  - tptacek 11y ago
    I'm not just trying to be argumentative. Here, let me quote more specifically: no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. I feel like I must be misreading something. Didn't they say earlier that they found secrets from their account credentials database on a customer instance that was used to attack (apparently) PagerDuty? That's not "no evidence of abuse or misuse of Linode’s infrastructure"?
    - fweespeech 11y ago
      Honestly, it is likely because they are spinning/lying that it is confusing to you. If you just assume they are attempting misdirection it makes perfect sense for them to insist things are fine.
    - drewcrawford 11y ago
      you forgot "that would have resulted in the disclosure of customer credentials". I'm speculating, but having been a fly on the wall at this kind of meeting before, here's my theory of how this went down: Manager: "So somebody got the key to generate one-time password tokens for PagerDuty. How did that happen?" Engineer: "I have no idea." Manager: "What about that Lish vulnerability? Could it have been that?" Engineer: "Could have been." Manager: "We don't know?" Engineer: "I mean, they could have gotten it that way, but we don't have logging for security vulnerabilities, because, you know, we would just fix them instead of logging that they occurred..." Manager: "Okay, I want a second opinion. Time to bring in $SECURITY_PARTNER." $SECURITY_PARTNER: "What Engineer said. And here's a bill for our time." Manager: "What about if the customer screwed up and lost a device? Could it be that?" Engineer: "It could be that." Support: "PagerDuty said something about wiping their phone." Engineer: "If they did lose a device, it would be indistinguishable from the evidence we have." Manager: "So, maybe we weren't hacked after all?" Engineer: "Maybe not." Manager: "Any other theories?" Engineer: "I just want to go on record... AGAIN... that I don't trust our ColdFusion infrastructure at all. The people responsible have all left, and none of us understand it. We need to rewrite in a cooler language, like Python. And hire some actual security people." Manager: "Sigh. We've gotten enough flak over this that I guess I'll put it in the budget. Okay, good meeting everyone. I'll write this up in a blogpost, because we need to show transparency. That way customers are at least as confused as we are."
      - TheSwordsman 11y ago
        In our case someone logged in to the Linode Manager first. Whomever logged in was confident they knew the password of the user and the MFA token. That would indicate that Linode's password hashes may have been cracked offline.
      - 11y ago
- novaleaf 11y ago
  from the disclosure: > After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation. so, it was an inside job then?

by chc 11y ago

tptacek 11y ago

Ok, now help me understand how that can be true given this quote from the advisory: The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access. Did they get compromised or not?

deleted 11y ago
[deleted]
craig131 11y ago
"no evidence" or insufficient logging. The wording is spun to their favor throughout the entire advisory. Like "Security Investigation Retrospective" making it sound like they have the whole thing wrapped up. Also that everyone will be fine once their passwords are reset, when it sounds like they still don't really have any solid idea if they fixed the problem or not.
chc 11y ago
My reading was that: 1. Their security partner (whoever that is) didn't see the Lish vulnerability, either because Linode's security team had already fixed it or because they just missed it. 2. Their records weren't sufficient to show the breach itself.
- tptacek 11y ago
  I'm not just trying to be argumentative. Here, let me quote more specifically: no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. I feel like I must be misreading something. Didn't they say earlier that they found secrets from their account credentials database on a customer instance that was used to attack (apparently) PagerDuty? That's not "no evidence of abuse or misuse of Linode’s infrastructure"?
  - fweespeech 11y ago
    Honestly, it is likely because they are spinning/lying that it is confusing to you. If you just assume they are attempting misdirection it makes perfect sense for them to insist things are fine.
  - drewcrawford 11y ago
    you forgot "that would have resulted in the disclosure of customer credentials". I'm speculating, but having been a fly on the wall at this kind of meeting before, here's my theory of how this went down: Manager: "So somebody got the key to generate one-time password tokens for PagerDuty. How did that happen?" Engineer: "I have no idea." Manager: "What about that Lish vulnerability? Could it have been that?" Engineer: "Could have been." Manager: "We don't know?" Engineer: "I mean, they could have gotten it that way, but we don't have logging for security vulnerabilities, because, you know, we would just fix them instead of logging that they occurred..." Manager: "Okay, I want a second opinion. Time to bring in $SECURITY_PARTNER." $SECURITY_PARTNER: "What Engineer said. And here's a bill for our time." Manager: "What about if the customer screwed up and lost a device? Could it be that?" Engineer: "It could be that." Support: "PagerDuty said something about wiping their phone." Engineer: "If they did lose a device, it would be indistinguishable from the evidence we have." Manager: "So, maybe we weren't hacked after all?" Engineer: "Maybe not." Manager: "Any other theories?" Engineer: "I just want to go on record... AGAIN... that I don't trust our ColdFusion infrastructure at all. The people responsible have all left, and none of us understand it. We need to rewrite in a cooler language, like Python. And hire some actual security people." Manager: "Sigh. We've gotten enough flak over this that I guess I'll put it in the budget. Okay, good meeting everyone. I'll write this up in a blogpost, because we need to show transparency. That way customers are at least as confused as we are."
    - TheSwordsman 11y ago
      In our case someone logged in to the Linode Manager first. Whomever logged in was confident they knew the password of the user and the MFA token. That would indicate that Linode's password hashes may have been cracked offline.
    - 11y ago
novaleaf 11y ago
from the disclosure: > After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation. so, it was an inside job then?