5 ms·
Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 https
by TheSwordsman 11y ago
Hey There,
I'm a PagerDuty employee and am the same individual who made this post on the last HN thread:
* https://news.ycombinator.com/item?id=10845985 https://news.ycombinator.com/item?id=10845985
Unfortunately, there are some facts in Linode's post that are not correct.
>On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notification confirming a root password reset for one of their Linodes. Our initial investigation showed the unauthorized login was successful on the first attempt and resembled normal activity.
This is almost correct. Someone got in to our account on the first try. They knew the password and a valid TOTP token. Although, Linode's email isn't what notified is, it was our intrusion detection system.
>On July 12, in anticipation of law enforcement’s involvement, the customer followed up with a preservation request for a Linode corresponding to an IP address believed to be involved in the unauthorized access. We honored the request and asked the customer to provide us with any additional evidence (e.g., log files) that would support the Linode being the source of malicious activity. Neither the customer nor law enforcement followed up and, because we do not examine customer data without probable cause, we did not analyze the preserved image.
This is partially correct. We informed Linode that we saw suspicious activity within their network, and reached out to them to inform them. We provided any and all logs we had. We also informed them that we passed the info on to law enforcement, in case they wanted to proactively preserve the data. The knew we had no further information, and as such didn't ask for anything additional.
>On the same day, the customer reported that the user whose account was accessed had lost a mobile device several weeks earlier containing the 2FA credentials required to access the account, and explained that the owner attempted to remotely wipe the device some time later. In addition, this user employed a weak password. In light of this information, and with no evidence to support that the credentials were obtained from Linode, we did not investigate further.
The story behind the mobile device is totally incorrect. The user did not lose their device, the device had been restored (intentionally wiped) 9 months prior to the compromise. The user got a new device, and never set up MFA on their new phone after wiping the old one. The device was, and still is, in the user's possession. The device has not been powered on in a long while.
The user who was compromised was no longer in possession of their MFA secret. They deleted it, intentionally, with no backups existing.
If anyone here is going to be at Velocity 2016 in Santa Clara, or at Monitorama PDX 2016, I'll be giving talks on how PagerDuty was compromised back in July. This includes full details of how this happened, including the details of the mobile device referenced above. There are some details in my talk that don't line up with the blog post provided by Linode. :)
- tptacek 11y agoWow.
- stock_toaster 11y agoNo kidding. After reading all this I think people would be nuts to continue using Linode for anything critical (hell, maybe even for anything at all) at this point. How many times has this kind of thing happened to them now?
- apandhi 11y agoLinode dropping the ball with PagerDuty is huge. I can't even begin to imagine how much money your customers could lose if PagerDuty infrastructure went down.
- jonursenbach 11y agoWhich makes me wonder why they're even with Linode.
- TheSwordsman 11y agoWe aren't. Did a full migration in August, 2015.
- jsmthrowaway 11y agoI watched PagerDuty leadership run out of the room when Amazon completely shit itself during a conference in 2011, and I heard a few months later that they'd signed up with Linode at some point after that. I could probably put two and two together there as a reliability strategy to back up against AWS failures. Keep in mind Linode had a pretty good rep at the time. I wouldn't second guess them on the call, as I probably would have made it at the time, too. They didn't dig in or lock themselves in with debt, and bailed when the time was right, too.
- stephendicato 11y ago> Although, Linode's email isn't what notified is, it was our intrusion detection system. Are you able to elaborate on this? I understand you may not want to name specific vendors/products in the name of operational security but it sounds like in this scenario whatever is in place actually did its job.