8 ms·
PagerDuty hacked. And finally comes clean 21 days later
- shazow 11y ago> Based on the investigation, the attacker bypassed multiple layers of authentication and gained unauthorised access to an administrative panel provided by one of our infrastructure providers. Sounds like a VPS provider screwed up? If that's the case, would love to know which one it was. :/
- thaumaturgy 11y agoThey have a couple of subdomains hosted with AWS, in the us-west region looks like. Not really definitive, but AWS is common infrastructure for a lot of startups these days, and it wouldn't be the first time that someone managed to find their way into an AWS admin panel.
- deleted 11y ago[deleted]
- ryanlol 11y agoLinode seems like the more likely culprit here.
- thaumaturgy 11y agoGiven Linode's past history, I'd regrettably have to agree. I didn't see evidence of Linode infrastructure when I posted my earlier comment. It's frustrating that PagerDuty can't / won't share that information. I can't see how that's helping law enforcement's investigation, and it's potentially putting a lot of other people at risk depending on the nature and severity of the compromise.
- fsaintjacques 11y agoOr maybe they had a weak password for their administrative account@VPS?
- cddotdotslash 11y agoIt does not immediately mean that. If it were AWS, there are ten thousand things PagerDuty could have done wrong: lost an access key, made an IAM role too permissive and lost access to the instance, turned off key-based ssh, made the ports open to the public, hard coded credentials, simply used a weak password for their console, not enabled TFA, lost the root credentials somewhere, allowed a privilege escalation by giving one key the ability to create more keys and attach permissions. The list goes on. Plug: I'm working on an open-source and hosted project called CloudSploit that can scan for these kinds of things. Check it out at https://cloudsploit.com https://cloudsploit.com
- senthilnayagam 11y agoJust PR content, too little, too late. now wait and watch if someone will dump this on pastebin or some torrent. compare it with what slack posted under similar scenario http://slackhq.com/post/114696167740/march-2015-security-incident-and-launch-of-2fa http://slackhq.com/post/114696167740/march-2015-security-inc...
- davidgerard 11y ago* cue PagerDuty "sad trombone" alert sound
- davidgerard 11y agocue PagerDuty "sad trombone" alert sound
- deleted 11y ago[deleted]
- zaidf 11y agoHas anyone written a checklist for CEOs/CTOs of stuff to do/avoid when such incidents are discovered?
- FLGMwt 11y agoI would love this from someone with the know
- sarciszewski 11y agoIn my opinion: the Pager Duty team did okay. Disclosing sooner would have been cool, but then again law enforcement might have had the gun to their heads on disclosing anything. They explained their bcrypt implementation awkwardly, though. This is what their code did: https://github.com/plataformatec/devise/blob/1a0192201b317d3f1bac88f5c5b4926d527b1b39/lib/devise/encryptor.rb https://github.com/plataformatec/devise/blob/1a0192201b317d3... I would not have advised them to go the pepper route, but it's probably a moot point now. In their favor: When asked, they gave specific answers up to the point that they were allowed to by law enforcement. More companies should follow their example here.
- eddyg 11y agoRecent discussion on HN: http://news.ycombinator.com/item?id=9977399 http://news.ycombinator.com/item?id=9977399
- juliangregorian 11y agoThis is way, way too prevalent -- people thinking that by doing more things to the password you are being more secure. The bcrypt algo already salts, there's no need to prepend extra stuff. And if you really had 80 chars of preliminary table condiments and your bcrypt implementation truncated at 72 -- well that's just textbook.
- jacquesm 11y agoIt was less than a week ago that someone here proclaimed loudly that 'security is just as bad as premature optimization'. I wonder how the pagerduty folks feel right now, but I'll bet that 'premature optimization' is not the first thought that comes to mind. A good start-up fires on all cylinders and strikes a balance when it comes to features, user friendliness, performance and security. Ignore any one of those and you'll lose the game eventually. Let's hope they pull together and up their game then manage to survive, I'd trust them much better after failing like this than before assuming they learned their lesson.
- meowface 11y ago>It was less than a week ago that someone here proclaimed loudly that 'security is just as bad as premature optimization'. Although I work in infosec, I could see how this case could be made if your company/service does not handle or store any customer or financial data at all, and does not have any particularly sensitive information. Otherwise, it's a pretty ridiculous statement.
- jacquesm 11y agoThat definitely wasn't the context. I won't link to the comment here because I think it is an embarrassment to the maker but it's easy enough to find if you want it badly.
- andrewljohnson 11y agoYeah, jacques is definitely pulling my comment far out of context. I was commenting on his suggestion that everyone start serving their own JavaScript files, and writing custom code to check their sanctity. I can't see the parallel here, or how that practice would have helped PagerDuty.