4 ms·

BrowserStack attack post-mortem

51 points by simonsarris 12y ago

akerl_ 12y ago
"All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked." I appreciate that they're trying to communicate the strength of bcrypt, but this is more than a bit hyperbolic.
- smoyer 12y ago
  That passage also raised the hair on the back of my neck ... I was trying to ignore it when I came across this phrase: "Our restoration process is indeed tamper-proof." (and another fragment from their user documentation). Given the current climate in computer security, I think we should be much more careful about how we phrase our guarantees and more specifically how secure each service is. These statements imply guarantees that are simply impossible to make ... and we've seen flaws in our crypto tools change the security landscape overnight.
deleted 12y ago
[deleted]
sehrope 12y ago
> The old prototype machine had our AWS API access key and secret key. Once the hacker gained access to the keys, he created an IAM user, and generated a key-pair. Ouch! This is why you must practice the principle of least privilege when provisioning access keys, especially ones that can control your entire infrastructure. > He was then able to run an instance inside our AWS account using these credentials, and mount one of our backup disks. Good security practices are like an onion ... it's got many layers and will most likely make you cry. This is why you should be encrypting your backups. The backup program doesn't even need to be able to read the backups it creates, it only needs to to be able to write them (i.e. the public half of the key). If you're thinking, "Why does that matter? It can already read the plain-text data it's backing up!" then you're not thinking about history or multiple servers. Each may be able to read the current state of its own file system, but if they only have the public half, they can't read the historical state or each other's backups. The damage would be limited to a single server and only what's live on it at that moment. > We were able to verify the actions of the hacker using AWS CloudTrail, which confirmed that no other services were compromised, no other machines were booted, and our AMIs and other data stores were not copied. CloudTrail is awesome and if you're on AWS you really should enable it. It costs peanuts compared to what it provides. In cases like this, where the attacker takes over your entire infrastructure, it may be the only log you have of what happened.
travem 12y ago
See https://news.ycombinator.com/item?id=8594652 https://news.ycombinator.com/item?id=8594652 for more discussion on this.
Someone1234 12y ago
> Our database logs confirmed that user data was partially copied, but no user test history was compromised. Therefore all user data remains wholly intact. You what? You said hashed passwords and email addresses were taken for 5K users. What does "wholly intact" mean? > All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked. However, as an added precaution, we suggest that users change their BrowserStack account passwords. Those 5K passwords are compromised. It isn't an "added precaution" it is a mandatory requirement. The whole point of bcrypt is to buy you time to change your password, they have misunderstood that and think bcrypt is "unbreakable" so people don't need to change passwords. That's super ignorant. I would expire all 5K users (or ALL users) passwords immediately and send them recovery emails.
- icelancer 12y ago
  Expire all is easily the right decision.
general_failure 12y ago
Oh wow. Basically the whole infrastructure was compromised. > He began to copy one of our tables, which contained partial user information, including email IDs, hashed passwords, and last tested URL. His copy operation locked the database table, which raised alerts on our monitoring system. How does a copy operation lock a database table? Because you set thresholds on the number of reads? I find it a little surprising that such a knowledgeable hacker would make such a rookie mistake. Can someone clarify?
- rememberlenny 12y ago
  Copy operations regularly lock tables for large databases. I see that frequently for backpubs on our mysql db.
nodesocket 12y ago
I'd be curious to know the exact shellshock entry point into the original target server and how it was used to gain the AWS credentials on that server specifically.
nodesocket 12y ago
> We have a trace and the IP of the hacker. I doubt they have the hackers real IP. The hacker was almost certainly using a proxy right? Even if they have an IP address of the hacker, what is the likelihood of tracking it down to a physical person?
DougN7 12y ago
They said they had thousands of servers. Can that possibly be true? Is BrowserStack really that large?

51 points by simonsarris 12y ago

"All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked." I appreciate that they're trying to communicate the strength of bcrypt, but this is more than a bit hyperbolic.

smoyer 12y ago
That passage also raised the hair on the back of my neck ... I was trying to ignore it when I came across this phrase: "Our restoration process is indeed tamper-proof." (and another fragment from their user documentation). Given the current climate in computer security, I think we should be much more careful about how we phrase our guarantees and more specifically how secure each service is. These statements imply guarantees that are simply impossible to make ... and we've seen flaws in our crypto tools change the security landscape overnight.

deleted 12y ago

[deleted]

sehrope 12y ago

> The old prototype machine had our AWS API access key and secret key. Once the hacker gained access to the keys, he created an IAM user, and generated a key-pair. Ouch! This is why you must practice the principle of least privilege when provisioning access keys, especially ones that can control your entire infrastructure. > He was then able to run an instance inside our AWS account using these credentials, and mount one of our backup disks. Good security practices are like an onion ... it's got many layers and will most likely make you cry. This is why you should be encrypting your backups. The backup program doesn't even need to be able to read the backups it creates, it only needs to to be able to write them (i.e. the public half of the key). If you're thinking, "Why does that matter? It can already read the plain-text data it's backing up!" then you're not thinking about history or multiple servers. Each may be able to read the current state of its own file system, but if they only have the public half, they can't read the historical state or each other's backups. The damage would be limited to a single server and only what's live on it at that moment. > We were able to verify the actions of the hacker using AWS CloudTrail, which confirmed that no other services were compromised, no other machines were booted, and our AMIs and other data stores were not copied. CloudTrail is awesome and if you're on AWS you really should enable it. It costs peanuts compared to what it provides. In cases like this, where the attacker takes over your entire infrastructure, it may be the only log you have of what happened.

travem 12y ago

See https://news.ycombinator.com/item?id=8594652 https://news.ycombinator.com/item?id=8594652 for more discussion on this.

Someone1234 12y ago

> Our database logs confirmed that user data was partially copied, but no user test history was compromised. Therefore all user data remains wholly intact. You what? You said hashed passwords and email addresses were taken for 5K users. What does "wholly intact" mean? > All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked. However, as an added precaution, we suggest that users change their BrowserStack account passwords. Those 5K passwords are compromised. It isn't an "added precaution" it is a mandatory requirement. The whole point of bcrypt is to buy you time to change your password, they have misunderstood that and think bcrypt is "unbreakable" so people don't need to change passwords. That's super ignorant. I would expire all 5K users (or ALL users) passwords immediately and send them recovery emails.

icelancer 12y ago
Expire all is easily the right decision.

general_failure 12y ago

Oh wow. Basically the whole infrastructure was compromised. > He began to copy one of our tables, which contained partial user information, including email IDs, hashed passwords, and last tested URL. His copy operation locked the database table, which raised alerts on our monitoring system. How does a copy operation lock a database table? Because you set thresholds on the number of reads? I find it a little surprising that such a knowledgeable hacker would make such a rookie mistake. Can someone clarify?

rememberlenny 12y ago
Copy operations regularly lock tables for large databases. I see that frequently for backpubs on our mysql db.

nodesocket 12y ago

I'd be curious to know the exact shellshock entry point into the original target server and how it was used to gain the AWS credentials on that server specifically.

nodesocket 12y ago

> We have a trace and the IP of the hacker. I doubt they have the hackers real IP. The hacker was almost certainly using a proxy right? Even if they have an IP address of the hacker, what is the likelihood of tracking it down to a physical person?

DougN7 12y ago

They said they had thousands of servers. Can that possibly be true? Is BrowserStack really that large?