15 ms·

Small oversights get made in organizations of all sizes and often have large ramifications, as in this case. Good processes help minimize the exposure but the r

Small oversights get made in organizations of all sizes and often have large ramifications, as in this case. Good processes help minimize the exposure but the real world is too messy for them to act as guarantees. A security-conscious client recently noticed that due to an artifact of MySQL's (terrible) built-in replication, they'd been running production databases with passwordless accounts allowed to connect from any host. If someone on the internet found the hostname and guessed the right username, they'd have full access. This client has stringent controls in place to prevent just such eventualities, but this one slipped through. You can't be perfect. Computers are hard. The implications are frightening in this day and age, when so much of our business is conducted online, including highly-sensitive governmental business. The recent cyberattacks on servers hosting data regarding high-profile political figures will probably drive a program of mandatory licensure to work in IT as well as an even harsher version of the CFAA, both of which are very bad.

istsehrgut 10y ago
"security-conscious"
yeukhon 10y ago
> If someone on the internet found the hostname and guessed the right username, they'd have full access. This client has stringent controls in place to prevent just such eventualities, but this one slipped through. You can't be perfect. I don't know the details, but this sounds like someone is running MySQL on 0.0.0.0 interface, much like MongoDB's old default.
jlgaddis 10y ago
> This client has stringent controls in place to prevent just such eventualities, ... Like a firewall that allows anyone, anywhere in the world to connect to the common database ports? That is, they didn't have a firewall with a default deny policy? It doesn't at all like they had "stringent controls" in place, to me.
- cookiecaper 10y ago
  They do have a firewall with a default deny policy, but due to an overlooked configuration issue and a miscommunication on the network team, it failed to include this server (and this server only). "They should have auditors to make sure that doesn't happen!", you might say. Well, they do. That's how they caught it. No matter how much someone wants to pretend their process is perfect, the fact is that in practice, simple, embarrassing mistakes can and do happen. Facebook is a company packed with many competent software, network, security, and infrastructure engineers, no one can doubt that. I'm sure they have at least semi-formalized code review processes and a security checklist pre-deployment. Yet they still overlooked this pretty permissions issue which anyone who has ever developed a multi-user web app has encountered. It's easy to armchair quarterback it, but keep in mind, one day it may be your organization up on HN. Pretending like you're invincible is not productive for anything except your personal ego.

istsehrgut 10y ago

"security-conscious"

yeukhon 10y ago

> If someone on the internet found the hostname and guessed the right username, they'd have full access. This client has stringent controls in place to prevent just such eventualities, but this one slipped through. You can't be perfect. I don't know the details, but this sounds like someone is running MySQL on 0.0.0.0 interface, much like MongoDB's old default.

jlgaddis 10y ago

> This client has stringent controls in place to prevent just such eventualities, ... Like a firewall that allows anyone, anywhere in the world to connect to the common database ports? That is, they didn't have a firewall with a default deny policy? It doesn't at all like they had "stringent controls" in place, to me.

cookiecaper 10y ago
They do have a firewall with a default deny policy, but due to an overlooked configuration issue and a miscommunication on the network team, it failed to include this server (and this server only). "They should have auditors to make sure that doesn't happen!", you might say. Well, they do. That's how they caught it. No matter how much someone wants to pretend their process is perfect, the fact is that in practice, simple, embarrassing mistakes can and do happen. Facebook is a company packed with many competent software, network, security, and infrastructure engineers, no one can doubt that. I'm sure they have at least semi-formalized code review processes and a security checklist pre-deployment. Yet they still overlooked this pretty permissions issue which anyone who has ever developed a multi-user web app has encountered. It's easy to armchair quarterback it, but keep in mind, one day it may be your organization up on HN. Pretending like you're invincible is not productive for anything except your personal ego.