8 ms·

I discovered thousands of open databases on AWS

199 points by fagnerbrack 5y ago

jack335 5y ago
Another great example why I love to use managed services / serverless services and not take care of security groups and NACLs
- jfkimmes 5y ago
  Interesting. The conclusion you draw from this differs wildly from my conclusion. Whenever I read stories like these, it seems clear to me that someone moved to the cloud in order to not have to care about security. The 'cloud does everything for you!'. Just like you imply in your answer that PaaS, the next level of abstraction, will solve all your security problems. This move, however, will inevitably lead to a situation where people work with new and complex systems that they don't understand (remember: not having to understand them is the sole reason they use them). Unfortunately, working with complex systems you don't understand is the number one reason for vulnerabilities in the first place. I am not convinced that a service exists that abstracts security away from you.
  - sofixa 5y ago
    There's another side to this. The people deploying things with a security group open to the internet and no auth might have been saved by a grumpy network admin forcing them to use a VPN or even a firewall, but that's IMHO only hiding the real problem ( security not taken seriously) behind a thin fence. When that fence gets breached, everything would be up for grabs, so the main difference with "the cloud" is that such terrible security postures are easier and faster to spot ( from both sides). Zero trust everything.
  - BrandoElFollito 5y ago
    It depends which kind of security and which provider. I would trust Microsoft more on patches/configurations for an email system than something which is managed on premises. You need to have really good people to maintain a good level of security. Not only technically good, but also with a string cold management that will force updates even if it means the CEO will not get his maol for 15 minutes - and say that this is life and that the discussion i sover. On top of that, MS would (I hope) install patches on their customer-facing systems in advance of an official patch release. The above applies to the majority of large SaaS services. Now when you have a "Platform", a hoster that requires you to bring in knowledge and not only data then it gets dangerous. You need to maintain the security of what you bring in. This can be an OS (your "Platform" provides VMs), or code (your "Platform" provides code runners). Unfortunately, when a company moves to the cloud, they sometimes forget to do this assessment and end up with monstrosities they installed themselves (which is not different, security wise, from having it on premises - augmente nu the 7B population that potentially has now access)
  - notwedtm 5y ago
    I think the word "managed" is the clear differentiator here. There is a huge difference between setting up ElasticSearch on some EC2 instances yourself, and paying ElasticCo for a managed cluster. I would expect the latter to be sure by default.
- ThrashBeard 5y ago
  Managed services don't save you from that. You still have to configure security groups to reach them and a lot of devs just use the easiest way and expose the thing to the internet.
pevey 5y ago
This same article was posted previously, so I will post the same response I posted on the other thread: Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on completion of the install it gives you a warning about this and tells you what to do to set a password. That is a small improvement. Most people would not have the service bound to a public interface, but for those who do for whatever reason have a set up where they are accessing it remotely, at least now there is a tip off that it is completely open to the world by default. This is different from pretty much any other service you might install. MariaDB for instance by default does not allow remote root login even if you change the config to bind it to 0.0.0.0. A lot of people are just totally unaware of this issue. When I read about database leakage, I generally assume 90% probability it was elastic, these days. Defaults are so important.
- XorNot 5y ago
  My solution to this has always been to auto-generate a password and dump it in a by file. If it's a test instance then this is a nice obvious place to go get it from. If it's not then the service isn't open by default, and in both cases we're not asking people to enable remote access themselves by some scheme which may allow them to do it without setting a password. Same thing with TLS IMO: let certs be specified, or default to generating some - any option is better then default no encryption.
- bryans 5y ago
  There are lots of lessons in the internet's limited history that show poorly conceived defaults are the most common and dangerous mistake. Microsoft let Frontpage Extension credentials default to the domain name and a blank password through multiple major versions, without any prompt to change anything (you had to seek out the settings, and it wasn't even obvious where they were). Throughout the late 90s and early 00s, entire hosting companies were continuously compromised because of this, along with nearly every school district in the US and tons of major universities. Schools typically had the worst security and most available processing power and bandwidth, already making them the preferred targets of script kiddies everywhere. That exploit was single-handedly responsible for hundreds of thousands of bots (and more likely millions in later years) which wreaked havoc on the internet as a whole -- these were the days when a handful of bots on good connections could take down the Amazons of the world for lengthy periods (days, not hours). That they let it go on for so long was comical at the time, but in hindsight seems criminally negligent.
helsinkiandrew 5y ago
Previous comments: https://news.ycombinator.com/item?id=30144021 https://news.ycombinator.com/item?id=30144021
IncRnd 5y ago
This has been going on for years with AWS. Unfortunately, there has always been unprotected data sitting on AWS for anyone to read. This Google Search returned an extremely high number of mentions of various companies' ills, a high signal to noise ratio. It's very saddening to see that. https://www.google.com/search?q=unprotected+databases+on+aws https://www.google.com/search?q=unprotected+databases+on+aws
jeroenhd 5y ago
I used to think you have to be pretty good af your job to get trusted to deploy stuff to the cloud for even medium sized companies. As these articles keep reminding me, you only need to fake competence to management to get the green light. Even if you forget that the cloud is the internet and that the entire internet can reach you over the internet, it doesn't take a genius to set up a password for a cloud service. I have no idea how many of these databases are honey pots, but finding open and vulnerable servers is depressingly simple. I suppose it's kind of liberating to know that you can be dumb enough to fling patient data into an unprotected cloud server and still get a job in IT. The bar is really set that low.
- mettamage 5y ago
  > As these articles keep reminding me, you only need to fake competence to management to get the green light. Well, managers aren't scientists looking for truth. They are people running businesses that need to stay afloat and get stuff done. There's a lot of wiggle room there, mostly due to time pressure (from a charitable perspective, less charitable: also because of company politics).
- mushyhammer 5y ago
  > you only need to fake competence to management With so many 10x full-stack developers who're just looking to get sh done and clock out, it can't be otherwise. If managing a server becomes just another bullet point in a developer job, this is exactly what we get. Sure I can deploy stuff, but you shouldn't trust me. Unfortunately many are not so honest or even aware of the complexity (unknown unknowns).
  - mtberatwork 5y ago
    > If managing a server becomes just another bullet point in a developer job, this is exactly what we get. Yup...the same management are the ones that laid off the DBAs, system admins, network engineers, etc because "the cloud" hand waves all this stuff away now.
    - 5y ago
rmetzler 5y ago
> DevOps, Developers, and IT practitioners often misconfigure some of the following: Binding the socket on the wrong network interfaces. For example, listening to connections from 0.0.0.0/* — So it is visible to all network interfaces, instead of only the inner-network interface IP address (172.x.x.x) Binding to 0.0.0.0 is unfortunately the default for Docker. I wish it would have been different.
- kawsper 5y ago
  Even if you've firewalled you machine, Docker will happily open up the ports to the world, you have to run it with "--iptables=false" to stop it. I accidentally opened up a Redis test-instance to the world like this.
  - qwertox 5y ago
    Is there any additional information on this? Assuming I add a rule via `iptables -I INPUT -s 123.123.123.123 -p tcp --dport 8080 -j DROP` and on interface:port 0.0.0.0:8080 of the host is a container listening which got run as `docker run -d -p 0.0.0.0:8080:8080 some/server:latest` Will that container still be accessible to 123.123.123.132?
    - l33tman 5y ago
      I think so, this has been discussed in many HN threads and it always surprises a lot of people how big of a hole this is. I was certainly not aware of it the first time I read about it. Docker inserts its own iptables rules before your drop rule I think to handle its virtual interfaces.
    - malka 5y ago
      Yes. Docker will create specific rules for the lifetime of the container. I was bit by this once. Be very careful with docker. Do not rely blindly on your firewall of it runs on the same machine.
    - remram 5y ago
      The Docker container is using a separate network namespace with a separate virtual interface that packets are routed to. The "PREROUTING" chain runs before the "INPUT" chain so yes, the rules you put in "INPUT" won't apply.
qxmat 5y ago
Network layering is still important on the cloud but many DevOps appear to miss that. Logical cloud protection layers - like security groups - are an additional security layer but not a replacement for proper route design.
- orev 5y ago
  So many have been sold on cloud using the idea that they don’t need to worry about this stuff, because the big cloud provider has better security teams, etc. It’s not surprising at all that you end up with issues like this — any Ops person could’ve seen it coming from miles away.
k8sToGo 5y ago
That is why you configure your security groups properly.
l33tman 5y ago
From the article its main focus vulnerability seems to be setups that don't use VPCs, which is default on AWS since long. It discusses human oversights like a server that needs to accept connections from other servers and where they then have opened a too large IP block. But this only makes sense if they were using internet IPs.. within a VPC this wouldn't matter. With some even rudimentary knowledge of how IPV4 works this wouldn't be an issue on AWS, there are good defaults and tools to help avoiding this, no matter if you run Dockerized databases or bare metal dbs or use RDS.. you really have to explicitly open up your servers to end up open on the net..
- nickjj 5y ago
  I think in a lot of cases folks still make certain things public instead of using a VPC out of convenience. For example if a developer needs to access a resource within a VPC then this sky rockets complexity. You can't just connect to something running in a VPC from the outside world. You'd have to install and configure a VPN and it becomes a whole ceremony for both the person setting all of this up and the person connecting. Using a VPC/VPN and keeping your important things internal (like a database or staging environment, etc.) is a really good idea but unless you have someone who has a really solid understanding of this it's a whole lot easier to make something public, restrict a security group by IP and keep a whitelist of IPs up to date for developer access.
  - wbkang 5y ago
    How about just setting up a bastion host and then forwarding ports via SSH? It's much simpler than setting up a VPN.
fredsmith219 5y ago
It's like the days of sa/no password all over again. Made easier because of the published CIDR blocks.
- remram 5y ago
  If you are not using Python, scanning large swaths of IP addresses or even the whole IPv4 space for open port 9200 is very doable.
  - sethammons 5y ago
    What's the problem with Python in this situation? Concurrency and Gil locks?
    - remram 5y ago
      I read a little fast, he didn't scan with Python, he just used Python to analyze the results. My bad.
rr808 5y ago
One thing I've been surprised with is how difficult it is to set up database connectivity. I thought you'd just connect to a private VPC but it makes connections and management difficult. Much easier to go with external IP and rely on firewall rules - though then you risk ending up open.
bokohut 5y ago
Many years ago it was BBS systems and then as Windows took off it became SMB shares, fun times for sure. It then transitioned to databases open and exposed to the WAN and once those admins became educated the targets were RDP clients that were poorly designed, radmin anyone? Everything to this point led to the Payment Card Industry (P.C.I.) founding as losses accrued and blame assignment for cost absorption was needed. As APIs took off, where we are now, the era of data vacuuming everything insecure continues inclusive of AWS. The future however will redefine what it means to build secure software from day zero as IoT devices that control life and property will continue to grow in popularity as well as attack. Industrial, commercial and residential targets grow by the day such as SCADA, lighting control, hot water tanks, faucets, toilets and all the other devices that will become smart in one's life but I would be remiss to not mention the most important items that will enable remote life manipulation, medical devices. What you cannot see matters most and as connectivity of everything grows exponentially one is likely to not even see it coming. Stay Healthy!

199 points by fagnerbrack 5y ago

Another great example why I love to use managed services / serverless services and not take care of security groups and NACLs

jfkimmes 5y ago
Interesting. The conclusion you draw from this differs wildly from my conclusion. Whenever I read stories like these, it seems clear to me that someone moved to the cloud in order to not have to care about security. The 'cloud does everything for you!'. Just like you imply in your answer that PaaS, the next level of abstraction, will solve all your security problems. This move, however, will inevitably lead to a situation where people work with new and complex systems that they don't understand (remember: not having to understand them is the sole reason they use them). Unfortunately, working with complex systems you don't understand is the number one reason for vulnerabilities in the first place. I am not convinced that a service exists that abstracts security away from you.
- sofixa 5y ago
  There's another side to this. The people deploying things with a security group open to the internet and no auth might have been saved by a grumpy network admin forcing them to use a VPN or even a firewall, but that's IMHO only hiding the real problem ( security not taken seriously) behind a thin fence. When that fence gets breached, everything would be up for grabs, so the main difference with "the cloud" is that such terrible security postures are easier and faster to spot ( from both sides). Zero trust everything.
- BrandoElFollito 5y ago
  It depends which kind of security and which provider. I would trust Microsoft more on patches/configurations for an email system than something which is managed on premises. You need to have really good people to maintain a good level of security. Not only technically good, but also with a string cold management that will force updates even if it means the CEO will not get his maol for 15 minutes - and say that this is life and that the discussion i sover. On top of that, MS would (I hope) install patches on their customer-facing systems in advance of an official patch release. The above applies to the majority of large SaaS services. Now when you have a "Platform", a hoster that requires you to bring in knowledge and not only data then it gets dangerous. You need to maintain the security of what you bring in. This can be an OS (your "Platform" provides VMs), or code (your "Platform" provides code runners). Unfortunately, when a company moves to the cloud, they sometimes forget to do this assessment and end up with monstrosities they installed themselves (which is not different, security wise, from having it on premises - augmente nu the 7B population that potentially has now access)
- notwedtm 5y ago
  I think the word "managed" is the clear differentiator here. There is a huge difference between setting up ElasticSearch on some EC2 instances yourself, and paying ElasticCo for a managed cluster. I would expect the latter to be sure by default.
ThrashBeard 5y ago
Managed services don't save you from that. You still have to configure security groups to reach them and a lot of devs just use the easiest way and expose the thing to the internet.

pevey 5y ago

This same article was posted previously, so I will post the same response I posted on the other thread: Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on completion of the install it gives you a warning about this and tells you what to do to set a password. That is a small improvement. Most people would not have the service bound to a public interface, but for those who do for whatever reason have a set up where they are accessing it remotely, at least now there is a tip off that it is completely open to the world by default. This is different from pretty much any other service you might install. MariaDB for instance by default does not allow remote root login even if you change the config to bind it to 0.0.0.0. A lot of people are just totally unaware of this issue. When I read about database leakage, I generally assume 90% probability it was elastic, these days. Defaults are so important.

XorNot 5y ago
My solution to this has always been to auto-generate a password and dump it in a by file. If it's a test instance then this is a nice obvious place to go get it from. If it's not then the service isn't open by default, and in both cases we're not asking people to enable remote access themselves by some scheme which may allow them to do it without setting a password. Same thing with TLS IMO: let certs be specified, or default to generating some - any option is better then default no encryption.
bryans 5y ago
There are lots of lessons in the internet's limited history that show poorly conceived defaults are the most common and dangerous mistake. Microsoft let Frontpage Extension credentials default to the domain name and a blank password through multiple major versions, without any prompt to change anything (you had to seek out the settings, and it wasn't even obvious where they were). Throughout the late 90s and early 00s, entire hosting companies were continuously compromised because of this, along with nearly every school district in the US and tons of major universities. Schools typically had the worst security and most available processing power and bandwidth, already making them the preferred targets of script kiddies everywhere. That exploit was single-handedly responsible for hundreds of thousands of bots (and more likely millions in later years) which wreaked havoc on the internet as a whole -- these were the days when a handful of bots on good connections could take down the Amazons of the world for lengthy periods (days, not hours). That they let it go on for so long was comical at the time, but in hindsight seems criminally negligent.

helsinkiandrew 5y ago

Previous comments: https://news.ycombinator.com/item?id=30144021 https://news.ycombinator.com/item?id=30144021

IncRnd 5y ago

This has been going on for years with AWS. Unfortunately, there has always been unprotected data sitting on AWS for anyone to read. This Google Search returned an extremely high number of mentions of various companies' ills, a high signal to noise ratio. It's very saddening to see that. https://www.google.com/search?q=unprotected+databases+on+aws https://www.google.com/search?q=unprotected+databases+on+aws

jeroenhd 5y ago

I used to think you have to be pretty good af your job to get trusted to deploy stuff to the cloud for even medium sized companies. As these articles keep reminding me, you only need to fake competence to management to get the green light. Even if you forget that the cloud is the internet and that the entire internet can reach you over the internet, it doesn't take a genius to set up a password for a cloud service. I have no idea how many of these databases are honey pots, but finding open and vulnerable servers is depressingly simple. I suppose it's kind of liberating to know that you can be dumb enough to fling patient data into an unprotected cloud server and still get a job in IT. The bar is really set that low.

mettamage 5y ago
> As these articles keep reminding me, you only need to fake competence to management to get the green light. Well, managers aren't scientists looking for truth. They are people running businesses that need to stay afloat and get stuff done. There's a lot of wiggle room there, mostly due to time pressure (from a charitable perspective, less charitable: also because of company politics).
mushyhammer 5y ago
> you only need to fake competence to management With so many 10x full-stack developers who're just looking to get sh done and clock out, it can't be otherwise. If managing a server becomes just another bullet point in a developer job, this is exactly what we get. Sure I can deploy stuff, but you shouldn't trust me. Unfortunately many are not so honest or even aware of the complexity (unknown unknowns).
- mtberatwork 5y ago
  > If managing a server becomes just another bullet point in a developer job, this is exactly what we get. Yup...the same management are the ones that laid off the DBAs, system admins, network engineers, etc because "the cloud" hand waves all this stuff away now.
  - 5y ago

rmetzler 5y ago

> DevOps, Developers, and IT practitioners often misconfigure some of the following: Binding the socket on the wrong network interfaces. For example, listening to connections from 0.0.0.0/* — So it is visible to all network interfaces, instead of only the inner-network interface IP address (172.x.x.x) Binding to 0.0.0.0 is unfortunately the default for Docker. I wish it would have been different.

kawsper 5y ago
Even if you've firewalled you machine, Docker will happily open up the ports to the world, you have to run it with "--iptables=false" to stop it. I accidentally opened up a Redis test-instance to the world like this.
- qwertox 5y ago
  Is there any additional information on this? Assuming I add a rule via `iptables -I INPUT -s 123.123.123.123 -p tcp --dport 8080 -j DROP` and on interface:port 0.0.0.0:8080 of the host is a container listening which got run as `docker run -d -p 0.0.0.0:8080:8080 some/server:latest` Will that container still be accessible to 123.123.123.132?
  - l33tman 5y ago
    I think so, this has been discussed in many HN threads and it always surprises a lot of people how big of a hole this is. I was certainly not aware of it the first time I read about it. Docker inserts its own iptables rules before your drop rule I think to handle its virtual interfaces.
  - malka 5y ago
    Yes. Docker will create specific rules for the lifetime of the container. I was bit by this once. Be very careful with docker. Do not rely blindly on your firewall of it runs on the same machine.
  - remram 5y ago
    The Docker container is using a separate network namespace with a separate virtual interface that packets are routed to. The "PREROUTING" chain runs before the "INPUT" chain so yes, the rules you put in "INPUT" won't apply.

qxmat 5y ago

Network layering is still important on the cloud but many DevOps appear to miss that. Logical cloud protection layers - like security groups - are an additional security layer but not a replacement for proper route design.

orev 5y ago
So many have been sold on cloud using the idea that they don’t need to worry about this stuff, because the big cloud provider has better security teams, etc. It’s not surprising at all that you end up with issues like this — any Ops person could’ve seen it coming from miles away.

k8sToGo 5y ago

That is why you configure your security groups properly.

l33tman 5y ago

From the article its main focus vulnerability seems to be setups that don't use VPCs, which is default on AWS since long. It discusses human oversights like a server that needs to accept connections from other servers and where they then have opened a too large IP block. But this only makes sense if they were using internet IPs.. within a VPC this wouldn't matter. With some even rudimentary knowledge of how IPV4 works this wouldn't be an issue on AWS, there are good defaults and tools to help avoiding this, no matter if you run Dockerized databases or bare metal dbs or use RDS.. you really have to explicitly open up your servers to end up open on the net..

nickjj 5y ago
I think in a lot of cases folks still make certain things public instead of using a VPC out of convenience. For example if a developer needs to access a resource within a VPC then this sky rockets complexity. You can't just connect to something running in a VPC from the outside world. You'd have to install and configure a VPN and it becomes a whole ceremony for both the person setting all of this up and the person connecting. Using a VPC/VPN and keeping your important things internal (like a database or staging environment, etc.) is a really good idea but unless you have someone who has a really solid understanding of this it's a whole lot easier to make something public, restrict a security group by IP and keep a whitelist of IPs up to date for developer access.
- wbkang 5y ago
  How about just setting up a bastion host and then forwarding ports via SSH? It's much simpler than setting up a VPN.

fredsmith219 5y ago

It's like the days of sa/no password all over again. Made easier because of the published CIDR blocks.

remram 5y ago
If you are not using Python, scanning large swaths of IP addresses or even the whole IPv4 space for open port 9200 is very doable.
- sethammons 5y ago
  What's the problem with Python in this situation? Concurrency and Gil locks?
  - remram 5y ago
    I read a little fast, he didn't scan with Python, he just used Python to analyze the results. My bad.

rr808 5y ago

One thing I've been surprised with is how difficult it is to set up database connectivity. I thought you'd just connect to a private VPC but it makes connections and management difficult. Much easier to go with external IP and rely on firewall rules - though then you risk ending up open.

bokohut 5y ago

Many years ago it was BBS systems and then as Windows took off it became SMB shares, fun times for sure. It then transitioned to databases open and exposed to the WAN and once those admins became educated the targets were RDP clients that were poorly designed, radmin anyone? Everything to this point led to the Payment Card Industry (P.C.I.) founding as losses accrued and blame assignment for cost absorption was needed. As APIs took off, where we are now, the era of data vacuuming everything insecure continues inclusive of AWS. The future however will redefine what it means to build secure software from day zero as IoT devices that control life and property will continue to grow in popularity as well as attack. Industrial, commercial and residential targets grow by the day such as SCADA, lighting control, hot water tanks, faucets, toilets and all the other devices that will become smart in one's life but I would be remiss to not mention the most important items that will enable remote life manipulation, medical devices. What you cannot see matters most and as connectivity of everything grows exponentially one is likely to not even see it coming. Stay Healthy!