6 ms·

Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on

by pevey 5y ago

Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on completion of the install it gives you a warning about this and tells you what to do to set a password. That is a small improvement. Most people would not have the service bound to a public interface, but for those who do for whatever reason have a set up where they are accessing it remotely, at least now there is a tip off that it is completely open to the world by default. This is different from pretty much any other service you might install. MariaDB for instance by default does not allow remote root login even if you change the config to bind it to 0.0.0.0. A lot of people are just totally unaware of this issue. When I read about database leakage, I generally assume 90% probability it was elastic, these days. Defaults are so important.

jsiaajdsdaa 5y ago
Yikes lol. I spin up tons of self hosted mysql boxes on public IPs but never would I ever expose 3306 or allow remote root lol. I just have a thin API in Java that feeds queries into it with parameterized statements
- technion 5y ago
  I think people would be surprised how often it's entirely deliberate. I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".
  - KronisLV 5y ago
    > I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops". I guess there is some merit to the zero trust security model, since it should urge you to secure every bit of software that you run, without necessarily setting up bastion hosts/DMZs or other types of "server-based ACLs" like that: https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model (since i've seen the fact that you have a "perimeter" get used as an excuse to get lazy about security aside from it) Yet, i've found that it's just too risky, given how brittle most of our software, including its security functionality, actually is. Just the other day i had even the simple blog software that i use (Grav) fail enforcing its admin login functionality, something that seems incredibly simple on the surface and should be bulletproof, yet somehow wasn't: https://blog.kronis.dev/everything%20is%20broken/grav-security-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-securi... Thus, personally i've found that what's worthy of consideration is taking a bit from both approaches. For example, running everything in containers with overlay networks (not even dedicated service meshes necessarily, even the way how Docker Swarm does it is pretty good: https://docs.docker.com/network/network-tutorial-overlay/ https://docs.docker.com/network/network-tutorial-overlay/) that can expose your apps/web servers on every node where necessary, yet keep your DBs constrained to running inside of the overlay network and thus remain inaccessible from the outside. If and when you need to connect them, maybe run a container with SSH and WireGuard/OpenVPN that let's you forward ports and thus get access to the overlay network, or alternatively just use orchestrator functionality (for example, Kubernetes allows this, albeit i dislike its complexity otherwise) for this.
    - pnutjam 5y ago
      I failed hard on an edit to pamd, which resulted in any password being accepted for any user. Anything but no password. Not something I would usually test for, but lesson learned. Thank his noodliness we don't have ssh exposed anywhere outside our network.
  - throwaway_3101 5y ago
    I have stopped trying to be the "adult" in our company. I admit that I don't know everything, but two decades of experience has given me some insights. But when I make a suggestion with examples to back them up, my experience is often used against me. I have learned to let it go and allow them to fail on their own. I'm not going to work there for long anyway
    - Foobar8568 5y ago
      So much of this, a company I worked for was like that, I brought up that credit cards were not encrypted/that the monitoring was everywhere but not on critical systems, that dev were using backdoors ( if dev return admin basically) etc, nothing changed. Recently they reported a data leak, no shit...
      - onemoresoop 5y ago
        I think dataleaks aren't punished hard enough. If they did and stung the wallet really hard the industry would quickly adapt to better security requirements.
        toss1 5y ago
        I'm no so sure -- many managers would just see it as an expeditious cost of doing business, unless it got to kill-the-company-now levels of fines. I was once talking with an FCC agent about the status of some boards we were importing, and he told me about how Michael Dell would blatantly violate FCC regs with their computers, fight it for a while, then pay the fine out of "Federal Regulatory Account" -- he just happily violated the law and paid the fines enough to have a specific bank acct for that. He was not amused. I'm pretty sure that it would require a demonstrably real threat of jail all the way up the management chain -- that might focus their minds a bit
        onemoresoop 5y ago
        No jail, crippling fines that if the behavior doesn't change would wipe them. One example would be sufficient and the industry would fix itself up.
- noduerme 5y ago
  Like you, I've got lots of RDS instances (mysql). Generally they're accessible only to certain EC2 instances, so I just SSH into the appropriate EC2 to access the database. There are sometimes cases where it's expedient to expose them, like if I need to quickly test some backend code remotely before deploying it. I keep a security group that allows inbound on 0.0.0.0 :3306 and just assign it if I need to for a couple minutes. Even if it's only open for a few minutes, there will be logs of malicious actors trying to brute force the DB's password. So how this comes as a surprise to admins is beyond me, but I haven't played with Elasticsearch at all.
  - unilynx 5y ago
    If you’re already ssh-ing to the database, why not tunnel the db connection over ssh? It would probably be less effort than updating security rules each time
    - suifbwish 5y ago
      Ssh tunnels are so badass. I’ve heard of people performing encrypted db replication through them between data centers
      - deberon 5y ago
        I’ve witnessed somebody piping dd over an ssh tunnel to write blocks from one machine directly to another. Highly NOT recommended but still one of the neatest things I’ve seen. It worked.
        drewzero1 5y ago
        Sounds like a neat party trick! (At least, if you've got a really stable connection.) Did they do an md5sum afterward, or how did they know it worked?
        suifbwish 5y ago
        That sort of thing can be used for making disk images. Restoring them would be the only way to verify them as checking the md5sum of the file is not possible without a reference backup and if you have a reference backup why would you be using dd through a pipe lol. Definitely a cool trick though.
- samstave 5y ago
  I mean, especially on AWS, controlling access to 3306 is dead simple. and literally the one thing that anyone running a DB (which should be everyone) does to ensure that only certain machines, vlans etc can talk to 3306 on said DB instances... Like literally ONLY this internal IP is allowed to contact the DB. Also. as far back as 2014 we had AWS scanning all our machines constantly for open ports and alerts on such - we had to get an approval from within AWS to continue to portscan our own machines... they were pretty on top of port scanning behavior... and I am surprised that the stance they take hasnt improved in all these years.
colechristensen 5y ago
I’m pretty sure amazon now scans for and warns you if you’re running an open elastic service on the public internet.
Sebguer 5y ago
Mongo for years defaulted to being open to the internet, with no authentication.
- patrec 5y ago
  And it's probably the main reason MongoDB is a thing at all.
- gqewogpdqa 5y ago
  As, surprisingly, did so many other databases and operating systems. VAX/VMS was "FIELD/SERVICE" on all new systems for many years.Then all these systems got connected to DECNet/BITNET, etc and all heck broke loose. No real excuse for anybody to have ever configured their systems this way.
- new_guy 5y ago
  When I learnt Mongo, that was literally the first thing I had to lookup because the docs didn't explain it, or even mention it was insecure. But once it was set up it gives you a login, and I realized it was completely public, anyone could just login there. It's a horrible default to have, but it's weird more people didn't realise how insecure it was.
  - MaBeuLux88 5y ago
    This doc page has been up for as long as I can remember: https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec.... It's also stated in the production notes: https://docs.mongodb.com/manual/administration/production-notes/#networking https://docs.mongodb.com/manual/administration/production-no....
nullify88 5y ago
Re Elasticsearch. It used to be much worse. Even basic authentication was only available to those with an xpack subscription, it didnt have any security unless you paid money or used Nginx or another server to apply the auth and proxy the connection. That didn't stop people from making instances available publicly, and it was only until Elasticsearch started hitting the news about all the open instances did they make it available in the free tier from May 2019 (6.8.0 / 7.1.0)
Cthulhu_ 5y ago
Honestly, a lot of these services should be secure and closed by default; they make them open by default to onboard people more easily when they're in the POC phase of their project. But there's nothing as permanent as a temporary solution.

by pevey 5y ago

jsiaajdsdaa 5y ago

Yikes lol. I spin up tons of self hosted mysql boxes on public IPs but never would I ever expose 3306 or allow remote root lol. I just have a thin API in Java that feeds queries into it with parameterized statements

technion 5y ago
I think people would be surprised how often it's entirely deliberate. I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".
- KronisLV 5y ago
  > I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops". I guess there is some merit to the zero trust security model, since it should urge you to secure every bit of software that you run, without necessarily setting up bastion hosts/DMZs or other types of "server-based ACLs" like that: https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model (since i've seen the fact that you have a "perimeter" get used as an excuse to get lazy about security aside from it) Yet, i've found that it's just too risky, given how brittle most of our software, including its security functionality, actually is. Just the other day i had even the simple blog software that i use (Grav) fail enforcing its admin login functionality, something that seems incredibly simple on the surface and should be bulletproof, yet somehow wasn't: https://blog.kronis.dev/everything%20is%20broken/grav-security-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-securi... Thus, personally i've found that what's worthy of consideration is taking a bit from both approaches. For example, running everything in containers with overlay networks (not even dedicated service meshes necessarily, even the way how Docker Swarm does it is pretty good: https://docs.docker.com/network/network-tutorial-overlay/ https://docs.docker.com/network/network-tutorial-overlay/) that can expose your apps/web servers on every node where necessary, yet keep your DBs constrained to running inside of the overlay network and thus remain inaccessible from the outside. If and when you need to connect them, maybe run a container with SSH and WireGuard/OpenVPN that let's you forward ports and thus get access to the overlay network, or alternatively just use orchestrator functionality (for example, Kubernetes allows this, albeit i dislike its complexity otherwise) for this.
  - pnutjam 5y ago
    I failed hard on an edit to pamd, which resulted in any password being accepted for any user. Anything but no password. Not something I would usually test for, but lesson learned. Thank his noodliness we don't have ssh exposed anywhere outside our network.
- throwaway_3101 5y ago
  I have stopped trying to be the "adult" in our company. I admit that I don't know everything, but two decades of experience has given me some insights. But when I make a suggestion with examples to back them up, my experience is often used against me. I have learned to let it go and allow them to fail on their own. I'm not going to work there for long anyway
  - Foobar8568 5y ago
    So much of this, a company I worked for was like that, I brought up that credit cards were not encrypted/that the monitoring was everywhere but not on critical systems, that dev were using backdoors ( if dev return admin basically) etc, nothing changed. Recently they reported a data leak, no shit...
    - onemoresoop 5y ago
      I think dataleaks aren't punished hard enough. If they did and stung the wallet really hard the industry would quickly adapt to better security requirements.
      - toss1 5y ago
        I'm no so sure -- many managers would just see it as an expeditious cost of doing business, unless it got to kill-the-company-now levels of fines. I was once talking with an FCC agent about the status of some boards we were importing, and he told me about how Michael Dell would blatantly violate FCC regs with their computers, fight it for a while, then pay the fine out of "Federal Regulatory Account" -- he just happily violated the law and paid the fines enough to have a specific bank acct for that. He was not amused. I'm pretty sure that it would require a demonstrably real threat of jail all the way up the management chain -- that might focus their minds a bit
        onemoresoop 5y ago
        No jail, crippling fines that if the behavior doesn't change would wipe them. One example would be sufficient and the industry would fix itself up.
noduerme 5y ago
Like you, I've got lots of RDS instances (mysql). Generally they're accessible only to certain EC2 instances, so I just SSH into the appropriate EC2 to access the database. There are sometimes cases where it's expedient to expose them, like if I need to quickly test some backend code remotely before deploying it. I keep a security group that allows inbound on 0.0.0.0 :3306 and just assign it if I need to for a couple minutes. Even if it's only open for a few minutes, there will be logs of malicious actors trying to brute force the DB's password. So how this comes as a surprise to admins is beyond me, but I haven't played with Elasticsearch at all.
- unilynx 5y ago
  If you’re already ssh-ing to the database, why not tunnel the db connection over ssh? It would probably be less effort than updating security rules each time
  - suifbwish 5y ago
    Ssh tunnels are so badass. I’ve heard of people performing encrypted db replication through them between data centers
    - deberon 5y ago
      I’ve witnessed somebody piping dd over an ssh tunnel to write blocks from one machine directly to another. Highly NOT recommended but still one of the neatest things I’ve seen. It worked.
      - drewzero1 5y ago
        Sounds like a neat party trick! (At least, if you've got a really stable connection.) Did they do an md5sum afterward, or how did they know it worked?
      - suifbwish 5y ago
        That sort of thing can be used for making disk images. Restoring them would be the only way to verify them as checking the md5sum of the file is not possible without a reference backup and if you have a reference backup why would you be using dd through a pipe lol. Definitely a cool trick though.
samstave 5y ago
I mean, especially on AWS, controlling access to 3306 is dead simple. and literally the one thing that anyone running a DB (which should be everyone) does to ensure that only certain machines, vlans etc can talk to 3306 on said DB instances... Like literally ONLY this internal IP is allowed to contact the DB. Also. as far back as 2014 we had AWS scanning all our machines constantly for open ports and alerts on such - we had to get an approval from within AWS to continue to portscan our own machines... they were pretty on top of port scanning behavior... and I am surprised that the stance they take hasnt improved in all these years.

colechristensen 5y ago

I’m pretty sure amazon now scans for and warns you if you’re running an open elastic service on the public internet.

Sebguer 5y ago

Mongo for years defaulted to being open to the internet, with no authentication.

patrec 5y ago
And it's probably the main reason MongoDB is a thing at all.
gqewogpdqa 5y ago
As, surprisingly, did so many other databases and operating systems. VAX/VMS was "FIELD/SERVICE" on all new systems for many years.Then all these systems got connected to DECNet/BITNET, etc and all heck broke loose. No real excuse for anybody to have ever configured their systems this way.
new_guy 5y ago
When I learnt Mongo, that was literally the first thing I had to lookup because the docs didn't explain it, or even mention it was insecure. But once it was set up it gives you a login, and I realized it was completely public, anyone could just login there. It's a horrible default to have, but it's weird more people didn't realise how insecure it was.
- MaBeuLux88 5y ago
  This doc page has been up for as long as I can remember: https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec.... It's also stated in the production notes: https://docs.mongodb.com/manual/administration/production-notes/#networking https://docs.mongodb.com/manual/administration/production-no....

nullify88 5y ago

Re Elasticsearch. It used to be much worse. Even basic authentication was only available to those with an xpack subscription, it didnt have any security unless you paid money or used Nginx or another server to apply the auth and proxy the connection. That didn't stop people from making instances available publicly, and it was only until Elasticsearch started hitting the news about all the open instances did they make it available in the free tier from May 2019 (6.8.0 / 7.1.0)

Cthulhu_ 5y ago

Honestly, a lot of these services should be secure and closed by default; they make them open by default to onboard people more easily when they're in the POC phase of their project. But there's nothing as permanent as a temporary solution.