6 ms·

Yikes lol. I spin up tons of self hosted mysql boxes on public IPs but never would I ever expose 3306 or allow remote root lol. I just have a thin API in Java t

Yikes lol. I spin up tons of self hosted mysql boxes on public IPs but never would I ever expose 3306 or allow remote root lol. I just have a thin API in Java that feeds queries into it with parameterized statements

technion 5y ago
I think people would be surprised how often it's entirely deliberate. I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".
- KronisLV 5y ago
  > I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops". I guess there is some merit to the zero trust security model, since it should urge you to secure every bit of software that you run, without necessarily setting up bastion hosts/DMZs or other types of "server-based ACLs" like that: https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model (since i've seen the fact that you have a "perimeter" get used as an excuse to get lazy about security aside from it) Yet, i've found that it's just too risky, given how brittle most of our software, including its security functionality, actually is. Just the other day i had even the simple blog software that i use (Grav) fail enforcing its admin login functionality, something that seems incredibly simple on the surface and should be bulletproof, yet somehow wasn't: https://blog.kronis.dev/everything%20is%20broken/grav-security-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-securi... Thus, personally i've found that what's worthy of consideration is taking a bit from both approaches. For example, running everything in containers with overlay networks (not even dedicated service meshes necessarily, even the way how Docker Swarm does it is pretty good: https://docs.docker.com/network/network-tutorial-overlay/ https://docs.docker.com/network/network-tutorial-overlay/) that can expose your apps/web servers on every node where necessary, yet keep your DBs constrained to running inside of the overlay network and thus remain inaccessible from the outside. If and when you need to connect them, maybe run a container with SSH and WireGuard/OpenVPN that let's you forward ports and thus get access to the overlay network, or alternatively just use orchestrator functionality (for example, Kubernetes allows this, albeit i dislike its complexity otherwise) for this.
  - pnutjam 5y ago
    I failed hard on an edit to pamd, which resulted in any password being accepted for any user. Anything but no password. Not something I would usually test for, but lesson learned. Thank his noodliness we don't have ssh exposed anywhere outside our network.
- throwaway_3101 5y ago
  I have stopped trying to be the "adult" in our company. I admit that I don't know everything, but two decades of experience has given me some insights. But when I make a suggestion with examples to back them up, my experience is often used against me. I have learned to let it go and allow them to fail on their own. I'm not going to work there for long anyway
  - Foobar8568 5y ago
    So much of this, a company I worked for was like that, I brought up that credit cards were not encrypted/that the monitoring was everywhere but not on critical systems, that dev were using backdoors ( if dev return admin basically) etc, nothing changed. Recently they reported a data leak, no shit...
    - onemoresoop 5y ago
      I think dataleaks aren't punished hard enough. If they did and stung the wallet really hard the industry would quickly adapt to better security requirements.
      - toss1 5y ago
        I'm no so sure -- many managers would just see it as an expeditious cost of doing business, unless it got to kill-the-company-now levels of fines. I was once talking with an FCC agent about the status of some boards we were importing, and he told me about how Michael Dell would blatantly violate FCC regs with their computers, fight it for a while, then pay the fine out of "Federal Regulatory Account" -- he just happily violated the law and paid the fines enough to have a specific bank acct for that. He was not amused. I'm pretty sure that it would require a demonstrably real threat of jail all the way up the management chain -- that might focus their minds a bit
        onemoresoop 5y ago
        No jail, crippling fines that if the behavior doesn't change would wipe them. One example would be sufficient and the industry would fix itself up.
noduerme 5y ago
Like you, I've got lots of RDS instances (mysql). Generally they're accessible only to certain EC2 instances, so I just SSH into the appropriate EC2 to access the database. There are sometimes cases where it's expedient to expose them, like if I need to quickly test some backend code remotely before deploying it. I keep a security group that allows inbound on 0.0.0.0 :3306 and just assign it if I need to for a couple minutes. Even if it's only open for a few minutes, there will be logs of malicious actors trying to brute force the DB's password. So how this comes as a surprise to admins is beyond me, but I haven't played with Elasticsearch at all.
- unilynx 5y ago
  If you’re already ssh-ing to the database, why not tunnel the db connection over ssh? It would probably be less effort than updating security rules each time
  - suifbwish 5y ago
    Ssh tunnels are so badass. I’ve heard of people performing encrypted db replication through them between data centers
    - deberon 5y ago
      I’ve witnessed somebody piping dd over an ssh tunnel to write blocks from one machine directly to another. Highly NOT recommended but still one of the neatest things I’ve seen. It worked.
      - drewzero1 5y ago
        Sounds like a neat party trick! (At least, if you've got a really stable connection.) Did they do an md5sum afterward, or how did they know it worked?
      - suifbwish 5y ago
        That sort of thing can be used for making disk images. Restoring them would be the only way to verify them as checking the md5sum of the file is not possible without a reference backup and if you have a reference backup why would you be using dd through a pipe lol. Definitely a cool trick though.
samstave 5y ago
I mean, especially on AWS, controlling access to 3306 is dead simple. and literally the one thing that anyone running a DB (which should be everyone) does to ensure that only certain machines, vlans etc can talk to 3306 on said DB instances... Like literally ONLY this internal IP is allowed to contact the DB. Also. as far back as 2014 we had AWS scanning all our machines constantly for open ports and alerts on such - we had to get an approval from within AWS to continue to portscan our own machines... they were pretty on top of port scanning behavior... and I am surprised that the stance they take hasnt improved in all these years.

technion 5y ago

I think people would be surprised how often it's entirely deliberate. I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops".

KronisLV 5y ago
> I've had to get told "this is why we used the cloud... so dinosaur sysadmins like you wouldn't slow us down" by many different companies where I've raised this, and I've had this debate in startup communities where "just open everything up" gets defended as "agile" and "modern devops". I guess there is some merit to the zero trust security model, since it should urge you to secure every bit of software that you run, without necessarily setting up bastion hosts/DMZs or other types of "server-based ACLs" like that: https://en.wikipedia.org/wiki/Zero_trust_security_model https://en.wikipedia.org/wiki/Zero_trust_security_model (since i've seen the fact that you have a "perimeter" get used as an excuse to get lazy about security aside from it) Yet, i've found that it's just too risky, given how brittle most of our software, including its security functionality, actually is. Just the other day i had even the simple blog software that i use (Grav) fail enforcing its admin login functionality, something that seems incredibly simple on the surface and should be bulletproof, yet somehow wasn't: https://blog.kronis.dev/everything%20is%20broken/grav-security-is-broken https://blog.kronis.dev/everything%20is%20broken/grav-securi... Thus, personally i've found that what's worthy of consideration is taking a bit from both approaches. For example, running everything in containers with overlay networks (not even dedicated service meshes necessarily, even the way how Docker Swarm does it is pretty good: https://docs.docker.com/network/network-tutorial-overlay/ https://docs.docker.com/network/network-tutorial-overlay/) that can expose your apps/web servers on every node where necessary, yet keep your DBs constrained to running inside of the overlay network and thus remain inaccessible from the outside. If and when you need to connect them, maybe run a container with SSH and WireGuard/OpenVPN that let's you forward ports and thus get access to the overlay network, or alternatively just use orchestrator functionality (for example, Kubernetes allows this, albeit i dislike its complexity otherwise) for this.
- pnutjam 5y ago
  I failed hard on an edit to pamd, which resulted in any password being accepted for any user. Anything but no password. Not something I would usually test for, but lesson learned. Thank his noodliness we don't have ssh exposed anywhere outside our network.
throwaway_3101 5y ago
I have stopped trying to be the "adult" in our company. I admit that I don't know everything, but two decades of experience has given me some insights. But when I make a suggestion with examples to back them up, my experience is often used against me. I have learned to let it go and allow them to fail on their own. I'm not going to work there for long anyway
- Foobar8568 5y ago
  So much of this, a company I worked for was like that, I brought up that credit cards were not encrypted/that the monitoring was everywhere but not on critical systems, that dev were using backdoors ( if dev return admin basically) etc, nothing changed. Recently they reported a data leak, no shit...
  - onemoresoop 5y ago
    I think dataleaks aren't punished hard enough. If they did and stung the wallet really hard the industry would quickly adapt to better security requirements.
    - toss1 5y ago
      I'm no so sure -- many managers would just see it as an expeditious cost of doing business, unless it got to kill-the-company-now levels of fines. I was once talking with an FCC agent about the status of some boards we were importing, and he told me about how Michael Dell would blatantly violate FCC regs with their computers, fight it for a while, then pay the fine out of "Federal Regulatory Account" -- he just happily violated the law and paid the fines enough to have a specific bank acct for that. He was not amused. I'm pretty sure that it would require a demonstrably real threat of jail all the way up the management chain -- that might focus their minds a bit
      - onemoresoop 5y ago
        No jail, crippling fines that if the behavior doesn't change would wipe them. One example would be sufficient and the industry would fix itself up.

noduerme 5y ago

Like you, I've got lots of RDS instances (mysql). Generally they're accessible only to certain EC2 instances, so I just SSH into the appropriate EC2 to access the database. There are sometimes cases where it's expedient to expose them, like if I need to quickly test some backend code remotely before deploying it. I keep a security group that allows inbound on 0.0.0.0 :3306 and just assign it if I need to for a couple minutes. Even if it's only open for a few minutes, there will be logs of malicious actors trying to brute force the DB's password. So how this comes as a surprise to admins is beyond me, but I haven't played with Elasticsearch at all.

unilynx 5y ago
If you’re already ssh-ing to the database, why not tunnel the db connection over ssh? It would probably be less effort than updating security rules each time
- suifbwish 5y ago
  Ssh tunnels are so badass. I’ve heard of people performing encrypted db replication through them between data centers
  - deberon 5y ago
    I’ve witnessed somebody piping dd over an ssh tunnel to write blocks from one machine directly to another. Highly NOT recommended but still one of the neatest things I’ve seen. It worked.
    - drewzero1 5y ago
      Sounds like a neat party trick! (At least, if you've got a really stable connection.) Did they do an md5sum afterward, or how did they know it worked?
    - suifbwish 5y ago
      That sort of thing can be used for making disk images. Restoring them would be the only way to verify them as checking the md5sum of the file is not possible without a reference backup and if you have a reference backup why would you be using dd through a pipe lol. Definitely a cool trick though.

samstave 5y ago

I mean, especially on AWS, controlling access to 3306 is dead simple. and literally the one thing that anyone running a DB (which should be everyone) does to ensure that only certain machines, vlans etc can talk to 3306 on said DB instances... Like literally ONLY this internal IP is allowed to contact the DB. Also. as far back as 2014 we had AWS scanning all our machines constantly for open ports and alerts on such - we had to get an approval from within AWS to continue to portscan our own machines... they were pretty on top of port scanning behavior... and I am surprised that the stance they take hasnt improved in all these years.