10 ms·

2 cents his hosts file points localhost to the prod db IP

pvorb 6y ago
Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.
- junon 6y ago
  That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively. Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.
  - pvorb 6y ago
    Why do you think they are hacks? Could you please elaborate? In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.
    - deleted 6y ago
      [deleted]
kjaftaedi 6y ago
I think so as well. It doesn't even make sense to connect to a managed database using 'localhost'. Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.
- drdaeman 6y ago
  That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`). It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)
dingdingdang 6y ago
Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.
- robryan 6y ago
  Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.
  - rwbhn 6y ago
    Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.
radu_floricica 6y ago
I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.
robryan 6y ago
One solution to this is to make sure only the production servers can connect to the production database.
buzer 6y ago
My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.

Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.

junon 6y ago
That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively. Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.
- pvorb 6y ago
  Why do you think they are hacks? Could you please elaborate? In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.
  - deleted 6y ago
    [deleted]

kjaftaedi 6y ago

I think so as well. It doesn't even make sense to connect to a managed database using 'localhost'. Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.

drdaeman 6y ago
That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`). It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)

dingdingdang 6y ago

Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.

robryan 6y ago
Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.
- rwbhn 6y ago
  Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.

radu_floricica 6y ago

I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.

robryan 6y ago

One solution to this is to make sure only the production servers can connect to the production database.

buzer 6y ago

My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.