4 ms·

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use dif

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, just the common sense conclusion. Now it would be a good time to check those auditing logs / access logs you're suppose to have them enabled on said production machine.

toredash 6y ago
2 cents his hosts file points localhost to the prod db IP
- pvorb 6y ago
  Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.
  - junon 6y ago
    That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively. Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.
    - pvorb 6y ago
      Why do you think they are hacks? Could you please elaborate? In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.
      - deleted 6y ago
        [deleted]
- kjaftaedi 6y ago
  I think so as well. It doesn't even make sense to connect to a managed database using 'localhost'. Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.
  - drdaeman 6y ago
    That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`). It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)
- dingdingdang 6y ago
  Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.
  - robryan 6y ago
    Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.
    - rwbhn 6y ago
      Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.
- radu_floricica 6y ago
  I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.
- robryan 6y ago
  One solution to this is to make sure only the production servers can connect to the production database.
- buzer 6y ago
  My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.
ir123 6y ago
If managed DBs on DigitalOcean are anything like those on AWS, you can not run commands directly on them since SSH is prohibited. EDIT: there's also the deal with different credentials for dev and prod envs.
skytreader 6y ago
> Instead one of your gremlins ran this function directly on the production machine. Exactly my first hypothesis too. But then keepthescore claims, > of course we use different passwords and users for development and production. How would this hypothesis explain that? --- Metadialogue: John Watson: "I deduce that someone changed the dev config source so that it uses the production config values." Sherlock Holmes: "My dear Watson, while that is sensible, it seems to me that the balance of probability leans towards their production instance also having the development access credentials." --- Just my way of saying, I think this case isn't as shut and closed as most comments (including parent) imply. I personally find the /etc/host mapping a likelier hypothesis but even that can't explain how different credentials failed to prevent this. Without more details coming from a proper investigation, we are just piling assumptions on top of assumptions. We are making bricks, without enough clay, as Holmes would say.
- thamer 6y ago
  Agreed, it seems like most people making suggestions above are missing the point about credentials. The code they present explicitly references `config.DevelopmentConfig`: database = config.DevelopmentConfig.DB_DATABASE user = config.DevelopmentConfig.DB_USERNAME password = config.DevelopmentConfig.DB_PASSWORD One way this could happen is if the objects under `config` were loaded from separate files, and the dev file was changed to a symlink to the prod file. So `config.DevelopmentConfig` always loads /opt/myapp/config/dev.cfg but a developer had dev.cfg -> prod.cfg and the prod credentials and connection details were loaded into `config.DevelopmentConfig`. Just an idea.
dschuetz 6y ago
As someone else said it below: "hardcoded to localhost" doesn't mean it's hardcoded. It means it goes to whatever localhost resolves to. Really hardcoded should ALWAYS mean: 127.0.0.1

toredash 6y ago

2 cents his hosts file points localhost to the prod db IP

pvorb 6y ago
Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.
- junon 6y ago
  That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively. Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.
  - pvorb 6y ago
    Why do you think they are hacks? Could you please elaborate? In my opinion, they nicely abstract over server hardware and services running on them, so one is able to have simple infrastructure-as-code management of otherwise complicated setups.
    - deleted 6y ago
      [deleted]
kjaftaedi 6y ago
I think so as well. It doesn't even make sense to connect to a managed database using 'localhost'. Managed databases are never localhost. They are hosted outside your VPS and you use a DNS name to connect to them.
- drdaeman 6y ago
  That could've happened if the database is not accessible from the Internet and they were using a tunnel which binds to localhost (e.g. `ssh -L`). It does make sense to connect to localhost on dev machines. But if that's the setup, I guess one should avoid from tunneling to localhost to avoid potentially dangerous confusion (hmm... I think I'm guilty of that on a couple projects, need to check if that's true...)
dingdingdang 6y ago
Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") that way local specific scripts can be stripped even if in same repo.
- robryan 6y ago
  Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.
  - rwbhn 6y ago
    Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.
radu_floricica 6y ago
I'm betting on a tunnel, myself. And grandparent is probably wrong, they most likely have dedicated mysql machines so "localhost" will never be the db.
robryan 6y ago
One solution to this is to make sure only the production servers can connect to the production database.
buzer 6y ago
My guess would be something like PgBouncer. Someone may have installed it to the production server at some point in the past.

ir123 6y ago

If managed DBs on DigitalOcean are anything like those on AWS, you can not run commands directly on them since SSH is prohibited. EDIT: there's also the deal with different credentials for dev and prod envs.

skytreader 6y ago

> Instead one of your gremlins ran this function directly on the production machine. Exactly my first hypothesis too. But then keepthescore claims, > of course we use different passwords and users for development and production. How would this hypothesis explain that? --- Metadialogue: John Watson: "I deduce that someone changed the dev config source so that it uses the production config values." Sherlock Holmes: "My dear Watson, while that is sensible, it seems to me that the balance of probability leans towards their production instance also having the development access credentials." --- Just my way of saying, I think this case isn't as shut and closed as most comments (including parent) imply. I personally find the /etc/host mapping a likelier hypothesis but even that can't explain how different credentials failed to prevent this. Without more details coming from a proper investigation, we are just piling assumptions on top of assumptions. We are making bricks, without enough clay, as Holmes would say.

thamer 6y ago
Agreed, it seems like most people making suggestions above are missing the point about credentials. The code they present explicitly references `config.DevelopmentConfig`: database = config.DevelopmentConfig.DB_DATABASE user = config.DevelopmentConfig.DB_USERNAME password = config.DevelopmentConfig.DB_PASSWORD One way this could happen is if the objects under `config` were loaded from separate files, and the dev file was changed to a symlink to the prod file. So `config.DevelopmentConfig` always loads /opt/myapp/config/dev.cfg but a developer had dev.cfg -> prod.cfg and the prod credentials and connection details were loaded into `config.DevelopmentConfig`. Just an idea.

dschuetz 6y ago

As someone else said it below: "hardcoded to localhost" doesn't mean it's hardcoded. It means it goes to whatever localhost resolves to. Really hardcoded should ALWAYS mean: 127.0.0.1