5 ms·

VPN to production and same hostname for dev and prod?

by sdepablos 6y ago

kerng 6y ago
...and same credentials apparently also - there are lots of things that could have prevented something like this.
- k__ 6y ago
  Keep going, I'm writing this down!
  - jlgaddis 6y ago
    Take the "drop database" bit (on the production database) away from your developers, too. As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete"). If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course. Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however! Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.
    - asddubs 6y ago
      that's true, I totally forgot about SQL permissions, there's so many failsafes for this
    - astura 6y ago
      As a developer at all the jobs I've had, I've never even had access to the production database, full stop - only the server admins did. If I needed something from prod I'd go through them. I don't even consider it inconvenient.
      - Izmaki 6y ago
        The distance between the developer and the production database is directly proportional with the size of the company. The same is true for the amount of paperwork needed for even the simplest things. For some company sizes a feedback cycle measured in days is not good enough and may kill the business. There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D
- asddubs 6y ago
  could also be that they just have an if staging/else somewhere for the credentials (but of course this isn't a counterpoint to the root of your point - the script wiping the database shouldn't be using anything that does something like this, and you probably shouldn't do it at all)
- tedk-42 6y ago
  Yeah this stuck out to me as well. To be honest if they ARE using the same creds in prod as they are in dev, this is the first thing they should fix because it would have easily prevented the whole mess. I'll give them the benefit of the doubt and suggest that it wasn't the case and that it's unlikely that a dev or OPs person was tunnelling through a bastion of some kind to run the script. If they pulled their credentials from a store and populated the config object that way, it's very possible someone actually loaded the production credentials by mistake into the development secrets file. The CI/pipeline system has permissions/network access to deploy to any environment, hence how it ran the script to drop the table. I'm purely speculating another alternative to the much worse case of the tunnelling scenario outline.

VPN to production and same hostname for dev and prod?

kerng 6y ago

...and same credentials apparently also - there are lots of things that could have prevented something like this.

k__ 6y ago
Keep going, I'm writing this down!
- jlgaddis 6y ago
  Take the "drop database" bit (on the production database) away from your developers, too. As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete"). If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course. Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however! Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.
  - asddubs 6y ago
    that's true, I totally forgot about SQL permissions, there's so many failsafes for this
  - astura 6y ago
    As a developer at all the jobs I've had, I've never even had access to the production database, full stop - only the server admins did. If I needed something from prod I'd go through them. I don't even consider it inconvenient.
    - Izmaki 6y ago
      The distance between the developer and the production database is directly proportional with the size of the company. The same is true for the amount of paperwork needed for even the simplest things. For some company sizes a feedback cycle measured in days is not good enough and may kill the business. There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D
asddubs 6y ago
could also be that they just have an if staging/else somewhere for the credentials (but of course this isn't a counterpoint to the root of your point - the script wiping the database shouldn't be using anything that does something like this, and you probably shouldn't do it at all)
tedk-42 6y ago
Yeah this stuck out to me as well. To be honest if they ARE using the same creds in prod as they are in dev, this is the first thing they should fix because it would have easily prevented the whole mess. I'll give them the benefit of the doubt and suggest that it wasn't the case and that it's unlikely that a dev or OPs person was tunnelling through a bastion of some kind to run the script. If they pulled their credentials from a store and populated the config object that way, it's very possible someone actually loaded the production credentials by mistake into the development secrets file. The CI/pipeline system has permissions/network access to deploy to any environment, hence how it ran the script to drop the table. I'm purely speculating another alternative to the much worse case of the tunnelling scenario outline.