5 ms·
So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for p
by protomyth 11y ago
So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?
- jewel 11y agoThe change was to add a new index and then remove the old one. It's something that would have passed review. The cause was a defect in the tooling. The requests weren't tied together in the way they were displayed and so the DBA removed the old index first.
- protomyth 11y agoThat's not a tooling defect, that's a process and personal knowledge of the system defect. No DBA should remove an index without knowing what is replacing it or that code got deployed to make it obsolete. Checking if that index is currently being used would be a minimum. If a DBA cannot stop a deploy you have process problems.
- mst 11y agoThe post mortem specifically noted that it looked unused, presumably because the code in master now claimed to depend on the new index.
- protomyth 11y agoI do not mean by looking at the source code. I mean by looking at the production database that you are about to remove an index from. This is a financial institution, and I am really struggling with this post mortem to understand how they do not have more ceremony around production changes. Checks and balances are important.
- aidos 11y agoI suspect that it's more common than you'd guess. In one situation I was working for a supplier to a massive corporation (a household name). They had all sorts of red tape in the process. Every change required filling in forms and getting official sign off from several parties before getting the release code over. Anyway, we had a database user that was so restricted the we couldn't run the install process of a new product. I knew they ran the releases as SA so the start of my release script upgraded all the permissions of our user to give full access to everything.
- protomyth 11y agoI guess its very common, but this is a financial institution. What you describe as a solution strikes me as an amazing way to destroy production data. Upgrading every user can basically lead to one hell of an amazing outside attack. I now only have to get one user / password to compromise your database. A certain amount of red tape is a needed thing to make sure you don't affect your customer's business.
- aidos 11y agoTrue, I didn't stop to consider the financial aspect. Was more just pointing out that there's often this separation between the people making the changes and the people making sure it's safe to release. But really, the safe to release step is often not going to catch things – in many cases because it's barely checked. To be fair, I only upgraded my user to give full access to my db and I revoked the permissions once the install script had finished running. I'm not a complete monster :-)
- protomyth 11y agoGood... deploys are a strange world sometime. I just didn't want to hear that you had set yourself up for a CEE (career ending event). People tend to be a tad bit wacky and whacky after a security breech.