6 ms·
Testing in production is the best thing we have ever done with our customers. The key is to be able to quickly undo some scary new thing and get back to a know
by bob1029 6y ago
Testing in production is the best thing we have ever done with our customers.
The key is to be able to quickly undo some scary new thing and get back to a known-good point. Also, you need to make sure the testing you are doing is easily distinguished from normal business activities. For us, the humble feature flag is all it took to get unblocked on moving mountains of complex scary things to production. Being able to reassure the customer that we can instantly revert a piece of experimental functionality has been a game changer.
We used to spend months agonizing over how the test environment is such a poor facsimile of production and complaining about vendor XYZ not setting up various things that we would need to prove our code works as expected. We used to make insane bullshit promises about how if things worked well in staging the move to production should be flawless. Those were some dark times for us.
- Spivak 6y agoIsn't this just blue-green deployments but pushed into your app's codebase instead of the deployment posture? I have no doubt that this works for you but why not just just flip between two versions of your app?
- milesvp 6y agoYeah, never underestimate the value of solid rollback steps. Never deploy anything without some plan to get the system into the previous state, ideally with a button push. And for those rare cases when you can only roll forward, like with a major schema change, make sure that is easy to do too. There’s nothing like the stress of a broken build to make you stupid.
- sojournerc 6y agoMajor schema changes can also be done in a backwards compatible/recoverable way - I like to follow this procedure. Specifically the "The Five Phases of a Live Schema Change" section: https://queue.acm.org/detail.cfm?id=3300018 https://queue.acm.org/detail.cfm?id=3300018
- bob1029 6y agoWe have spent a lot of time to make our schema as stable as possible in the face of potential changes. In places where we might end up with 100+ columns that vary wildly depending on context of use, we use JSON documents which can store objects of arbitrary complexity. This is a very delicate balancing act for us, because we also like to be able to write SQL queries against important business keys, and having those wrapped up in JSON can make it difficult to get good, fast results. In many cases we will duplicate the storage of some information so that it can be indexed as well as be available in the complex object graph.
- deleted 6y ago[deleted]
- karlmdavis 6y agoBoth of these things are true: 1. Developers can be far more productive when they can run everything locally. 2. Until you’ve deployed into and tested against production, you don’t know if your shit actually works. In fact, they’re complementary: developers should have excellent local dev facilities and excellent facilities for running experiments in production.
- bob1029 6y agoI completely agree. We have written a simulator for the production system that allows us some reasonable degree of make-believe regarding our local development efforts. We also use this in our QA environments, and any other system which isn't permitted direct access to the production systems. Everyone on the team is aware of the caveats as well. We say things like "It worked on simulator, but we still need to test in live".