3 ms·
I've been at multiple companies with formalized postmortem processes that involved final documents that were widely available within the company. And at each of
by mandevil 5y ago
I've been at multiple companies with formalized postmortem processes that involved final documents that were widely available within the company. And at each of them, I spent a few hours one afternoon reading down the last 30 or so. And I was struck by how many of them had the same root cause (different at each company, because each companies culture and tech-stack is unique, which creates unique failure modes). So at $firstCompany, 75% of their errors were "Dev made a change in the big ball of mud that is our product to fix a bug in feature A, didn't realize that code was reused in feature B, and since we don't have sufficient automated testing feature B was never checked before it hit prod." At $secondCompany, a software consultancy, all of the failures were because each deployment was a unique special snowflake with no backups or dev-test environment (they had originally been created with hardware for those, but that hardware was inevitably cannibalized to keep prod going or used to add a new feature to prod, without back-fill). At $thirdCompany, they had a massively complex roll-out process with larger and larger rings of internal dog-fooding, and developers kept screwing it up, advancing software that was known bad or hurrying things along too quickly or missing roll-out dates or whatever.
Reading a bunch of these documents in a row made the failure mode for the corporate culture clear and obvious. The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. At $firstCompany I asked my bosses to take a dev team and fix enough stuff so that we could run a full automated test suite reliably before every deployment, was told it wasn't a good idea, and left the company shortly afterwards. At $secondCompany they tried things to get that to work, but had major cultural problems with getting individuals on each team to have the discipline to backfill the hardware and actually keep prod and dev-test identical. At $thirdCompany they looked at the same problem of individual discipline and created a team of professional non-developers who were full time focused on controlling software roll-outs so that every new roll-out went to the right level at the right time. We devs gratefully handed over this complex process to that team, and the professionals did a great job.
Be like $thirdCompany: figure out your larger internal pain points and fix those; you actually need to be hit over the head with the data on this to make it clear how your culture and tech stack are failing.
- hinkley 5y ago> The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. In Good to Great this is an aspect of Confronting the Brutal Facts. https://www.jimcollins.com/concepts/confront-the-brutal-facts.html https://www.jimcollins.com/concepts/confront-the-brutal-fact...