5 ms·

A bit of an aside, but after someone introduced me to the notion of Reversible Decisions, it quickly became apparent to me that the solution to the bikeshed pro

by hinkley 3mo ago

A bit of an aside, but after someone introduced me to the notion of Reversible Decisions, it quickly became apparent to me that the solution to the bikeshed problem is to throw money at it before the roosters can start preening about which color the shed should be. Decisions that are reversible should just go with the instinctive answer of whoever volunteers to work on it. I've been in many meeting rooms where, because of the number and caliber of people in the room, we've blown $5000 worth of combined salary arguing about basically nothing. I've been in a few where that number was well over $10k. If you're going to assign a relatively medium talent engineer to solve a problem, it's cheaper to let them solve it twice, maybe even three times, than it is to try to figure out what the right solution is before touching a keyboard. It helps them grow to give them that autonomy, and more importantly training your team out of reflexively reaching for optimization for every single feature saves gobs of money over time. The interface for a piece of code matters to everyone. The internal implementation details mostly matter to the bus number on that code. If they're happy with it, that matters a lot. That can be overridden by the consequences of that design, but I've seen a couple cases where the bus number for a module wanted a solution with fewer consequences but the group wisdom wanted something flashier but also more brittle.

pinkmuffinere 3mo ago
+1! I've fallen in love with many of Amazon's in-group concepts, and maybe I'm just drinking the koolaide, but they have the concept of a "two-way door", which is exactly this -- a decision that can be made, unmade, remade, etc relatively cheaply. If you can identify that a choice isn't very dangerous, you can focus on the things that really are instead.
- shepherdjerred 3mo ago
  Amazon's leadership principles are fantastic. They are applicable to anyone doing any job at any level.
  - pbgcp2026 3mo ago
    ... so were "HP Way", "Siebel Principles", "Hyperion Essbase ethics" and many others, collecting dust on library shelves while MBA students getting instant high from Jack Welch's "Winning".
    - hinkley 3mo ago
      Jack lived long enough to become the villain. Motorola will tell you that they invented Six Sigma and Welch copied it. They also made Digital Six Sigma and that community was a cult.
- hinkley 3mo ago
  Amazon unfortunately downplays the value of ethics over numbers, and we see how that has turned out. The numbers say that counterfeit goods are just fine.
  - pinkmuffinere 3mo ago
    To be clear, I'm not endorsing Amazon writ large, I'm just saying I like the framing (ha!) as "two-way doors". This is maybe equivalent to your concept of reversible decisions, just by another name. While we are griping about Amazon, my main complaints are about the fulfillment centers/working conditions, the anti-union activity, building their own versions of products that sell well on their platform, and the penalizing of companies that price higher on-amazon than off-amazon.
busterarm 3mo ago
This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them. I worked at a large, publicly-traded multinational where decades prior and they were still just a 4 man startup they decided the database server and all timestamps should be in the local timezone. They are still using EST today even when they have global sharding of their customers/databases between US, EU, LATAM, SEA... --- you're also assuming that the product roadmap will afford your engineers any time to build it the second, third, fourth, etc. time.
- hinkley 3mo ago
  When you first introduce the idea you'll find a bunch of people think decisions are Reversible which are not, as you say. And the flip side of Reversible decisions is the Last Responsible Moment, which runs afoul of Hofstadter's Law, and people wait until halfway past the last responsible moment. The key to reversing a decision is getting over Sunk Cost, to start thinking of some code as scaffolding. Scaffolding allows you to get on to other work and then remove it after, because it's either not needed or the 'real' solution has been installed. People get defensive when you propose to rip out their code. Hey that code made us $250k at a time when we were about to miss payroll. Yes. It did. Thank you for your service. But now it's costing us $30k a month and that shit needs to go. Getting people to figure out that if a decision is important, making it later is actually the sane thing to do, is a challenge. Because many people's intuition is that we should put energy into this now while it's fresh and we have abundant energy. We can 'solve' it and not have it dangling over our heads. But we don't know the right answer yet. We don't know the strength of our tools or the expertise of our coworkers. The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes. (And everything should be in GMT unless you can literally point me to a several hundred page treatise on why another time zone is the correct one. Yeah I've worked west coast places that got bought by NY or Chicago companies and it's a clusterfuck if you both didn't use GMT)
  - busterarm 3mo ago
    Don't get me wrong, I'm a fan of the idea. I pretty much follow a lot of what you're saying without giving it names. I'm just used to people giving lip service to certain ideas as an excuse to move with less friction in their org and end up doing long-term organizational damage. Thank you for the added detail. > The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes. Also agree, but teams use sprints and "the roadmap" as a way to say no to fixing bottlenecks they've created for other teams and don't want to take the time and effort to resolve.
happymellon 3mo ago
> but I've seen a couple cases where the bus number for a module wanted a solution with fewer consequences but the group wisdom wanted something flashier but also more brittle. Only a couple? This is some peoples entire job, we can refer to them as "enterprise architecture". Lived experience is a decade ago, and you once saw Kafka solve someone's queuing problem so now everyone's api interface has to run over Kafka. Even synchronous ones.
- hinkley 3mo ago
  Let me rephrase that. I've seen a few cases where it became a major issue, rather than just being a fact. Often people are smart enough to not ask questions they don't want the answer to. If the three people trusted to handle a problem all are in agreement as to how, then the peanut gallery may just be confounding issues in order to pad resumes or to feel important and involved.
crabbone 3mo ago
This is what usually happens if you let incompetent people make design decision: they stay with the first and worst version forever. Right now I'm in the process of leaving a company where this happened over and over again. The company's product and design aspects were always neglected in this specific way: whenever a problem was discovered its solution was assigned to whoever would write the code to solve it. That's it. The QA would be notified a week or two before the release and after the solution was "finished". The QA would then not be allowed to even comment on the overall design, only the superficial bug reports were allowed to go through. Needless to say the product's code-base is a dumpster. The morale is low because it's a daunting task to deal with this dumpster on a daily basis. Now, even if you wanted to, you wouldn't have a budget to fix the previous design mistakes because of the layers of more mistakes that were added on top of them due to the low morale and lack of budget / lack of procedure for making better design decisions. The only thing that keeps the product afloat is its uniqueness in a rather niche field and the backing of a large company that acquired it, but doesn't really depend on it and has no time for a thorough audit.
- luipugs 3mo ago
  I would have assumed code reviews would have sussed out those issues earlier in the cycle?
  - crabbone 3mo ago
    To comment on my concrete (and quite miserable) situation, here's how it went (and does, even worse with the advent of AI, to this day): * A report comes from the field (to the support team) that something is broken (or highly desired by the customer). * The support team assigns the ticket to the lead engineer. * The lead engineer writes the code and after a quick battle with CI pushes it all the way to the customer in need. * Since it's the lead engineer, he has the authority to merge PRs w/o consulting anyone, which is what he does. You just pull the changes and marvel at them, or at the sunrise, whichever you like best. The "urgency" is used as a justification to skip the due process. Now, when AI came into play, two horrible things were added into the mix: * AI audit and code review. The audit is a 50/50 chance between finding a real problem vs misunderstanding of the purpose of the code or the context in which it is executed. The AI will also come up with a solution that is either completely wrong, unnecessary or touching too many things for a human to track. Code review acts in a similar way, except, technically it fills the role of a human reviewing the code, so it gives a sense of false security to anyone making changes. * AI generating code that is hard for humans to follow. First and most obvious problem is the volume. In minutes AI generates changes that will take months of effort to review. But this is only the beginning of the problem. When reading code written by humans, another human builds a model of the author's approach, ability, intention and permissions. When it comes to AI, these... let's call them "dimensions" are all over the place. The generated code you read may seem very plausible for a while until you stumble into a crucial functionality that completely invalidates everything you've learned about the change so far. The change may make unnecessary segues into the territory it was never meant to touch. So, even when there are code reviews, they now miss a lot more than they used to when only humans were writing code. Also, and this is a pathologically bad, but a very common practice: the reviewer's name is missing from the record. You run git-blame and only see the author, not the reviewer. When things go down, the author takes the heat and the reviewer is nowhere to be found. Subsequently, reviewers don't feel like they need to care as much about the outcomes. Oh, and one more thing. To be effective at reviewing anything you'd want to try things... as in to test them. Perhaps come up with some idea about what things must happen and what things absolutely must not. But where do you find those? Usually... in a design document, s.a. a PRD, but in the case where that design (allegedly) was solely in the head of the person writing the code, how do you even know if the behavior you conjecture the code has (because there are no tests) is the desired one?
ukoki 3mo ago
This reminds me of my experience as a senior contractor joining teams that seem to enjoy endless discussion. For example, when implementing a new feature using an open-source library or framework they will want to make a prototype of all possible libraries, make a report about each of them, have some big discussions about which to choose, and only _then_ start the real implementation. My advice is always to just pick the one your gut tells you is the right one -- if the prototype works, then that _is_ the implementation and we can stop considering alternatives unless we have a good reason to reconsider them. If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll.
- Ma8ee 3mo ago
  > If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll. My life became soo much easier when I had that insight. I used to spend so much mental energy on decisions that most often were inconsequential.
  - Ferret7446 3mo ago
    I'd like to know where you're getting these decisions. All of my hard decisions are due to not knowing what technical obstacles we'll run into later.
    - Ma8ee 3mo ago
      That is often a different type of difficulties. I was thinking about decision like, should I buy car A or car B. I've done my research. Economy, safety, et c. are very similar. But it is a lot of money, and I don't know which is the "best", and of course I want the "best". I could spend weeks on a decision like this. Now I just acknowledge that they seem to be equal. Car B comes in a nice shade of red that I really like, so I buy car B. Regarding your question, there's always going to be known and unknown unknowns down the line. My only advice in that case is to get to the unknowns as fast as possible. And from the start take into account that you most likely will have to rebuild a big part when you learn more. That is one of the reasons modularity and decoupling is so important. You want to be able to easily scrap and replace one part without it affecting everything. And keep it simple. Never try to build something that should be capable to handle all imaginable scenarios in the future. You'll be bitten by one of the unimaginable ones, and your complex "can handle everything" architecture will be too rigid to change.
mr_toad 3mo ago
> If you're going to assign a relatively medium talent engineer to solve a problem, it's cheaper to let them solve it twice, maybe even three times, than it is to try to figure out what the right solution is before touching a keyboard. What’s even worse is when a lot of talented people spend a lot of time planning a solution on paper which turns out not to be workable in practice, and then they throw good money after bad trying to get the unworkable solution to work, rather than admitting they were wrong.
tuatoru 3mo ago
In accounting this is the concept of "materiality". If some asks "is this material?", they are asking if it's worth spending significant time looking at it or deciding. Remember to * 3 when assessing. Choose wrong, and you do the job, undo it, and do it again.
- hinkley 3mo ago
  Always fun when they manage to categorize it wrong 3 times in a row. One of the hardest parts of my job is trying to convince people where the pool of blood on the floor is coming from. We are forever stitching up the wrong wounds and missing others entirely. Pain sneaks up on people and they often don't realize they're in pain until someone takes it away. This is probably why Refactoring is one of my favorite tech books. It's a set of tools and justifications for chipping away at that pain bit by bit, instead of tolerating more and more while productivity grinds to a halt. At the end of the day, if only three people really have to care about a thing and they're fine with it, is it their business to run the project the way they prefer/tolerate? It does matter if the opportunity costs and externalities are affecting other projects. If they don't have time to work on other features because of the mistakes they never correct. If other people's flow is constantly interrupted trying to work around their quirky stuff. But if we don't see any of that, or only rarely, then it's more that my internal architect trying to be bossy rather than lift all boats.

by hinkley 3mo ago

pinkmuffinere 3mo ago

+1! I've fallen in love with many of Amazon's in-group concepts, and maybe I'm just drinking the koolaide, but they have the concept of a "two-way door", which is exactly this -- a decision that can be made, unmade, remade, etc relatively cheaply. If you can identify that a choice isn't very dangerous, you can focus on the things that really are instead.

shepherdjerred 3mo ago
Amazon's leadership principles are fantastic. They are applicable to anyone doing any job at any level.
- pbgcp2026 3mo ago
  ... so were "HP Way", "Siebel Principles", "Hyperion Essbase ethics" and many others, collecting dust on library shelves while MBA students getting instant high from Jack Welch's "Winning".
  - hinkley 3mo ago
    Jack lived long enough to become the villain. Motorola will tell you that they invented Six Sigma and Welch copied it. They also made Digital Six Sigma and that community was a cult.
hinkley 3mo ago
Amazon unfortunately downplays the value of ethics over numbers, and we see how that has turned out. The numbers say that counterfeit goods are just fine.
- pinkmuffinere 3mo ago
  To be clear, I'm not endorsing Amazon writ large, I'm just saying I like the framing (ha!) as "two-way doors". This is maybe equivalent to your concept of reversible decisions, just by another name. While we are griping about Amazon, my main complaints are about the fulfillment centers/working conditions, the anti-union activity, building their own versions of products that sell well on their platform, and the penalizing of companies that price higher on-amazon than off-amazon.

busterarm 3mo ago

This often massively discounts the cost of reversing decisions. People often work to build things without any thought given to those who have to maintain it afterwards. Especially when it's not them. I worked at a large, publicly-traded multinational where decades prior and they were still just a 4 man startup they decided the database server and all timestamps should be in the local timezone. They are still using EST today even when they have global sharding of their customers/databases between US, EU, LATAM, SEA... --- you're also assuming that the product roadmap will afford your engineers any time to build it the second, third, fourth, etc. time.

hinkley 3mo ago
When you first introduce the idea you'll find a bunch of people think decisions are Reversible which are not, as you say. And the flip side of Reversible decisions is the Last Responsible Moment, which runs afoul of Hofstadter's Law, and people wait until halfway past the last responsible moment. The key to reversing a decision is getting over Sunk Cost, to start thinking of some code as scaffolding. Scaffolding allows you to get on to other work and then remove it after, because it's either not needed or the 'real' solution has been installed. People get defensive when you propose to rip out their code. Hey that code made us $250k at a time when we were about to miss payroll. Yes. It did. Thank you for your service. But now it's costing us $30k a month and that shit needs to go. Getting people to figure out that if a decision is important, making it later is actually the sane thing to do, is a challenge. Because many people's intuition is that we should put energy into this now while it's fresh and we have abundant energy. We can 'solve' it and not have it dangling over our heads. But we don't know the right answer yet. We don't know the strength of our tools or the expertise of our coworkers. The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes. (And everything should be in GMT unless you can literally point me to a several hundred page treatise on why another time zone is the correct one. Yeah I've worked west coast places that got bought by NY or Chicago companies and it's a clusterfuck if you both didn't use GMT)
- busterarm 3mo ago
  Don't get me wrong, I'm a fan of the idea. I pretty much follow a lot of what you're saying without giving it names. I'm just used to people giving lip service to certain ideas as an excuse to move with less friction in their org and end up doing long-term organizational damage. Thank you for the added detail. > The product roadmap is now and has ever been complete bullshit. Refactoring teaches that you amortize rework across all new stories. That's just how it goes. Also agree, but teams use sprints and "the roadmap" as a way to say no to fixing bottlenecks they've created for other teams and don't want to take the time and effort to resolve.

happymellon 3mo ago

> but I've seen a couple cases where the bus number for a module wanted a solution with fewer consequences but the group wisdom wanted something flashier but also more brittle. Only a couple? This is some peoples entire job, we can refer to them as "enterprise architecture". Lived experience is a decade ago, and you once saw Kafka solve someone's queuing problem so now everyone's api interface has to run over Kafka. Even synchronous ones.

hinkley 3mo ago
Let me rephrase that. I've seen a few cases where it became a major issue, rather than just being a fact. Often people are smart enough to not ask questions they don't want the answer to. If the three people trusted to handle a problem all are in agreement as to how, then the peanut gallery may just be confounding issues in order to pad resumes or to feel important and involved.

crabbone 3mo ago

This is what usually happens if you let incompetent people make design decision: they stay with the first and worst version forever. Right now I'm in the process of leaving a company where this happened over and over again. The company's product and design aspects were always neglected in this specific way: whenever a problem was discovered its solution was assigned to whoever would write the code to solve it. That's it. The QA would be notified a week or two before the release and after the solution was "finished". The QA would then not be allowed to even comment on the overall design, only the superficial bug reports were allowed to go through. Needless to say the product's code-base is a dumpster. The morale is low because it's a daunting task to deal with this dumpster on a daily basis. Now, even if you wanted to, you wouldn't have a budget to fix the previous design mistakes because of the layers of more mistakes that were added on top of them due to the low morale and lack of budget / lack of procedure for making better design decisions. The only thing that keeps the product afloat is its uniqueness in a rather niche field and the backing of a large company that acquired it, but doesn't really depend on it and has no time for a thorough audit.

luipugs 3mo ago
I would have assumed code reviews would have sussed out those issues earlier in the cycle?
- crabbone 3mo ago
  To comment on my concrete (and quite miserable) situation, here's how it went (and does, even worse with the advent of AI, to this day): * A report comes from the field (to the support team) that something is broken (or highly desired by the customer). * The support team assigns the ticket to the lead engineer. * The lead engineer writes the code and after a quick battle with CI pushes it all the way to the customer in need. * Since it's the lead engineer, he has the authority to merge PRs w/o consulting anyone, which is what he does. You just pull the changes and marvel at them, or at the sunrise, whichever you like best. The "urgency" is used as a justification to skip the due process. Now, when AI came into play, two horrible things were added into the mix: * AI audit and code review. The audit is a 50/50 chance between finding a real problem vs misunderstanding of the purpose of the code or the context in which it is executed. The AI will also come up with a solution that is either completely wrong, unnecessary or touching too many things for a human to track. Code review acts in a similar way, except, technically it fills the role of a human reviewing the code, so it gives a sense of false security to anyone making changes. * AI generating code that is hard for humans to follow. First and most obvious problem is the volume. In minutes AI generates changes that will take months of effort to review. But this is only the beginning of the problem. When reading code written by humans, another human builds a model of the author's approach, ability, intention and permissions. When it comes to AI, these... let's call them "dimensions" are all over the place. The generated code you read may seem very plausible for a while until you stumble into a crucial functionality that completely invalidates everything you've learned about the change so far. The change may make unnecessary segues into the territory it was never meant to touch. So, even when there are code reviews, they now miss a lot more than they used to when only humans were writing code. Also, and this is a pathologically bad, but a very common practice: the reviewer's name is missing from the record. You run git-blame and only see the author, not the reviewer. When things go down, the author takes the heat and the reviewer is nowhere to be found. Subsequently, reviewers don't feel like they need to care as much about the outcomes. Oh, and one more thing. To be effective at reviewing anything you'd want to try things... as in to test them. Perhaps come up with some idea about what things must happen and what things absolutely must not. But where do you find those? Usually... in a design document, s.a. a PRD, but in the case where that design (allegedly) was solely in the head of the person writing the code, how do you even know if the behavior you conjecture the code has (because there are no tests) is the desired one?

ukoki 3mo ago

This reminds me of my experience as a senior contractor joining teams that seem to enjoy endless discussion. For example, when implementing a new feature using an open-source library or framework they will want to make a prototype of all possible libraries, make a report about each of them, have some big discussions about which to choose, and only _then_ start the real implementation. My advice is always to just pick the one your gut tells you is the right one -- if the prototype works, then that _is_ the implementation and we can stop considering alternatives unless we have a good reason to reconsider them. If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll.

Ma8ee 3mo ago
> If a decision is hard to make, its often because the differences are small, and therefore it doesn't matter much and you might as well decide by dice-roll. My life became soo much easier when I had that insight. I used to spend so much mental energy on decisions that most often were inconsequential.
- Ferret7446 3mo ago
  I'd like to know where you're getting these decisions. All of my hard decisions are due to not knowing what technical obstacles we'll run into later.
  - Ma8ee 3mo ago
    That is often a different type of difficulties. I was thinking about decision like, should I buy car A or car B. I've done my research. Economy, safety, et c. are very similar. But it is a lot of money, and I don't know which is the "best", and of course I want the "best". I could spend weeks on a decision like this. Now I just acknowledge that they seem to be equal. Car B comes in a nice shade of red that I really like, so I buy car B. Regarding your question, there's always going to be known and unknown unknowns down the line. My only advice in that case is to get to the unknowns as fast as possible. And from the start take into account that you most likely will have to rebuild a big part when you learn more. That is one of the reasons modularity and decoupling is so important. You want to be able to easily scrap and replace one part without it affecting everything. And keep it simple. Never try to build something that should be capable to handle all imaginable scenarios in the future. You'll be bitten by one of the unimaginable ones, and your complex "can handle everything" architecture will be too rigid to change.

mr_toad 3mo ago

> If you're going to assign a relatively medium talent engineer to solve a problem, it's cheaper to let them solve it twice, maybe even three times, than it is to try to figure out what the right solution is before touching a keyboard. What’s even worse is when a lot of talented people spend a lot of time planning a solution on paper which turns out not to be workable in practice, and then they throw good money after bad trying to get the unworkable solution to work, rather than admitting they were wrong.

tuatoru 3mo ago

In accounting this is the concept of "materiality". If some asks "is this material?", they are asking if it's worth spending significant time looking at it or deciding. Remember to * 3 when assessing. Choose wrong, and you do the job, undo it, and do it again.

hinkley 3mo ago
Always fun when they manage to categorize it wrong 3 times in a row. One of the hardest parts of my job is trying to convince people where the pool of blood on the floor is coming from. We are forever stitching up the wrong wounds and missing others entirely. Pain sneaks up on people and they often don't realize they're in pain until someone takes it away. This is probably why Refactoring is one of my favorite tech books. It's a set of tools and justifications for chipping away at that pain bit by bit, instead of tolerating more and more while productivity grinds to a halt. At the end of the day, if only three people really have to care about a thing and they're fine with it, is it their business to run the project the way they prefer/tolerate? It does matter if the opportunity costs and externalities are affecting other projects. If they don't have time to work on other features because of the mistakes they never correct. If other people's flow is constantly interrupted trying to work around their quirky stuff. But if we don't see any of that, or only rarely, then it's more that my internal architect trying to be bossy rather than lift all boats.