8 ms·

The carefulness knob

186 points by azhenley 2y ago

AnotherGoodName 2y ago
Fwiw in a real world scenario it'd be more helpful to hear "the timeline has risks" alongside a statement of a concrete process you might not be doing given that timeline. Everyone already knows about diminishing returns, we don't need a lesson on that.
- Swizec 2y ago
  My favorite tool when defining project timelines: What are we not doing? There's an infinite number of nice-to-haves. A nice good deadline makes it super easy to clarify what you actually need vs what you only want.
  - TylerE 2y ago
    and you'd be amazed when you start really having these discussions with the client how often stuff ends up not only not being needed, but often going right past 'nice to have' and right to 'lets not'. The problem is often the initial problem being WAY overspecified in ways that don't ACTUALLY matter but generate tons of extra work.
    - whstl 2y ago
      Yeah, to me this kind of thing is much better than the carefulness knob. Delaying or just not doing certain features that have low ROI can drastically shorten the development time without really affecting quality. This is something that as an industry we seem to have unlearned. Sure, it still exists in the startup space, with MVPs, but elsewhere it's very difficult. In the last 20 years I feel like engineers have been pushed more and more away from the client, and very often you just get "overspecified everything" from non-technical Product Managers and have to sacrifice in quality instead.
      - TylerE 2y ago
        I had one today that with one email went from "I want this whole new report over our main entity type with 3 user specified parameters" to "actually, just add this one existing (in the db) column to this one existing report and that totally solves my actual problem". My time going from something like 2 days to 15 minutes + 10 minutes to write the email.
        whstl 2y ago
        For me the biggest one lately was due to miscommunication and bad assumptions. The designer was working in a redesign in their own free time, so they were using this "new design" as a template for all recent mockups. The Product Manager just created tickets with the new design and was adamant that changing the design was part of the requirements. The feature itself was simple, but the redesign was significantly hard. Talking with the business person revealed that they were not even aware of the redesign and it was blocked until next year.
- TylerE 2y ago
  If everyone actually knew this stuff, this entire class of problem would cease to exist. Given that it has not...
- ChrisMarshallNY 2y ago
  I use a fairly simple “checkerboard” table to display risk, and encourage planning for mitigation/prevention/remedy[0]. In my experience, risk has two dimensions (probability and severity), and three ways to be handled (prevention, mitigation, and remediation). [0] https://littlegreenviper.com/risky-business/ https://littlegreenviper.com/risky-business/
deleted 2y ago
[deleted]
hinkley 2y ago
I’ve been in this industry a long time. I’ve read Lying with Statistics, and a bunch of Tufte. I don’t think it would be too much hyperbole to say I’ve spent almost a half a year of cumulative professional time (2-3 hours a month) arguing with people about bad graphs. And it’s always about the same half dozen things or variants on them. The starting slope of the line in your carefulness graph has no slope. Which means you’re basically telling X that we can turn carefulness to 6 with no real change in delivery date. Are you sure that’s the message you’re trying to send? Managers go through the five stages of grief every time they ask for a pony and you counteroffer with a donkey. And the charts often offer them a pony instead of a donkey. Doing the denial, anger and bargaining in a room full of people becomes toxic, over time. It’s a self goal but bouncing it off the other team’s head. Don’t do that.
- debacle 2y ago
  Maybe we need to start with Zeno. Doing anything at all is relatively impossible.
  - hinkley 2y ago
    Maybe an excellent joke for the pre meeting. I don’t think the boss will find it funny though.
    - debacle 2y ago
      It's true, though: accomplishing anything at all is relatively a feat. You have two ways to contrast a retrospective: what if we had been perfect, and what if we had done nothing. Sometimes a comparison against the latter is more informative.
- kmoser 2y ago
  > The starting slope of the line in your carefulness graph has no slope. Which means you’re basically telling X that we can turn carefulness to 6 with no real change in delivery date. This strikes me as a pedantic argument, since the graph was clearly drawn by hand and is meant to illustrate an upward curving line. Now, maybe there's essentially no clear difference between 5 and 5.1, but when you extrapolate out to where 6 would be (about 65 pixels to the right of 5, if I can be pedantic for a moment), there actually is a difference.
123pie123 2y ago
It generally comes down to the age old question, pick two out of: Quality, Speed or Cost
terlisimo 2y ago
A personal anecdote: One of my guys made a mistake while deploying some config changes to Production and caused a short outage for a Client. There's a post-incident meeting and the client asks "what are we going to do to prevent this from happening in the future?" - probably wanting to tick some meeting boxes. My response: "Nothing. We're not going to do anything." The entire room (incl. my side) looks at me. What do I mean, "Nothing?!?". I said something like "Look, people make mistakes. This is the first time that this kind of mistake had happened. I could tell people to double-check everything, but then everything will be done twice as slowly. Inventing new policies based on a one-off like this feels like an overreaction to me. For now I'd prefer to close this one as human error - wontfix. If we see a pattern of mistakes being made then we can talk about taking steps to prevent them." In the end the conceded that yeah, the outage wasn't so bad and what I said made sense. Felt a bit proud for pushing back :)
- debacle 2y ago
  Good manager, have a cookie.
- tpoacher 2y ago
  Good, but I would have preferred a comment about 'process gates' somewhere in there [0]. I.e. rather than say "it's probably nothing let's not do anything" only to avoid the extreme "let's double check everything from now on for all eternity", I would have preferred a "Let's add this temporary process to check if something is actually wrong, but make sure it has a clear review time and a clear path to being removed, so that the double-checking doesn't become eternal without obvious benefit". [0] https://news.ycombinator.com/item?id=33229338 https://news.ycombinator.com/item?id=33229338
  - Aeolun 2y ago
    Nothing more permanent than a temporary process. When you have zero incidents using the temporary process people will automatically start to assume it’s due to the temporary process, and nobody will want to take responsibility for taking it out.
klabb3 2y ago
It’s not the right approach. Structural engineers shouldn’t let management fiddle with their safety standards to increase speed. They will still blame you when things fail. In software, you can’t just throw in yolo projects with much lower “carefulness” than the rest of the product, everything has maintenance. The TL in this case needs to establish a certain set of standards and practices. That’s not a choice you give away to another team on a per-feature basis. It’s also a ridiculous low bar for engineering managers to not even understand the most fundamental of tradeoffs in software. Of course they want things done faster, but then they can go escalate to the common boss/director and argue about prioritization against other things on the agenda. Not just “work faster”. Then they can go manage those whose work output is proportional to stress, not programmers.
- atoav 2y ago
  Which is why liability is needed.
- thrw42A8N 2y ago
  Management decides whether they build a cheap wooden building, a brick one, or a steel skyscraper. These all have different risk profiles. Safety is a business/management decision, even in structural engineering. A pedestrian bridge could be constructed to support tanks and withstand nuclear explosions, but why. Many engineered structures are actually extremely dangerous - for example mountain climbing trails. Also yes, you have many opportunities to just YOLO without significant consequences in software. A hackathon is a good example - I love them, always great to see the incredible projects at the end. The last one I visited was sponsored by a corporation and they straight up incorporated a startup next day with the winning team.
  - hiddencost 2y ago
    Expected use and desired tolerance is a management decision. Safety is still on engineers.
    - toss1 2y ago
karaterobot 2y ago
LT is a member of the leadership team. LT: Get it done quick, and don't break anything either, or else we're all out of a job. EM: Got it, yes sir, good idea! [EM surreptitiously turns the 'panic' dial to 10, which reduces a corresponding 'illusion of agency' dial down to 'normal']
PlunderBunny 2y ago
I like the idea of having an actual 'carefulness knob' prop and making the manager asking for faster delivery/more checks actually turn the knob themselves, to emphasise that they're the one responsible for the decision.
- nine_zeros 2y ago
  Yep. The best way to pushback is to ask your manager, "We'll do it fast since you are asking for it. What is the plan for contingencies in case things break?"
  - from-nibly 2y ago
    *when things break
    - kortilla 2y ago
      *if, it’s an increasing probably, not a guarantee I’ve seen code with next to no verification turn out great.
- fvdessen 2y ago
  I don’t think the manager needs a button to remember he’s responsible for the decisions, usually it’s the other way around, devs having to remember they’re not; I’ve been too many times the manager putting it on zero only to hear, “You can’t do that it has to be at least a nine !!!”.
  - Aeolun 2y ago
    I’ve seen maybe one out of ten managers actually take responsibility when things go to shit.
  - whatshisface 2y ago
    Letting your hands type an error is a trust fall that nobody is willing to make in an environment without a much higher than usual amount of trust.
lcuff 2y ago
I did a lot of the work in my 40 year software career as an individual, which meant it was on me to estimate the time of the task. My first estimate was almost always an "If nothing goes wrong" estimate. I would attempt to make a more accurate estimate by asking myself "is there a 50% chance I could finish early?". I considered that a 'true' estimate, and could rarely bring myself to offer that estimate 'up the chain' (I'm a wimp ...). When I hear "it's going to be tight for Q2", in the contexts I worked in, that meant "there's no hope". None of this invalidates the notion of a carefulness knob, but I do kinda laugh at the tenor of the imagined conversations that attribute a lot more accuracy to the original estimate that I ever found in reality in my career. Retired 5 years now, maybe some magic has happened while I wasn't looking.
- TylerE 2y ago
  More than once I've used the xkcd method (Pull a gut number out of thin air, then double the numerator and increment the unit e.g. 1 hour -> 2 days, 3 weeks -> 6 months). When dealing with certain customers this has proven disappointingly realistic.
hcarvalhoalves 2y ago
In real life you both can't afford having reputation risk and you need to ship anyway. If you have an incident, guess who's going to be liable – the manager or the name on the commit? Stop negotiating quality; negotiate scope and set a realistic time. Shipping a lot of crap faster is actually slower. 99% of the companies out there can't focus on doing _one_ thing _well_, that's how you beat the odds.
- phoronixrly 2y ago
  I got the sack when I did this last, which was better than the alternative – to keep trudging until the inevitable incident
JoshTriplett 2y ago
> TL: If we did that, we’d just be YOLO’ing our changes, not doing validation. Which means we’d increase the probability of incidents significantly, which end up taking a lot of time to deal with. I don’t think we’d actually end up delivering any faster if we chose to be less careful than we normally are. This is a really critical property that doesn't get highlighted nearly often enough, and I'm glad to see it reinforced here. Slow is smooth, smooth is fast. And predictable.
groby_b 2y ago
I am probably missing an essential point here, but my first reaction was "this is literally the quality part of the scope/cost/time quality trade-off triangle?" Has that become forgotten lore? (It might well be. It's old, and our profession doesn't do well with knowledge transmission. )
- 082349872349872 2y ago
  would it help if we had names for the various 2 of 3's? in my idiolect: cheap and cheerful : . cheap fast best of breed : good . fast mature technology : good cheap .
xylophile 2y ago
Lorin is always on point, and I appreciate the academic backing he brings to the subject. But for how many years do we need to tell MBAs that "running with scissors is bad" before it becomes common knowledge? (Too damn many.)
- Aeolun 2y ago
  I think the kind of people that run with scissors for years and somehow have nothing happen have a higher tendency to become MBAs.
1a527dd5 2y ago
In general, I found that when I've told people to be careful on that code path (because it has bitten me before) I don't get the sense that it is a welcomed warning. It's almost as if I'm questioning their skill as a engineer. I don't know about you but when I'm driving a road and there is black ice around the corner a warning from a fellow driver is welcomed.
- kayodelycaon 2y ago
  I’ve seen this happen a number of ways. Some jobs I’ve worked for, everyone’s ears perk up when they hear that and several people will curious. I’ve worked places where people’s egos were in the way of basic good sense. I’ve seen well intentioned people who were just flat out abrasive. I have no idea which situations you’re finding yourself in. It may help to sit back and see if you could word things different. I have gotten better communicating by asking the people I’m working with if there was a better way I could have said something. Some managers I’ve had had good advice. (I’ve also gotten myself dragged into their office for said advice.) I have no idea how you approached it, but you could let them decide if they want your advice and have specific examples on how things went wrong if they do. “Hey, I noticed you’re working on this. We’ve had some problems in the past. If you have time, I can go into more detail.” Then again you could just be working with assholes.
  - 1a527dd5 2y ago
    I think some of the blame _definitely_ lies with me. No doubt. The only sure fire way I've found that won't cause offence is to offer up a slack call/huddle so that they can hear my tone. I have been failing quite successfully at communicating my tone over text for some time now, so I confess that and admit it upfront.
glitchc 2y ago
So what does the graph look like from 1 to 5? Is zero defined? What does it mean?
- physicles 2y ago
  Others mentioned a parabola, which seems true to me. Imagine writing a database migration and not testing it. You could end up losing customer data, which is very expensive to "fix".
  - 082349872349872 2y ago
    Oracle is a thing because for the first several years of their existence they lost customer data on the regular, so the IBM DB people laughed at and ignored them.
scott_w 2y ago
As I get older, I find myself enjoying these types of stories less and less. My issue comes from the fact that nobody seems comfortable having a conversation about facts and data, instead resorting to childish analogies about turning knobs. That’s not how our jobs work. We don’t “adjust a carefulness meter.” We make conscious choices day to day based on the work we’re doing and past experience. As an EM, I’d be very disappointed if the incident post mortem was reduced to “your team needs to be more careful.” What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. We then need to lean on data and experience of what the trade offs of those changes would be. Asking a test? Go for it. Adding extra layers of approval before shipping? I’ll need to see some very strong reasons for that.
- Aeolun 2y ago
  > What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features. Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.
  - hamandcheese 2y ago
    > Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
  - scott_w 2y ago
    That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.
tazjin 2y ago
> I mean, in some sense, isn’t every incident in some sense a misjudgment of risk? How many times do we really say, “Hoo boy, this thing I’m doing is really risky, we’re probably going to have an incident!” Not many. Yeah, sure, that never happens. That's why "I told you so" is not at all a common phrase amongst folks working on reliability-related topics ;)
camgunz 2y ago
This feels like a really good starting point to me, but I just want to point out that there's a very low ceiling on the effectiveness of "carefulness". I can spend 8 hours scrutinizing code looking for problems, or I can spend 1 hour writing some tests for it. I can spend 30 minutes per PR checking it for style issues, or I can spend 2 hours adding a linter step to CI. The key here is automating your "carefulness" processes. This is how you push that effectiveness curve to the right. And, the corollary here is that a lack of IC carefulness is not to blame when things break. It is always, always, always process. And to reemphasize the main point of TFA, things breaking is often totally fine. The optimal position on the curve is almost never "things never break". The gulf between "things never break" and "things only break .0001% of the time" is a gulf of gazillions of dollars, if you can even find engineers motivated enough and processes effective enough to get you anywhere close to there. This is what SLAs are for: don't give your stakeholders false impressions that you'll always work forever because you're the smartest and most dedicated amongst all your competitors. All I want is an SLA and a compensation policy. That's professional; that's engineering.
- klysm 2y ago
  > lack of IC carefulness is not to blame when things break But if you can trust ICs a bit, you can move faster
PaulRobinson 2y ago
This is not how I'd expect any professional engineering team to do risk mitigation. This is not hard. Enumerate risks. List them. Talk about them. If you want to turn into something prioritisable, for each one, quantify them: on a scale of 1 to 10 what's the likelihood? On a scale of 1 to 10, what's the impact? Multiply the numbers. Communicate these numbers and see if others agree with your assessment of the numbers. As a team, if the product is more than 15, spend some time thinking about mitigation work you can do to reduce either likelihood or impact or both. The higher the number, the more important you put mitigations into your backlog or "definition of done". Below 15? Check with the team you're going to ignore this. Mitigations are extra work. They add time. They slow down delivery. That's fine, you add them to your backlog as dependent tasks, and your completion estimates move out. Need to hit a deadline? Look at descoping, and include in that descoping a conversation about removing some of the risk mitigations and accepting the risk likelihood and impact. Having been EM, TL and X in this story (and the TPM, PM, CTO and other roles), I don't want a "knob" that people are turning in their heads about their subjective measure of "careful". I want enumerated risks with quantified impact and likelihood and adult conversations about appropriate mitigations that lead to clear decisions.
- 4hg4ufxhy 2y ago
  You are just describing what would happen if knob is turned up. The whole point is whether or not to turn it up, not what the process is once its turned up. Not everyone can afford to waste time planning so much.
  - PaulRobinson 2y ago
    If you can't enumerate risks with the work you're about to do either you don't have any, or you haven't thought enough about it and you're yolo'ing. If you have a list of risks and you don't know how to mitigate them, you're just yolo'ing. "We don't have time to plan" is the biggest source of nonsense in this industry. The process I just described takes about 15 minutes to go through for a month's worth of work. Nobody is so busy they can't spare 15 minutes to think about things that might cause major problems that soak up far, far, far more time.
patwolf 2y ago
I like the idea of imagining that we can arbitrarily adjust the carefulness knob, but I don't think it works like that in reality. You can certainly spend more time writing tests, but a lot of the unforeseen problems I've hit over the years weren't caused by lack of testing--they were caused by unknown things that we couldn't have known regardless of how careful we were. It doesn't make for a very satisfying post mortem.
PaulHoule 2y ago
The dominant model in project management is "divide a project into a set of tasks and analyze the tasks independently". You'd imagine you could estimate the work requirement for a big project by estimating the tasks and adding them up, but you run into various problems. Some tasks are hard to estimate because they have an element of experimentation or research. Here a working model is the "run-break-fix" model where you expect to require an unknown number of attempts to solve the problem. In that case there are two variables you can control: (1) be able to solve the problem in less tries, and (2) take less time to make a try. The RBF model points out various problems with carelessness as an ideology. First of all, being careless can cause you to require more tries. Being careless can cause you to ship something that doesn't work. Secondly, and more important, the royal road to (2) is automation and the realization that slow development tools cause slow development. That is, careless people don't care if they have a 20 minutes build. It's a very fast way to make your project super-late. I worked at a place that organized a 'Hackathon' where we were supposed to implement something with our project in two hours. I told them, "that's alright, but it takes 20 minutes for us to build our system, so if we are maximally efficient we get 6 tries at this". The eng manager says "it doesn't take 20 minutes to build!" (he also says we "write unit tests" and we don't, he says we "handle errors with Either in Scala" which we usually don't, and says "we do code reviews", which I don't believe) I set my stopwatch, it takes 18 minutes. (It is creating numerous Docker images for various parts of the system that all need to get booted up) That organization was struggling with challenging requirements from multiple blue chip customers -- it's not quite true that turning that 20 minute build into a 2 minute build will accelerate development 10x but putting some care in this area should pay for itself. [1] https://www.amazon.com/Have-Fun-at-Work-Livingston/dp/0937063053 https://www.amazon.com/Have-Fun-at-Work-Livingston/dp/093706...
settsu 2y ago
All I see here is an all-too-common organizational issue that something like this is having to be explained to someone in a management role. They should know these things. And they should know them well. If your company is needing to have conversations like this more than rarely—let alone experiencing the actual issue being discussed—then that's a fundamental problem with leadership.
yc6f6rblvtdknh 2y ago
Vou atravessar aqui

186 points by azhenley 2y ago

Fwiw in a real world scenario it'd be more helpful to hear "the timeline has risks" alongside a statement of a concrete process you might not be doing given that timeline. Everyone already knows about diminishing returns, we don't need a lesson on that.

Swizec 2y ago
My favorite tool when defining project timelines: What are we not doing? There's an infinite number of nice-to-haves. A nice good deadline makes it super easy to clarify what you actually need vs what you only want.
- TylerE 2y ago
  and you'd be amazed when you start really having these discussions with the client how often stuff ends up not only not being needed, but often going right past 'nice to have' and right to 'lets not'. The problem is often the initial problem being WAY overspecified in ways that don't ACTUALLY matter but generate tons of extra work.
  - whstl 2y ago
    Yeah, to me this kind of thing is much better than the carefulness knob. Delaying or just not doing certain features that have low ROI can drastically shorten the development time without really affecting quality. This is something that as an industry we seem to have unlearned. Sure, it still exists in the startup space, with MVPs, but elsewhere it's very difficult. In the last 20 years I feel like engineers have been pushed more and more away from the client, and very often you just get "overspecified everything" from non-technical Product Managers and have to sacrifice in quality instead.
    - TylerE 2y ago
      I had one today that with one email went from "I want this whole new report over our main entity type with 3 user specified parameters" to "actually, just add this one existing (in the db) column to this one existing report and that totally solves my actual problem". My time going from something like 2 days to 15 minutes + 10 minutes to write the email.
      - whstl 2y ago
        For me the biggest one lately was due to miscommunication and bad assumptions. The designer was working in a redesign in their own free time, so they were using this "new design" as a template for all recent mockups. The Product Manager just created tickets with the new design and was adamant that changing the design was part of the requirements. The feature itself was simple, but the redesign was significantly hard. Talking with the business person revealed that they were not even aware of the redesign and it was blocked until next year.
TylerE 2y ago
If everyone actually knew this stuff, this entire class of problem would cease to exist. Given that it has not...
ChrisMarshallNY 2y ago
I use a fairly simple “checkerboard” table to display risk, and encourage planning for mitigation/prevention/remedy[0]. In my experience, risk has two dimensions (probability and severity), and three ways to be handled (prevention, mitigation, and remediation). [0] https://littlegreenviper.com/risky-business/ https://littlegreenviper.com/risky-business/

deleted 2y ago

[deleted]

hinkley 2y ago

I’ve been in this industry a long time. I’ve read Lying with Statistics, and a bunch of Tufte. I don’t think it would be too much hyperbole to say I’ve spent almost a half a year of cumulative professional time (2-3 hours a month) arguing with people about bad graphs. And it’s always about the same half dozen things or variants on them. The starting slope of the line in your carefulness graph has no slope. Which means you’re basically telling X that we can turn carefulness to 6 with no real change in delivery date. Are you sure that’s the message you’re trying to send? Managers go through the five stages of grief every time they ask for a pony and you counteroffer with a donkey. And the charts often offer them a pony instead of a donkey. Doing the denial, anger and bargaining in a room full of people becomes toxic, over time. It’s a self goal but bouncing it off the other team’s head. Don’t do that.

debacle 2y ago
Maybe we need to start with Zeno. Doing anything at all is relatively impossible.
- hinkley 2y ago
  Maybe an excellent joke for the pre meeting. I don’t think the boss will find it funny though.
  - debacle 2y ago
    It's true, though: accomplishing anything at all is relatively a feat. You have two ways to contrast a retrospective: what if we had been perfect, and what if we had done nothing. Sometimes a comparison against the latter is more informative.
kmoser 2y ago
> The starting slope of the line in your carefulness graph has no slope. Which means you’re basically telling X that we can turn carefulness to 6 with no real change in delivery date. This strikes me as a pedantic argument, since the graph was clearly drawn by hand and is meant to illustrate an upward curving line. Now, maybe there's essentially no clear difference between 5 and 5.1, but when you extrapolate out to where 6 would be (about 65 pixels to the right of 5, if I can be pedantic for a moment), there actually is a difference.

123pie123 2y ago

It generally comes down to the age old question, pick two out of: Quality, Speed or Cost

terlisimo 2y ago

A personal anecdote: One of my guys made a mistake while deploying some config changes to Production and caused a short outage for a Client. There's a post-incident meeting and the client asks "what are we going to do to prevent this from happening in the future?" - probably wanting to tick some meeting boxes. My response: "Nothing. We're not going to do anything." The entire room (incl. my side) looks at me. What do I mean, "Nothing?!?". I said something like "Look, people make mistakes. This is the first time that this kind of mistake had happened. I could tell people to double-check everything, but then everything will be done twice as slowly. Inventing new policies based on a one-off like this feels like an overreaction to me. For now I'd prefer to close this one as human error - wontfix. If we see a pattern of mistakes being made then we can talk about taking steps to prevent them." In the end the conceded that yeah, the outage wasn't so bad and what I said made sense. Felt a bit proud for pushing back :)

debacle 2y ago
Good manager, have a cookie.
tpoacher 2y ago
Good, but I would have preferred a comment about 'process gates' somewhere in there [0]. I.e. rather than say "it's probably nothing let's not do anything" only to avoid the extreme "let's double check everything from now on for all eternity", I would have preferred a "Let's add this temporary process to check if something is actually wrong, but make sure it has a clear review time and a clear path to being removed, so that the double-checking doesn't become eternal without obvious benefit". [0] https://news.ycombinator.com/item?id=33229338 https://news.ycombinator.com/item?id=33229338
- Aeolun 2y ago
  Nothing more permanent than a temporary process. When you have zero incidents using the temporary process people will automatically start to assume it’s due to the temporary process, and nobody will want to take responsibility for taking it out.

klabb3 2y ago

It’s not the right approach. Structural engineers shouldn’t let management fiddle with their safety standards to increase speed. They will still blame you when things fail. In software, you can’t just throw in yolo projects with much lower “carefulness” than the rest of the product, everything has maintenance. The TL in this case needs to establish a certain set of standards and practices. That’s not a choice you give away to another team on a per-feature basis. It’s also a ridiculous low bar for engineering managers to not even understand the most fundamental of tradeoffs in software. Of course they want things done faster, but then they can go escalate to the common boss/director and argue about prioritization against other things on the agenda. Not just “work faster”. Then they can go manage those whose work output is proportional to stress, not programmers.

atoav 2y ago
Which is why liability is needed.
thrw42A8N 2y ago
Management decides whether they build a cheap wooden building, a brick one, or a steel skyscraper. These all have different risk profiles. Safety is a business/management decision, even in structural engineering. A pedestrian bridge could be constructed to support tanks and withstand nuclear explosions, but why. Many engineered structures are actually extremely dangerous - for example mountain climbing trails. Also yes, you have many opportunities to just YOLO without significant consequences in software. A hackathon is a good example - I love them, always great to see the incredible projects at the end. The last one I visited was sponsored by a corporation and they straight up incorporated a startup next day with the winning team.
- hiddencost 2y ago
  Expected use and desired tolerance is a management decision. Safety is still on engineers.
  - toss1 2y ago

karaterobot 2y ago

LT is a member of the leadership team. LT: Get it done quick, and don't break anything either, or else we're all out of a job. EM: Got it, yes sir, good idea! [EM surreptitiously turns the 'panic' dial to 10, which reduces a corresponding 'illusion of agency' dial down to 'normal']

PlunderBunny 2y ago

I like the idea of having an actual 'carefulness knob' prop and making the manager asking for faster delivery/more checks actually turn the knob themselves, to emphasise that they're the one responsible for the decision.

nine_zeros 2y ago
Yep. The best way to pushback is to ask your manager, "We'll do it fast since you are asking for it. What is the plan for contingencies in case things break?"
- from-nibly 2y ago
  *when things break
  - kortilla 2y ago
    *if, it’s an increasing probably, not a guarantee I’ve seen code with next to no verification turn out great.
fvdessen 2y ago
I don’t think the manager needs a button to remember he’s responsible for the decisions, usually it’s the other way around, devs having to remember they’re not; I’ve been too many times the manager putting it on zero only to hear, “You can’t do that it has to be at least a nine !!!”.
- Aeolun 2y ago
  I’ve seen maybe one out of ten managers actually take responsibility when things go to shit.
- whatshisface 2y ago
  Letting your hands type an error is a trust fall that nobody is willing to make in an environment without a much higher than usual amount of trust.

lcuff 2y ago

I did a lot of the work in my 40 year software career as an individual, which meant it was on me to estimate the time of the task. My first estimate was almost always an "If nothing goes wrong" estimate. I would attempt to make a more accurate estimate by asking myself "is there a 50% chance I could finish early?". I considered that a 'true' estimate, and could rarely bring myself to offer that estimate 'up the chain' (I'm a wimp ...). When I hear "it's going to be tight for Q2", in the contexts I worked in, that meant "there's no hope". None of this invalidates the notion of a carefulness knob, but I do kinda laugh at the tenor of the imagined conversations that attribute a lot more accuracy to the original estimate that I ever found in reality in my career. Retired 5 years now, maybe some magic has happened while I wasn't looking.

TylerE 2y ago
More than once I've used the xkcd method (Pull a gut number out of thin air, then double the numerator and increment the unit e.g. 1 hour -> 2 days, 3 weeks -> 6 months). When dealing with certain customers this has proven disappointingly realistic.

hcarvalhoalves 2y ago

In real life you both can't afford having reputation risk and you need to ship anyway. If you have an incident, guess who's going to be liable – the manager or the name on the commit? Stop negotiating quality; negotiate scope and set a realistic time. Shipping a lot of crap faster is actually slower. 99% of the companies out there can't focus on doing _one_ thing _well_, that's how you beat the odds.

phoronixrly 2y ago
I got the sack when I did this last, which was better than the alternative – to keep trudging until the inevitable incident

JoshTriplett 2y ago

> TL: If we did that, we’d just be YOLO’ing our changes, not doing validation. Which means we’d increase the probability of incidents significantly, which end up taking a lot of time to deal with. I don’t think we’d actually end up delivering any faster if we chose to be less careful than we normally are. This is a really critical property that doesn't get highlighted nearly often enough, and I'm glad to see it reinforced here. Slow is smooth, smooth is fast. And predictable.

groby_b 2y ago

I am probably missing an essential point here, but my first reaction was "this is literally the quality part of the scope/cost/time quality trade-off triangle?" Has that become forgotten lore? (It might well be. It's old, and our profession doesn't do well with knowledge transmission. )

082349872349872 2y ago
would it help if we had names for the various 2 of 3's? in my idiolect: cheap and cheerful : . cheap fast best of breed : good . fast mature technology : good cheap .

xylophile 2y ago

Lorin is always on point, and I appreciate the academic backing he brings to the subject. But for how many years do we need to tell MBAs that "running with scissors is bad" before it becomes common knowledge? (Too damn many.)

Aeolun 2y ago
I think the kind of people that run with scissors for years and somehow have nothing happen have a higher tendency to become MBAs.

1a527dd5 2y ago

In general, I found that when I've told people to be careful on that code path (because it has bitten me before) I don't get the sense that it is a welcomed warning. It's almost as if I'm questioning their skill as a engineer. I don't know about you but when I'm driving a road and there is black ice around the corner a warning from a fellow driver is welcomed.

kayodelycaon 2y ago
I’ve seen this happen a number of ways. Some jobs I’ve worked for, everyone’s ears perk up when they hear that and several people will curious. I’ve worked places where people’s egos were in the way of basic good sense. I’ve seen well intentioned people who were just flat out abrasive. I have no idea which situations you’re finding yourself in. It may help to sit back and see if you could word things different. I have gotten better communicating by asking the people I’m working with if there was a better way I could have said something. Some managers I’ve had had good advice. (I’ve also gotten myself dragged into their office for said advice.) I have no idea how you approached it, but you could let them decide if they want your advice and have specific examples on how things went wrong if they do. “Hey, I noticed you’re working on this. We’ve had some problems in the past. If you have time, I can go into more detail.” Then again you could just be working with assholes.
- 1a527dd5 2y ago
  I think some of the blame _definitely_ lies with me. No doubt. The only sure fire way I've found that won't cause offence is to offer up a slack call/huddle so that they can hear my tone. I have been failing quite successfully at communicating my tone over text for some time now, so I confess that and admit it upfront.

glitchc 2y ago

So what does the graph look like from 1 to 5? Is zero defined? What does it mean?

physicles 2y ago
Others mentioned a parabola, which seems true to me. Imagine writing a database migration and not testing it. You could end up losing customer data, which is very expensive to "fix".
- 082349872349872 2y ago
  Oracle is a thing because for the first several years of their existence they lost customer data on the regular, so the IBM DB people laughed at and ignored them.

scott_w 2y ago

As I get older, I find myself enjoying these types of stories less and less. My issue comes from the fact that nobody seems comfortable having a conversation about facts and data, instead resorting to childish analogies about turning knobs. That’s not how our jobs work. We don’t “adjust a carefulness meter.” We make conscious choices day to day based on the work we’re doing and past experience. As an EM, I’d be very disappointed if the incident post mortem was reduced to “your team needs to be more careful.” What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. We then need to lean on data and experience of what the trade offs of those changes would be. Asking a test? Go for it. Adding extra layers of approval before shipping? I’ll need to see some very strong reasons for that.

Aeolun 2y ago
> What I want from a post mortem is to know how we could prevent, detect or mitigate similar incidents in future and to make those changes to code or process. The answer this post gives to that bizarre question that always gets asked, is ‘nothing’, unless you want to significantly adjust the speed that we deliver features. Any added process or check is going to impose overhead and make the team a little bit less happy. Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. In extremis, you’re reduced to a situation in which you have zero incidents, but you also have zero work getting done.
- hamandcheese 2y ago
  > Ocassionally you’ll have a unicorn situation where there is actually a relatively simple fix, but those are few and far between. Perhaps we have different backgrounds, but even in late stage startups I find there is an abundance of low hanging fruit and simple fixes. I'm sure it's different at Google, though.
- scott_w 2y ago
  That’s simply not true. Some processes are good, some post-mortem outcomes will focus on improving deployment speed (so you can revert changes faster), or improve your monitors so you can detect (and mitigate) incidents faster. On the other hand, enforcing a manual external QA check on every release WILL slow things down. You’re repeating the same mistake as the article by assuming “process” sits on a grade that naturally slows work down. This is because you’re not being precise in your reasoning. Look at the specifics and make a decision based on the facts in front of you.

tazjin 2y ago

> I mean, in some sense, isn’t every incident in some sense a misjudgment of risk? How many times do we really say, “Hoo boy, this thing I’m doing is really risky, we’re probably going to have an incident!” Not many. Yeah, sure, that never happens. That's why "I told you so" is not at all a common phrase amongst folks working on reliability-related topics ;)

camgunz 2y ago

This feels like a really good starting point to me, but I just want to point out that there's a very low ceiling on the effectiveness of "carefulness". I can spend 8 hours scrutinizing code looking for problems, or I can spend 1 hour writing some tests for it. I can spend 30 minutes per PR checking it for style issues, or I can spend 2 hours adding a linter step to CI. The key here is automating your "carefulness" processes. This is how you push that effectiveness curve to the right. And, the corollary here is that a lack of IC carefulness is not to blame when things break. It is always, always, always process. And to reemphasize the main point of TFA, things breaking is often totally fine. The optimal position on the curve is almost never "things never break". The gulf between "things never break" and "things only break .0001% of the time" is a gulf of gazillions of dollars, if you can even find engineers motivated enough and processes effective enough to get you anywhere close to there. This is what SLAs are for: don't give your stakeholders false impressions that you'll always work forever because you're the smartest and most dedicated amongst all your competitors. All I want is an SLA and a compensation policy. That's professional; that's engineering.

klysm 2y ago
> lack of IC carefulness is not to blame when things break But if you can trust ICs a bit, you can move faster

PaulRobinson 2y ago

This is not how I'd expect any professional engineering team to do risk mitigation. This is not hard. Enumerate risks. List them. Talk about them. If you want to turn into something prioritisable, for each one, quantify them: on a scale of 1 to 10 what's the likelihood? On a scale of 1 to 10, what's the impact? Multiply the numbers. Communicate these numbers and see if others agree with your assessment of the numbers. As a team, if the product is more than 15, spend some time thinking about mitigation work you can do to reduce either likelihood or impact or both. The higher the number, the more important you put mitigations into your backlog or "definition of done". Below 15? Check with the team you're going to ignore this. Mitigations are extra work. They add time. They slow down delivery. That's fine, you add them to your backlog as dependent tasks, and your completion estimates move out. Need to hit a deadline? Look at descoping, and include in that descoping a conversation about removing some of the risk mitigations and accepting the risk likelihood and impact. Having been EM, TL and X in this story (and the TPM, PM, CTO and other roles), I don't want a "knob" that people are turning in their heads about their subjective measure of "careful". I want enumerated risks with quantified impact and likelihood and adult conversations about appropriate mitigations that lead to clear decisions.

4hg4ufxhy 2y ago
You are just describing what would happen if knob is turned up. The whole point is whether or not to turn it up, not what the process is once its turned up. Not everyone can afford to waste time planning so much.
- PaulRobinson 2y ago
  If you can't enumerate risks with the work you're about to do either you don't have any, or you haven't thought enough about it and you're yolo'ing. If you have a list of risks and you don't know how to mitigate them, you're just yolo'ing. "We don't have time to plan" is the biggest source of nonsense in this industry. The process I just described takes about 15 minutes to go through for a month's worth of work. Nobody is so busy they can't spare 15 minutes to think about things that might cause major problems that soak up far, far, far more time.

patwolf 2y ago

I like the idea of imagining that we can arbitrarily adjust the carefulness knob, but I don't think it works like that in reality. You can certainly spend more time writing tests, but a lot of the unforeseen problems I've hit over the years weren't caused by lack of testing--they were caused by unknown things that we couldn't have known regardless of how careful we were. It doesn't make for a very satisfying post mortem.

PaulHoule 2y ago

The dominant model in project management is "divide a project into a set of tasks and analyze the tasks independently". You'd imagine you could estimate the work requirement for a big project by estimating the tasks and adding them up, but you run into various problems. Some tasks are hard to estimate because they have an element of experimentation or research. Here a working model is the "run-break-fix" model where you expect to require an unknown number of attempts to solve the problem. In that case there are two variables you can control: (1) be able to solve the problem in less tries, and (2) take less time to make a try. The RBF model points out various problems with carelessness as an ideology. First of all, being careless can cause you to require more tries. Being careless can cause you to ship something that doesn't work. Secondly, and more important, the royal road to (2) is automation and the realization that slow development tools cause slow development. That is, careless people don't care if they have a 20 minutes build. It's a very fast way to make your project super-late. I worked at a place that organized a 'Hackathon' where we were supposed to implement something with our project in two hours. I told them, "that's alright, but it takes 20 minutes for us to build our system, so if we are maximally efficient we get 6 tries at this". The eng manager says "it doesn't take 20 minutes to build!" (he also says we "write unit tests" and we don't, he says we "handle errors with Either in Scala" which we usually don't, and says "we do code reviews", which I don't believe) I set my stopwatch, it takes 18 minutes. (It is creating numerous Docker images for various parts of the system that all need to get booted up) That organization was struggling with challenging requirements from multiple blue chip customers -- it's not quite true that turning that 20 minute build into a 2 minute build will accelerate development 10x but putting some care in this area should pay for itself. [1] https://www.amazon.com/Have-Fun-at-Work-Livingston/dp/0937063053 https://www.amazon.com/Have-Fun-at-Work-Livingston/dp/093706...

settsu 2y ago

All I see here is an all-too-common organizational issue that something like this is having to be explained to someone in a management role. They should know these things. And they should know them well. If your company is needing to have conversations like this more than rarely—let alone experiencing the actual issue being discussed—then that's a fundamental problem with leadership.

yc6f6rblvtdknh 2y ago

Vou atravessar aqui