19 ms·

Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

• hackburg 3y ago
  [dead]
• Jtsummers 3y ago
  Only previous submission that had a discussion (and it's worth reading through): https://news.ycombinator.com/item?id=8940820 https://news.ycombinator.com/item?id=8940820 - Jan 24, 2015 (50 comments)
  • deleted 3y ago
    [deleted]
• ghaff 3y ago
  In about that same timeframe, Y2K is also really good example. Nothing of note really happened. But if people had just ignored it a lot of things probably would have.
  • rhcom2 3y ago
    Another example is ozone depletion.
    • ender341341 3y ago
      yeah, I've had that discussion "we were all worried about the ozone and nothing happened, this is just a repeat and people will stop talking about it any time now" and my response is "'Nothing' happened because the governments acted fast and turned it around so now it's mostly just boring stuff like catching some factoryy that decides to go cheap and use banned chemicals to save a buck"
  • gumby 3y ago
    This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs. There were a few jokes about it in the newspapers but by and large the public just ignored it. I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.
    • throwup238 3y ago
      > I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless. Unfortunately those fixes are slated for Humanity 2.0, Homo extinctus.
    • mcdonje 3y ago
      Y2K would negatively impact companies' bottom lines if not dealt with, directly, in a short and known timeline, in predictable ways. Climate change will impact companies in different ways at different times. Oil companies, for instance, are incentivized to ramp up production as much as possible while also diversifying. The way to get all companies to take appropriate actions is to not allow them to externalize costs. That means a significant carbon tax.
    • konfusinomicon 3y ago
      I'm sure I'm not the only one that can't wait for the 2038 problem to get closer. bank accounts gonna be jumping like a 3-peat Micheal Jordan when those consultation fee checks start rolling in.. provided the Boston Dynamics dogs haven't taken over yet
      • jcadam 3y ago
        Java EE will be the COBOL of 2037...
        • zilti 3y ago
  • _yb2s 3y ago
    What, what do you mean nothing happened? I'm in my underground bunker, do you think it would be safe to come out and look around? You guys are all in your Y2K underground bunkers still, right?
    • captainbland 3y ago
      Yeah, it's the year 24, don't worry about it. A human is probably still able to type this.
    • Smoosh 3y ago
      My advice looking back over the past 25 years . . . stay in the bunker.
    • ghaff 3y ago
      I didn't worry. Even took a trip to a different country. Though I can totally understand why some people (know you're joking) might have played it more conservatively.
  • HankB99 3y ago
    Yeah. I've heard folk who should know better (looking at you, John C. Dvorak) call Y2K a nothingburger and suggest the effort to prevent problems was a total boondoggle. Disclaimer: I was part of that effort. The funny thing was that I was brought back to a previous client to fix something that my previous efforts had literally caused. Took me 20 minutes to fix it once I saw what the problem was. And then there was a "while you're here, can you take a look at this ..." That lasted about two years until they closed the department and moved it to NY, NY. At least I got credit in terms of billable hours.
    • ghaff 3y ago
      While I often enjoyed reading Dvorak columns in PCMag, he was always more someone who liked to provoke people than offer thoughtful commentary.
  • Justsignedup 3y ago
    the readiness paradox: If you are prepared, nothing interesting happens, life moves on, people opened a notebook and pressed a few buttons. If you are not prepared, the texas grid freezes, people die, people lose their savings, and "nobody could have imagined it would be this bad".
    • nonrandomstring 3y ago
      Recommend "Left of Bang" In strategic sysadmin and cybersecurity this is staple thinking, and a useful part of what I teach now. Problem for vigilant thinkers is the rewards, as explained in TFA, are perverse. People are content with technology to be magical. They don't see the millions of people quietly repairing it and planning to keep it all going smoothly. To "bosses" cybersecurity only bring them problems, and cost them money apparently doing nothing. The same for any defence force, until you need it. Did some nice interviews for international sysadmin day last summer [1]. [0] https://www.goodreads.com/book/show/22663095-left-of-bang https://www.goodreads.com/book/show/22663095-left-of-bang [1] https://cybershow.uk/episodes.php?id=15 https://cybershow.uk/episodes.php?id=15
  • aksss 3y ago
    Not "probably". I was involved at ground level fixing code for BP in '98. Can say without a doubt that bad things would have happened. And not because my salary depended on it - there was no shortage of opportunities to work on other things, obviously. It was legitimately an issue that would have crippled the energy industry (major players, and all their many, many dependents). I infer from this experience that the impact would have been the same on many other industries (e.g. finance, resource development) directly and indirectly. But it is a great example. I still run into people who recall Y2K as an example of much ado about nothing. No, no!! It wasn't an issue for you because many people worked hard to prevent the problems. Those problems were not terribly complex just pervasive and critical, requiring a high LOE. It wasn't like a huge moon-landing engineering problem to be held up as some accomplishment of humanity, more akin fixing a lot of dumb Challenger o-ring problems before a blow-up.
• justinzollars 3y ago
  [flagged]
  • Jtsummers 3y ago
    If only! Lots of things seem obvious, like common sense, but they apparently aren't because the behavior of many organizations is to reward the firefighters, to push "work harder" over "work smarter", and other behaviors that (in the long term) impede their ability to improve or expand their capabilities. I liked the quote in the 2015 submission that twic posted [0], it relates back to the "seems obvious" sentiment. A lot of good solutions are simple (or composed of simple elements in a sane and reasonable fashion). But arriving at those solutions is decidedly not simple or obvious if we go by the Rube Goldberg machines that people end up contriving instead. [0] https://news.ycombinator.com/item?id=8940820#8941621 https://news.ycombinator.com/item?id=8940820#8941621
• pintxo 3y ago
  Or: The Preparedness paradox [1] / "There is no glory in prevention" [1] https://en.wikipedia.org/wiki/Preparedness_paradox https://en.wikipedia.org/wiki/Preparedness_paradox
  • justincredible 3y ago
    [dead]
  • LordNibbler 3y ago
    and the related discussion: https://news.ycombinator.com/item?id=32466346 https://news.ycombinator.com/item?id=32466346
• NovemberWhiskey 3y ago
  Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet. This is a management problem, because no-one wants to be accountable for a repeat incident even if it was rational to be working on something else more important.
  • thfuran 3y ago
    There's a ton of reactionary legislation on the books that exists pretty much entirely because politicians wanted to be seen doing something. It's mostly crap.
    • NovemberWhiskey 3y ago
      Yeah, the legendary politician's fallacy: 1) We must do something. 2) This is something. 3) Therefore, we must do this.
      • wolverine876 3y ago
        I thought that was the voter's fallacy.
        • renewiltord 3y ago
          The voter's fallacy is "this happened while this guy was in power; we should replace him" repeated ad inf.
      • shoo 3y ago
        https://www.youtube.com/watch?v=vidzkYnaf6Y https://www.youtube.com/watch?v=vidzkYnaf6Y
    • arp242 3y ago
      This is probably a bit too cynical; it's not just "politicians who want to be seen doing something"; the public often wants something done as well. And "we didn't do anything after that incident from ten years ago and now it happened again" is not a good look. Complex stories about trade-offs and the cure being worse than the disease often don't "play well" in the media, especially not with the opposition demanding that "something could have been done!" And corporations tend to be pathologically risk-averse. Politicians, the media, corporations, and the public all have a part in this.
• TesterVetter 3y ago
  [dead]
• thefatboy 3y ago
  just aint true if you file a bug report
• nomel 3y ago
  In our org, this was brought up. People who had problems, then fixed them, were the ones being seen and recognized by higher level management, and awarded for fixing the problems, when they, often, shouldn't have happened in the first place. Now we have something akin to a "person that prevents problems" award. It's a nice gesture, but, even though I received one, I think it's mostly nonsense. You don't get the face time/experience presenting with the higher level managers. It's almost impossible to prove that you prevented catastrophes, by working extra hours, or doing the right thing. Nobody likes a "Hey look, I told you so!" sort of perspective, no matter how it's framed. To me, this is the most gratification-limiting aspect of being a software engineer: time is required to test quality of foresight/design. Rewarding retroactively, for great past decisions, doesn't happen. There are systems I designed, when making a pittance for compensation, that's still being used a decade later, across multiple companies, almost entirely unmodified. That, now objectively, good work wasn't seen or recognized by anyone paying me at the time, and can't be seen by anyone now. I burned good time for a "good job me!" as a reward. As I get older, that mental masturbation seems less desirable, since it's usually at the expense of the surprisingly limited life credits that we spend on those tasks. But along with most other nerds, I can hardly stop myself because it's an innate obsession.
  • DanielHB 3y ago
    Reminds me that one time I found 17 databases in our org without backups configured. All I got was a thanks from some devops guy and no manager even heard of the problem...
  • pyrolistical 3y ago
    Award everybody with a bonus that can be deducted with every bug report
    • nomel 3y ago
      Oh, maybe adversarial salaries. You each, meticulously, go through the others code, trying to find as many bugs as possible, in your spare time. Each bug you find transfers a percentage of their salary into yours. End result: everyone's code is perfect, and everyone hates each other. :D
• esafak 3y ago
  This happens in organizations that don't measure and respect precursors or leading indicators of success/failure -- organizations that do not practice organizational observability, to coin a phrase. Which is to say, most of them, because this is not something one thinks of until one has suffered its absence. Your managers have to think like SREs, and be technically sophisticated enough to model and address this problem. Does anyone know any great management books on this subject?
  • diatone 3y ago
    This is a great insight, the idea that management has to look at leading indicators for the org itself. Makes me think that beyond line management, a manger’s job is to behave as an investor and steward of the org, instead of getting involved in the details. Not a management book per se, but Working Backwards popularised an Amazon approach of thinking about the organisation in terms of mechanisms. You might find that useful. I’ve always wanted to dig into Scarlet Ink, Dave Anderson’s blog, but never have, and I suspect it’ll be quite useful for you too.
  • godelski 3y ago
    > This happens in organizations that don't measure and respect precursors or leading indicators of success/failure This happens in organizations that become over reliant on metrics and indicators of success/failure instead of recognizing that these are fuzzy concepts and indicators are just that... indications, not answers.
  • notzane 3y ago
    The High Velocity Edge had a lot about this subject. The most memorable example was at a hospital, nurses would occasionally pick up the wrong vial. They always noticed before injecting, but there were lots of close calls that didn’t get tracked anywhere. Once they started tracking that as a leading indicator, it became pretty obvious that the vials needed a better design to prevent even the possibility of grabbing the wrong one. Also, if the hospital never realized this problem and someone got hurt, they would’ve ended up with a much worse solution like “double check really close”
• hakunin 3y ago
  I've been contemplating the issue in the headline, as pertains to my value. When I help somebody in 40 mins with something they've been stuck on for 3 months, my value is clear to everyone. When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear. Don't know how to deal with this paradox.
  • jvanderbot 3y ago
    Unfortunately one of the best ways is to toot your own horn. If you work with good folks, it will be appreciated for you to present a tricky problem you solved, or to present a design you put forth that would save everyone time.
  • runamuck 3y ago
    Easy - half a$$ your code and when it breaks - swoop in, "fix things" (actually do it right) and play the role of hero! (I've seen so-called "Rock Stars" at places I worked do this over and over)
    • samstave 3y ago
      For i in employment ensure there is someone who checks in with me every month when they read this line of code, else, stop code from running so they come to me.
    • syndicatedjelly 3y ago
      Would you feel good about being that kind of engineer, if the external validation was great enough?
      • circusfly 3y ago
        It's OK, the new validation method is AI driven, we're good.
      • sodapopcan 3y ago
        When the external validation is monetary then you gotta do what you gotta do.
• bluGill 3y ago
  I always get cynical when I see awards for software releases. Someone always gets an award for staying up late to fix a last minute problem. Normally the person that should have written good code in the first place so they wouldn't have had to come in late. The award should go to the person (likely people) who never got called in at all in the first place because we didn't find a bug in their code.
  • diamondo25 3y ago
    You guys get awards?
• coldtea 3y ago
  Except TSA security theater. They see their budgets grow and seen as essential, for preventing problems that never happened. But in this case, if those problems were to happen, they wouldn't be stopped by their measures.
• seaourfreed 3y ago
  The opposite. Department heads get fired for allowing blow-ups in their area, that should have been avoided. "Getting credit" means keeping the job managing the department via preventing problems.
  • al_borland 3y ago
    I’ve seen it mean the department head is told to cut staff. If things run smoothly with 100 people, let’s see how runs with 90, then 80, and so on. The preventative measures start taking a backseat as the staffing issues are felt. The cuts keep coming until something falls over and the department head is pushed out after failing to maintain the service.
  • euroderf 3y ago
    But then you need upper management that appreciates the "risk potential" of every area, and what is required to keep that risk at bay.
• datadrivenangel 3y ago
  This may be one of the biggest challenges facing human civilization. Climate Change, Financial Bubbles, etc.
• circusfly 3y ago
  Volvo probably is an exception in the car world.
• crestfallen 3y ago
  My memory is a bit hazy but it's a good story. At one place, there was some important order processing taking place. As is fairly typical, couldn't rely on getting all the required info. Or critically, getting all the required info correctly. Some extra data, slightly missing pieces but enough to work, etc. etc. Some could be pretty gross. We built validation to massage some inputs, modify processing, what have you to address as many of those as we could think of. The team put in some metrics to identify which validations were "triggered" for each order. Neat. If we'd add more, we'd add a date to it. It was great. We reported those stats so anyone could see anytime, but we'd also send out some comms about it every now and then. It also helped tremendously when a coworker or customer or whoever would say "Oh no! What happens if XYZ?" and we'd say no worries, we already addressed it, and we prevented #### orders from getting stuck for XYZ. It showed that the team was thoughtful, that we were invested in making this work, that work needed to continue to keep things running smoothly, and we had data to back that up. Really helped switch the conversation in the org because folks could see it. If someone pointed out that we hadn't thought of something, it honestly was more about what do we do vs. why didn't we think of this. (Yes, there is a comment here about poor org thinking or blame culture, but some of that exists everywhere.) More proactive. Recognition of preventive quality. People gave us accolades and it bubbled up to some good and real recognition at higher levels too. I'm mangling the words here a bit but I hope you get the idea.
  • strgcmc 3y ago
    There is a simple but undeniable element of genius to the idea of, instrumenting the number of validations triggered. I would hazard a guess that 90+% of teams would focus on an "order success rate" type of metric and call it a day, instead of a "# of times bad data was handled" metric, which is exactly how you escape this trap of good work going unnoticed. Thanks for sharing! Shamelessly stealing and applying this concept at work next week!
    • elbear 3y ago
      Yeah. Another way I would put it is to also look at inputs that strayed off the happy path. There may be something to learn from them as well, maybe something that adjusts the happy path.
• nick_m 3y ago
  Great article, and much of this resonates - but what template has been used here? It looks like it was created using LATEX.
• sudosteph 3y ago
  Reminds me of a place I used to work. Every time they asked for feedback, my refrain was "nothing here gets prioritized without a PIR (post-incident response)". Towards the end of my time there, whenever a PIR-related ticket showed up, I would mark it as a duplicate of whatever actual ticket I had dying in the backlog that would have prevented it. It really hurt team morale to have no influence in preventing foreseeable issues in our domain area. Most of the rest of the team had even stopped suggesting improvements altogether because management refused to let us pull our own tickets in.
• acheron 3y ago
  I don’t get credit for my tiger-repelling rock even though we haven’t seen any tigers.
  • qorrect 3y ago
    Are these magic rocks for sale?
• jgeada 3y ago
  Reminds me of this cartoon (which I have posted on my office): https://naksecurity.medium.com/the-detriments-of-hero-culture-3fc455963d6e https://naksecurity.medium.com/the-detriments-of-hero-cultur... This is why in many corporate cultures it pays not to proactively stop a problem that you know how to fix if the problem is not in your immediate problem area. Let it be noticed, let it become someone else's emergency, then fix it. Much better path to a reward that way. Of course, you should also be planning to leave said organization, in the long run it won't do well.
  • cjohnson318 3y ago
    I think that over time people realize who has a fire drill every month, and who quietly just gets things done.
    • serial_dev 3y ago
      I wouldn't be so sure. As an employee and team member, you'll be better perceived if you do your share of performative firefighting every one in a while. I wish it wasn't true, but it usually is. (Though at remote teams, it's less and less true, with the remote teams I was on, nobody really cared). "wow, last week we had such nasty bug, impossible to track down, also caused production reliability issues. Tom stayed up all night, and finally pushed the fix at 3 in the morning." Now, if you then say that it wouldn't have happened if 1. Tom didn't overcomplicate the system, 2. Tom learned our tools properly, 3. Tom actually understood the requirements and at least tested his changes at least once manually, 4. Wrote good code so that it's easy to troubleshoot 5. Wouldn't have forced a rushed PR review together with the product owner. You'll just sound like a bitter, know it all.
• reaperducer 3y ago
  Nobody ever gets credit for fixing problems that never happened (2001) [pdf] I'm reminded of this every time I see some YouTuber or other social media click bait claiming that the Y2K bug was no big deal. The reason it was no big deal is because thousands of graybeards, like myself, stayed up many long nights for months ahead of time making sure things would work. I still remember the tension during the countdown to midnight UTC. Then the tension rising again during the countdown to Eastern time. Then once more at local time. Only when it was 2000 in Pacific did we unclench our cheeks.
  • DinaCoder99 3y ago
    Depending on which Y2K bug you're referring to, this very much happened and is an easily believable scenario. I don't think this is a comparable situation.
  • IshKebab 3y ago
    I still don't quite buy that. Surely it's because computers mostly use epoch time for dates, not dd/mm/yy? Guess we'll find out in 2038.
    • contact9879 3y ago
      Now they do at least
      • IshKebab 3y ago
        They did then as well. I remember before 2000 people saying it wasn't going to be a big issue due to this.
        • svachalek 3y ago
          "Most" programs used epoch time, like video games. The minority that didn't were the big, stable, old programs that ran on big computers. Things like payroll and flight schedules.
    • 3y ago
• woutersf 3y ago
  I call these problems ticking time bombs. This way business has a catchy name to use for the thing they dont understand, its also very clear this needs to be looked at.
• deleted 3y ago
  [deleted]
• femto 3y ago
  Take a leaf from the OH&S book, as OH&S is all about stopping things from happening. Many work places have a huge "Days Since Last Lost Time Accident" sign hanging out the front. It increments every day and the aim is to get it as high as possible. The same metric could be applied to "Days Since Last Stuff Up That Cost Us". It would be easy thing to apply to an office door or cubicle wall.
  • faiD9Eet 3y ago
    I beg to differ. If you have a metric on "time since last outage", people will be incited to hide problems. There is extensive writing on this subject http://rachelbythebay.com/w/2021/06/01/count/ http://rachelbythebay.com/w/2021/06/01/count/
• legitster 3y ago
  This is wonderful description of the kind of hellhole corporate Scrum is turning into. Agile was literally about doing things quickly and quick cycles of capability improvement. But Scrum is a worse version of the planning processes it meant to replace! If anything, the way scrum lays out the work into immediate problems exacerbates the cycle. In the long run it just turns into a ticketing system where fires get pushed up and technical debt gets pushed down. It even spits out a super easy-to-track, meaningless set of efficiency numbers for consultants/executives to min-max!
  • dheera 3y ago
    > Scrum I always thought this just meant to scrumble every week to get things done with a weekly standup?
    • legitster 3y ago
      I believe it's actually a rugby term, and the idea is to wrestle things across the line at a regular cadence as a team. The unit of work is even called a sprint because the idea is specifically to commit to very intense units of work.
      • dheera 3y ago
        Some managers tell their teams that they need to be more agile, I guess when referee yells "scrumble!" everyone sprints with a lot of agility. I'm an engineer type, most of this makes very little sense to me.
  • shaneofalltrad 3y ago
    I found the idea Scrum being efficient when done right was true at one company I worked at. Everyone was committed to the process, the Scrum team included debt as a priority for 20% of the effort and everyone had a fairly accurate velocity that we could bake in an additional 20% of your work as your own interests, so stakeholder priorities filled that remaining 60%. Then we would pivot in some Sprints if any epic/team goals required a push to get done or other emergencies/bugs that required a change in priority.
• dmcq2 3y ago
  This is something that really annoyed me at a company I worked at for a while. And it was far worse - it wasn't a case of not being rewarded, people who planned properly and got the job done were punished for their efforts. If a project got behind they would take people from a project that was on track or better to put them on to it. I pleaded for a super green status for the best projects on the basis that the team would finish sooner and then the entire team could work on something else nnd get it done well too. They could take people from the next tier but that would be an incentive to do well. Not a hope.
• m3kw9 3y ago
  Like the button logic that works but never had problems?
• ranger207 3y ago
  Ah the old ops justification problem Everything is working => "What are we even paying you for?" Everything breaks => "What are we even paying you for?"
  • tap-snap-or-nap 3y ago
    Only take. No give. - Dog
• WalterBright 3y ago
  I tend to prefer working with people whose stuff "just works". I take that back. I strongly prefer working with them.
  • godelski 3y ago
    And people who comment their code. Even if it is just a little. I swear, 90% of my time in coding is just fiddling around with functions to see what they do so that I can actually pull them together in the right way. If there was just a little documentation this would greatly reduce my time. It should also be in every team's and company's best interest because people hours are expensive. It should also be in the best interest of an open source project because more people will be able to build on your stuff. It should also be in your own interest because taking the 30 seconds to a minute to write those lines interrupts you and allows you to rethink and verify your method. So not doing it is only in the interest of moving fast and writing spaghetti code. But you're not actually moving fast. You move fast at that moment, but not in the race.
    • WalterBright 3y ago
      Documentation exists in one of these states: 1. incomplete 2. wrong 3. missing My goal is to write code that is so clear it doesn't need documentation.
      • Panzer04 3y ago
        Some functions are doing jobs well-defined enough that this is sufficient, but there's plenty that aren't and do deserve additional explanation. I don't want to read the code of your function to figure out what its doing. I want to read the function signature, and if that's not clear enough a comment explaining its purpose and parameters. Code explains how it does something, but often does not clearly explain what that something is.
        • WalterBright 3y ago
          I don't expect to attain it, which is why I remarked on it as a goal. I've discovered that a lot can be done to eliminate the need for some forms of documentation. Also a lot can be done with the language to make it more expressive, and thus eliminating the need to document something. For example, having the parameter to a function declared `const` means the function cannot alter it, and this is checked by the compiler. It won't be necessary to mention that in the function documentation. P.S. This doesn't work in C and C++, despite them having a `const` type qualifier. This is because the `const` is not transitive, and can be legitimately cast away. Therefore, it's useless as a guarantee. D's `const` is transitive, and although you can cast it away in @system code, you're on your own with that.
• reso 3y ago
  This is true in a sense but also it isn't. Our institutions spend a lot of resources on preventing things from happening, are successful, and the people who work in these institutions get a paycheque every month. One could frame the entire US military budget as preventative spend.
  • al_borland 3y ago
    >One could frame the entire US military budget as preventative spend. A decent percentage of the population thinks the country should reduce military spending, because they are undervaluing the preventative spend.
    • yunwal 3y ago
      Or because the prevention ends up killing hundreds of thousands of innocent civilians. Oh, and it turns out the reason you supposedly went to war in the first place is completely fake.
• 0ooo0000o0o0o 3y ago
  Yup My manager just told me I got a 3/5 on my eval yet again because in the past six months - I did everything that was asked of me and - I did a lot of things that weren't asked of me and - I didn't catch anything on fire and - I put out someone else's fire and - I prevented several fires from ever being started in the first place, but that's not good enough. The person who started the fire that I put out is getting promoted because they showed initiative in starting the fire. I didn't show enough initiative putting it out for them or preventing fires on multiple other projects. Apparently creating two of my own projects that never caught on fire also didn't demonstrate enough initiative. From now on I'm applying for one new job per day on company time.
  • leovander 3y ago
    Don't take it personally, managers at bigger companies have to evaluate their teams on a bell curve. The whole team can't score a 5 and someone has to be worse than the group.
    • al_borland 3y ago
      It sounds like the fire starter should get the worse score. Of course, we’re only hearing one side of the story.
    • DanielHB 3y ago
      That is called stack ranking and even though it is used in a lot of companies it is widely considered a bad idea. A lot of large companies don't actually use it.
  • dilyevsky 3y ago
    Your manager probably just doesn’t like you, find a new one if you can, there’s no point swimming against the current - you’ll just get tired and get nowhere
• xorvoid 3y ago
  The corollary (if you’re cynical) is that you get Massive Credit for waiting to deploy a solution until the problem becomes a Major Crisis. That is, there’s a big incentive to to prepare the solution preemptively but not deploy.
• paxys 3y ago
  Went through this exact scenario at my job recently. As the tech lead/architect of an org I reviewed a bunch of recently released projects and identified areas that needed critical improvements because they had major reliability/performance issues. One team had multiple releases at the top of the list, and their PM and eng manager (and really everyone else up the chain) simply ignored all concerns because they had to prioritize feature updates instead. Fast forward a few months – I was on vacation and shit blew up. There was a sev 1 escalation, several customers were pissed, the CEO/CTO were involved. The team in question – the same one that wrote the shoddy code and ignored all warnings bells – worked round the clock and got the service back up. Now they are heroes for their effort and the same manager I mentioned has a great reputation at the company for being so active and communicative and showcasing his leadership during the outage.
  • mixmastamyk 3y ago
    Well, you could dust off the old email and 'mistakenly' send it again. Though it will seem petty, a small fraction might rethink the past few months.
    • notimetorelax 3y ago
      This can be framed constructively, do an analysis of what was fixed during the outages from the originally identified and now lobby for funding to fix the rest.
    • euroderf 3y ago
      It might be better to keep it low-profile. Go to a superior with the story of "told ya so".
      • fer 3y ago
        Actually any organisation worthy of the label "engineer" should do a thorough, no-blame, and collaborative post-mortem of such events so they don't happen again, and that would include warnings given. Feature on time + risk, feature late + no risk (and anything in between), it's in the end an engineering and business decision and either choice might be the right one depending on the circumstances.
• mkoubaa 3y ago
  Life is not fair move on
• _heimdall 3y ago
  This us an incentives problem, plain and simple. Its extremely difficult to quantify things that don't happen, and because we've been building our economies and societies around a heavy focus on quantitative metrics we end up only incentivizing people to solve problems. Even create problems on purpose if you need to, you'll get credit for fixing them. Human Resources departments often run into this exact scenario. Many of the tasks that fall into HR are hard or impossible to quantify, and if done well they're also solving problems before they manifest.
  • al_borland 3y ago
    I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.
    • _heimdall 3y ago
      I'm sure there are some that do this. I once worked with a test manager that was effectively judged by how many bugs were found and how many tests were written. He was actually a nice guy just trying to play the game given to him, but if you didn't know this you would think he was doing exactly as you described. There will always be a few bad eggs in the mix, but I'd check the incentives forced on them before judging too harshly.
      • al_borland 3y ago
        The ones I’m dealing with now aren’t very transparent about these things. I’ve tried fishing a little to see what’s behind it, but need to tread lightly to avoid getting yelled at for an hour.
• jb3689 3y ago
  In all seriousness, do companies which don't suffer from this dysfunction exist? If so, are they hiring?
• AlbertCory 3y ago
  Suppose on Sept.10, 2001, you were an airline executive who successfully forced installation of jimmy-proof, secure steel doors between the cockpit and the cabin? The only "credit" you'd get is "This guy cost the airlines god-know-how-much money, and for what? An imaginary threat!"
  • Sai_ 3y ago
    Wow, really puts things into perspective.
  • dhruvkb 3y ago
    The book "The Black Swan" by Nassim Nicholas Taleb contains this very example. It is a great read on why the improbable events have the biggest impact.
    • AlbertCory 3y ago
      which is where I got it (forget the attribution, though, but the name "Taleb" seems to trigger some people).
• msla 3y ago
  There's one thing I don't get about Y2K: It was supposed to be banks with lots of COBOL code being impacted, right? Banks issue 30-year mortgages. So, why weren't banks being impacted in the early 1970s with mortgages due to be retired in the early 1900s?
  • acdha 3y ago
    A better way to think about it is that you can trust the code you test. Y2K was a problem because it affected _everything_ because anywhere a date was used had to be checked. For mortgages, yes, banks did fix that code earlier but they did it because someone noticed a problem in 1980 and fixed that specific problem, not because they transformed how they wrote code and checked all of the millions of lines of other code which nobody had reported problems with. Roll ahead a couple decades, and now you have even more code with lots of interconnected assumptions and the guy who originally wrote it retired to the beach at Margaritaville with zero interest in going back to Cleveland to talk about date math. If you’re lucky, a pile of cash could change that. If you weren’t, they passed in a boating accident last month. My first tech job was working for a COBOL vendor in the mid 90s and heard a lot from customers and other people in the industry. Much of it was cosmetic (rolling from 1999 to 1900 on a display, etc.) but a lot of it would have had real consequences: not accepting credit cards with dates in the new millennium, calculating interest incorrectly, sending people incorrect notices or failing to send correct ones, etc. I remember a someone at Mastercard saying they wouldn’t have been able to processed credit card transactions at all after midnight if they hadn’t done several years of careful work. The two things which helped were that it got enough attention to get the business people to pay for remediation, and there were enough dates in the future that people got wake-up calls in different industries. Mortgages were the earliest but closer in you had things like credit card expiration dates which caused enough problems that people got serious about deploying updates. These days, I’m wondering how 2038 will go. We have a LOT more devices floating around and there are still plenty of new devices shipping with 32-bit embedded Linux which may never get updated. Hopefully most of that cheap IoT stuff won’t last another decade but I’m kind of something like an automotive company getting publicly outed for skimping on technical debt management and some people getting locked out of their smart houses.
• lifeisstillgood 3y ago
  I call this “I, Pencil Socialism”. We live in a world where everyone depends on the output Of everyone else - none of us could live without farmers and hauliers, doctors and street cleaners, all of whom Build on a foundation from the past Assigning value, billionaires collecting rent, that’s the flawed model It’s not capitalism that makes this world possible (the assumption of I Pencil). It’s the sharing, the trading, the leaving money on the table because it’s too complex to work out how much Elon Musk’s 3rd Grade Math teacher should get for teaching him the basics of finance. You don’t get credit - you get to live in the modern world If there are those who do not get to share equally in that world - that’s a bug not a feature. We need to fix the bug.
• marcus_holmes 3y ago
  I thought this was going to be about Y2K, since it was written just after. I worked for years in the late 90's on Y2K projects, helping to stop the UK's critical infrastructure from just stopping at midnight. Wales would not have water or gas without our efforts, for example. But I've heard people since say things like "why did we spend all that money on Y2K, when it clearly wasn't a problem since nothing happened?" and even "Y2K was a hoax invented by the IT industry". We won. We successfully stopped the Y2K bug, and it was hard work, and it wasn't obvious that we had caught it all by midnight. Yet rather than celebrating, some people saw it as evidence that we had been ripping them off. People are weird.
  • cachehit 3y ago
    I know of several other issues like that. I myself made sure the highscore in a video game I was working on would work :-) The thing that annoys me is that this is best case scenario when it comes to climate change. If we do in fact manage to avoid the apocalypse, all the "climate deniers" are going to feel vindicated.
• lanthade 3y ago
  My background is systems operation. The exception I could see to this is when there are rewards/incentives/etc for uptime or non-change-window uptime. Usually this would be coupled to meeting the goal of 4 9's or 5 9's of uptime, meeting SLA requirements, etc. Granted, in my time working for a major 3 letter tech company on account with a major finance company my highest exposure to VP/C*O types was fixing the fallout from a whole chain of corporate culture related mistakes. If I hadn't already had a job lined up elsewhere I probably could have gotten a good bump out of it. I doubt that would have happened if I'd identified and fixed the problems. Part of that busted corporate culture. We didn't have metrics around uptime/SLAs there though which considering the clients business was pretty ridiculous.
• jongjong 3y ago
  Nobody gets credit for fixing: - Problems that never happened. - Problems that people don't realize are happening (where people cannot relate visible symptoms to their root causes). You only get credit for addressing visible symptoms. Though, disturbingly, you will get credit even if the fix is only temporary... This creates an incentive to only address symptoms since not solving the root problem allows you to get credit over and over again for fixing the (recurring) symptoms... In fact, people will praise you even more as a 'Relentless hero who dedicated their entire life to the problem' LOL. Also, nobody gets punished for creating new problems if the visible symptoms cannot be traced back to the root cause. That's why I oppose complexity in all matters - It obscures root causes and sends everyone off on a wild goose chase addressing symptoms... Complexity turns everything into a frustrating game of whac-a-mole. The person causing a problem can even get the credit for solving symptoms of that problem.
• cm2012 3y ago
  I run an ad agency. I only touch ad accounts with pre-existing spend snd results for that reason. If I take the client from $200 CAC to $100 CAC I'm a hero. If I go from scratch to $100 they don't know my value and it's much more work.
• Horffupolde 3y ago
  It’s the Jonah paradox.
• didgetmaster 3y ago
  The same thing occurs for many optimizations. Unless something is unbearably slow and you speed it up 10x or 100x; most people won't notice. You might spend some time making an often called function return in 1 millisecond instead of 2 or 3 and some might notice that things 'seem' snappier; but no one knows why.
• DinaCoder99 3y ago
  Theoretically this is the military's entire job.
• ang_cire 3y ago
  Welcome to Infosec.
• AbraKdabra 3y ago
  I'm both a developer and a security consultant, both jobs deal with this problem. Being a security person, you go from being a no-one until something happens, then every eye is on you. And with the dev side, I work mostly on backend, so my work goes unnoticed always, the frontend dev gets all the credit, no matter how many thousands lines of code I commit, if something looks prettier or just works it's because of the frontend dev. There's two choices, you either don't give a flying fuck and deal with it, or you go as loud as you can to make yourself visible beforehand and try to get credit for your work, your choice.
• lulznews 3y ago
  Corollary - create fake problems then solve them.
  • dr_kiszonka 3y ago
    So become a management consultant? ; )
    • lulznews 3y ago
      Or mid mgmt at tech co
• resonious 3y ago
  Just responding to the title, but I think this is why it's important to have leadership with real "IC" experience. If leadership has an intuition for what's hard and what the failure modes are, they will be able to correctly divvy out acknowledgement more often than a "pure manager" type.
• hansonpeter 3y ago
  [dead]
• rvba 3y ago
  SAP S4/HANA- the "new" ERP software is hardcoded to calculate depreciation until 2045. As programmers you probably know it could be untill year 9999 or much further. But the consultants will not be out of work. Also a way to push customers to upgrade. Artificial problem. On a side note, I there is in my opinion a place for a new ERP disruptor. However making it is hard. Still even those small fish often manage to get money.
• steelframe 3y ago
  Having spent several years as a manager, as a once-again IC I now include lack of problems with my features in production in my written self-assessment. I'll often refer back to major features I shipped ~2 quarters ago with a statement like, "Project Foo continued to function as expected, scaling as designed under load while incurring zero production incidents." I'll often include the words "engineering and operations excellence" and "commitment to product quality" when doing so.
• doubloon 3y ago
  It seems like many of these TQM discussion miss one key factor of TPS. Respect. Engineers and management respect line workers. They have a word for the expert line workers, ("craftsmen"), Takumi. https://www.allaboutlean.com/toyotas-takumi/ https://www.allaboutlean.com/toyotas-takumi/ and Takumi have a valued place in the company. Without that basic level of respect... i dunno.. seems like it wont work
• alephnan 3y ago
  Companies say they care about security, but don’t pay as much
• Animats 3y ago
  This is a failure of internal accounting. If maintenance is reduced, the value of capital equipment declines faster. Production reject rate increases. Accounting systems are not good at catching that. Repetitive manufacturing operations which capture many metrics about what a steady production process is doing may capture such info. Non steady state systems have a real problem.
• misja111 3y ago
  It even gets worse sometimes. You detect a problem that poses a risk to your organization, manage to convince management that it's dangerous, you get some time to fix it and you fix it. Then, because it's fixed, nothing will happen. And then management frowns upon it and wonders why all this effort was needed. A big scale example of this was the Corona vaccination. After the large vaccination campaign in 2020, hospitalisation numbers stabilised to levels that were also seen sometimes in previous years during some flu epidemic. Which led to criticism from some people, who said that this was evidence that the whole Corona epidemic was a hoax, organised by a conspiracy of medicin manufactories and policitians who wanted to scare people to gain more power ..
• asah 3y ago
  FTFY: nobody ever gets credit if they don't publicize their successes. By default, fixing problems before they happen doesn't make headlines, so you have to find clever ways to earn publicity. One politician classic: let it become a "problem" enough to earn publicity, THEN fix it. Two examples: Y2K and Healthcare.gov
  • White_Wolf 3y ago
    spot on. As a general rule of thumb I keep a form of activity/issues log and send a monthly summart to the finance guys and head of dept along with "expected issues". Letting things crash and burn is a thing I learned from one of my managers a while back. It's great for raises, bouses and rep. Takes a good deal of effort not to fix things I know will break though.
• generic92034 3y ago
  There is an interesting twist to the issue if you are doing development support, having a customer ticket queue. If you are diligently fixing errors before they ship you can profit from that. The queue shrinks, the stress levels sink, management is happy with you. However, if the number of tickets gets too low over some period of time, well, management gets ideas that you have free capacity. ;)
• smallstepforman 3y ago
  Its tricky to get credit when your skills are x10 or higher compared to your team. 2 companies ago, a team of 40 engineers took 4 years and $29M to do a generation product update. The next company I worked for took $1M 5 engineers and 2 years (product from scratch). My current company I did the whole thing myself in 14 months (@ < $0.5M), basically on autopilot. Unfortunately, the stakeholders are unaware that they got enormous value by being lucky of hiring the right person for the project. And I still fight management/budget constraits and hot headedness. Almost enough to drive me to next employer.
• Craig_Rodriguez 3y ago
  [dead]
• keiferski 3y ago
  This is kind of a problem with preventive ethics in general. It seems to me that given two bad choices, the best option is actually neither - that is, to prevent the choice from being necessary. To use the common example of the trolley problem, the best answer isn’t to choose one option or another, it’s to prevent the trolley from being about to run someone over in the first place. Unfortunately the visibility of these non-events means that their heroes go unnoticed. Noticing them would require a fundamental reorganization of how we perceive time and the future.
• Sai_ 3y ago
  I like to classify what to prioritise by how many standard deviations from the happy path a certain issue is. For example, I’m working on a pay-as-you-go SaaS - so the probability of using the SaaS while your account balance is low is 1-SD away from the happy path. Figuring out if the user has JavaScript enabled (SaaS is an SPA) would mean the user first signed up with JavaScript enabled, then disabled it. Id put it at 3-SD away from the happy path. Keep doing this and work on the bugs nearest to the happy path. Of course, the 1-SD, 3-SD estimates are just hunches.
• AniseAbyss 3y ago
  [dead]
• gtroja 3y ago
  That's why I deploy buggy products
• butler14 3y ago
  My first ever job was a placement during university -- like a year in industry. It was a regular-joe IT support job at an eLearning company (if they still exist). The FD walked past me one day, having just left what was clearly a deeply uncomfortable meeting. He stopped in his tracks and asked, quite abruptly, why I didn't ever seem to look busy. I replied - with the arrogance of a cocky 20 year old - that it was because, right now, everything is working as it should. Nothing is broken and people are working and productive, which is because of the work of the IT department. The company later become insolvent and was dissolved. So it turns out it might have been him who should have been busier, and he was just taking it out on me.
• mrgriscom 3y ago
  This is why you don't travel back in time and kill Hitler
• antoinec 3y ago
  Reminds of when you see people questioning the usefulness of getting vaccines for some diseases because they have almost entirely disappeared for the past 50 years (especially in 1st world countries for some), without realizing that this is precisely the reason why they don't reappear.
• alsetmusic 3y ago
  Reminds me of the millennium problem. There was a post on the large forum asking if that issue was over-hyped because we didn't see the world crumble. The top response was that it was handled because we threw a lot of money at the problem. I tend to think that was right, but there's no way I know how to verify.
• wanawadhi 3y ago
  Nice article!
• T3RMINATED 3y ago
  [dead]
• agrippanux 3y ago
  On their start date, I would to tell new grads assigned SRE / DevOps roles that their job was thankless and at best if everything worked fine no one would notice them. However, if anything went wrong, they'd not only be on the hook to fix but would do so under tremendous pressure. "Who wants this responsibility?" I'd ask. Those who decided they were up for it usually did a great job, about half opted out at the end of the conversation and were put in roles more suited to their ability to handle stress.
• adi4213 3y ago
  For the auditory learners, here is a summarized audiobook of this PDF : https://player.oration.app/c3feabfb-d437-4545-ae17-235d50d6133a https://player.oration.app/c3feabfb-d437-4545-ae17-235d50d61...
• silent_cal 3y ago
  One time I worked on a team where we got rewarded for being "bug heros", because we resolved a lot of bugs. It was kind of funny because we also generated the bugs.
• wbsun 3y ago
  I work on a large distributed infrastructure. I always joked that my team's projects and people's careers are outage-driven: the only time we become important and people get opportunities for promotion is when where were big-enough outages that executives have to invest heavily on reliability or scalability. Other time, we are just minions who must listen to and serve feature or product teams. Nobody listen to us when we ask a product team to implement a reliability contract in their shining product. Pre-outage improvements, reliability defense in depth, eliminated scalability bottlenecks before they are hit, are all ignored by leadership and the company: it is just human nature that even though they understand you have to prepare for possible issues, if it hasn't happen yet, you won't take it seriously. I've seen this in many internal performance reviews and promotion committees. People who haven't ever got bitten badly by an outage may call these premature optimizations.
• booleandilemma 3y ago
  "Great wisdom is not obvious, great merit is not advertised. When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks of bravery?" - Sun Tzu
• zac23or 3y ago
  This explains anti-vaxxers beautifully. If everyone is vaccinated and the disease disappears (and kills few people) anti-vaxxers think: “why do we vaccinate everyone? The disease didn't kill anyone... and it disappeared."
  • capr 3y ago
    So if you have a headache and you take an aspirin and the headache disappears you're going to conclude it's from the aspirin, right?
    • zac23or 3y ago
      In headache case, the link between illness and cure is instantaneous. But when it comes to vaccination, due to the time needed for everything to take effect (many people need to be vaccinated, it takes time) anti-vaccines are unable to link the vaccine to a cure for the disease. For humans, time changes everything, even the ability to make a connection between illness and cure.
• nojvek 3y ago
  Nobody is a strong word. There are many competent engineering teams that understand the value of strong foundations. However the problem is quite real in large companies where promo driven development takes over and the incentives are shifted from building what is valuable to customers to building what is valuable to get a promo.
• williamcotton 3y ago
  Y’all don’t celebrate Petrov Day?
• Aerialguzman6 3y ago
  [dead]
• andrewwwalkerr 3y ago
  I'm curious about views from the crowd here on how to apply the recommendations from this article in the context of a startup I've worked in both heavy industry (mining /metals production) as well as my own startup for the last 7 years I definitely empathize with the view that diligently investing in people, and taking time to build sustainable processes that focus on the mid to long term is the ideal way to go... But when your runway is weeks, not months, and it's ship or die, seems hard to justify such a Rosy ideal Let the debate begin!
• Nevermark 3y ago
  Diagram 5 can be improved. [Time Spent on Improvement] is shown having a negative impact on [Time Spent Working]. But [Time Spent Working] should also have a negative impact going back to [Time Spent on Improvement]. This simple mutually negative feedback loop, i.e. a competitive relationship, better explains the bistable situation where either [Time Spent Working] reinforces itself, while holding [Time Spent on Improvement] down, or vice versa. Feedback loops reinforce themselves, but there is only so much time, so there will be a strong tendency for one feedback loops to dominate the time. (And since the Time Spent Working -> Reduce Performance Gap loop operates much faster, cheaper in the short run, and is much easier to reason about and manage, it usually wins.)
• timtas 3y ago
  To get credit for fixing a problem that never happened, usually you need to make things a lot worse.