12 ms·

The idea of fixing a whole class of problems is common in safety critical software. When you find the cause of a bug its not just about fixing the bug but looki

• l33t233372 4y ago
  > because we had no bugs in our software and there was no drama for the business to get what it wanted and needed I’m confused here. The business needed your software to have bugs for the drama? Surely this isn’t the whole story.
  • duncan-donuts 4y ago
    I think OP was trying to say something like, "Management replaced our team because we were quiet and didn't rally around problems like other teams". I do think it was poorly worded and a bit jaded.
  • wwweston 4y ago
    Management can be assured of worker productivity by evidence of activity or evidence of output. In some situations (probably a lot of software engineering situations) output is difficult to measure, and so the habit of tuning in to activity is adopted instead. Some may even forget the difference.
    • JohnFen 4y ago
      > output is difficult to measure But in software engineering, it isn't difficult to measure at all. We're developing a deliverable. You can measure if the deliverable happens on time and with acceptable quality.
      • californical 4y ago
        Not exactly. Your approach works if everyone knows ahead-of-time the exact amount of effort that something will take. In software, it’s rarely the case that a project complexity is fully understood from the beginning; at best, you can make a ballpark estimate. If you have a rock solid approach and design a system with little drama, which is released on time and with high quality, then you look like it wasn’t a very ambitious project. Or easier than expected. Maybe your team padded the estimates a lot and didn’t need to work that hard. If you are putting in late nights and weekends and constantly fighting to get features working, maybe management thinks that the project was way harder than expected. They’re so lucky to have someone as hardworking as you or this project never would’ve been done! Obviously there is a flaw in the logic — it’s possible that those assumptions are correct, and person 1 really was under-ambitious and person 2 is an incredible and dedicated engineer working on crazy hard problems. But it’s also possible that the first engineer was just better, and the second had terrible system design skills and constant spaghetti code that made a simple project seem complex. It can be really hard to tell the difference. Even if both end up delivering on time, the second looks more ambitious, like they’re taking on harder problems.
        • JohnFen 4y ago
          > Your approach works if everyone knows ahead-of-time the exact amount of effort that something will take. No, it really doesn't require that. But we're getting into the topic of project planning, which is a larger subject than we can tackle in the comments here. Fortunately, this is a topic discussed in great detail elsewhere. > But it’s also possible that the first engineer was just better, and the second had terrible system design skills and constant spaghetti code that made a simple project seem complex. Right, I was intending to cover that with my "acceptable quality" conditional.
          • l33t233372 4y ago
            Where is project planning discussed well, in your opinion?
            • 4y ago
  • HeyLaughingBoy 4y ago
    When the org that I was working for was doing TSP (https://segoldmine.ppi-int.com/node/67631 https://segoldmine.ppi-int.com/node/67631), our coach told me a story about another team and an engineer I knew very well. He got high praise for all the late nights and weekends he worked to get a product out the door. But on analysis of what he was working on and the bugs he had to deal with, if he had taken the TSP approach that my team was using, most of those bugs would never have been created in the first place and the product would have been finished much sooner. Drama gets noticed, just quietly ticking along, producing high-quality output really doesn't.
    • PaulKeeble 4y ago
      Its exactly this. Their approach produced lots of weekends and late nights and broken releases and they could be seen to be fixing things and responding like lightening to every issue. My team on the other hand was 9 to 5, everything just worked when we released it with few bugs and we and the business worked like normal human beings. The problem is that its invisible and there are no heroics, its just good solid engineering. Which considering it was a back end system for a bank is the right way for things to be. Management likes people who make heroic effort even if they are the cause of needing it they are visibly working hard even if they are making less progress.
    • ryandrake 4y ago
      Exactly. I've worked at software companies where the executives wouldn't believe any work was being done unless they could visibly see activity, and hear the "buzz" of "people doing things" and feel the drama of production emergencies and heroics. It felt like they were listening for intense movie-like typing on keyboards, watching for theatrics in front of whiteboards, project leads calling for standups, and so on. Those were the teams truly DoingThings™ and those teams were rewarded for their performance art. The team silently plugging away at their desks in chat, while they calmly deployed another build that passed all test automation--I'm not sure if leadership even knew who they were. EDIT: These folks almost certainly overlap with the ones pining for Return To Office instead of remote work: They miss the "hum" and "buzz" of SeriousBusiness™ happening all around them in the physical office.
    • 4y ago
  • T-Winsnes 4y ago
    It comes down to "the squeaky wheel gets the grease", if you sit quietly in the corner and get things done, and there are no problems with the work you do. You will be invisible to everyone outside those you work with directly. The person to makes a lot of noise, good or bad, is noticed Everyone say they want well engineered solutions, but in practice it's not what makes an impact.
• ericmcer 4y ago
  Yeah the only way to get rewarded for working on resiliency like this is if you do it very loudly. If you spend an extra day or two refactoring a feature so that it is more robust you will be looked at poorly versus the engineer who slaps a few more conditionals onto it to keep it working. Raise the issue with higher ups, maybe create some fancy charts about lost engineering time in the future, spin up specific tickets for refactoring, turn it into a two week project and you will get recognition. Management loves a chart about improving X by N% almost more than a shiny feature.
  • JohnFen 4y ago
    Most of the places I've worked weren't like this, but a couple were. I quit those positions on the grounds that I was clearly a poor cultural fit.
  • nonethewiser 4y ago
    If they dont like that you took more time to refactor instead of bandaid, they arent going to respond well to you boasting about it for weeks.
• sroussey 4y ago
  I like to rate engineers across various categories, but really I found that there were two general classes of great engineers: fast and slow. For marketing tests, etc., the whole team would be fast engineers. For payments, the whole team would be slow. Everything else I would try to have tension -- a mix of the two so they learn and appreciate each other. This is condensing a multidimensional vector into just a line, but effective enough to explain to non-engineers. If you are a precision (slow) engineer, then you are more likely on the backend, more likely to write tests, more likely to avoid costly errors. This will be wasted on some kinds of tasks (trying out new things, for example, but as a class these tasks can generate a lot of management interest). The only kind of management interest in the precision stuff is failure, and it is usually doomsday failure. That said, security is always present, and I am noodling ideas right now on how to eliminate whole classes of security and privacy issues while making it even easier faster for all engineer types.
  • Hermitian909 4y ago
    > two general classes of great engineers: fast and slow ... Everything else I would try to have tension -- a mix of the two To add to this, if you are one of these engineers a great career hack is to find an engineer you respect on the other end of the spectrum and partner with them. As a "slow" engineer I make sure our key interfaces are abstractions are correct and my "fast" partner ensures that everything gets shipped and that I don't sweat the small stuff. This has lead to a lot more successful and impactful projects than I could manage on my own or with another "slow" engineer.
  • JohnFen 4y ago
    Interesting. I never thought about dividing engineers this way. I do know that the two best engineers I've ever met were slow. Not just slow in working, but slow in everything. Speaking, moving, etc. This frustrated some people, but these engineers saved everyone else much time and frustration because their work was rock solid, well thought-out, and complete. And, once you accounted for time spent bug-fixing and validating things, they weren't actually slow.
    • dcow 4y ago
• fzeindl 4y ago
  I agree. I have written a blog article about this: https://fabianzeindl.com/posts/the-codequality-pyramid#testability https://fabianzeindl.com/posts/the-codequality-pyramid#testa...
  • lcall 4y ago
    Excellent article; thank you. I have written similarly in my notes, focused on the code maintainability (or component quality) part, but not in a shareable format yet. It is a sort of maturity model (I hope eventually to turn it into a grid with practices on the left side and levels across the top, so the RH column is near-ideal behaviors personally and organizationally). But the concept begins with a self-code review for each commit or PR, followed a code review from someone else (who ideally was not part of the design discussion) who asks questions like: is this code easily understood for a new maintainer? would I want to maintain this? How was it tested? Is any coordination in order with other stakeholders, or does documenation need to be updated? Is it clear why the code exists and the tests do what they do? And then improving over time. This came up for me after repeatedly encountering code that no one could understand but the original author ("bus count of one", or if one person gets hit by a bus we would be in trouble), or that was needlessly complex for the enjoyment of complexity (really), or that had tests where no one knew why they tested for certain behavior (until fortunately someone else came back from vacation after I was about to commit a change to the test). Etc, all for comprehensibility and maintainability, and reliability. Ps: overall, this is one of the most enjoyable HN discussions I have seen. pps: I also one started encouraging our team and others to maintain a set of wiki pages, listing all projects for which we were responsible, then for each one at least one simple page of documenation listing things like who are the stakeholders, where is the source code, any odd build or deployment steps, the key (one sentence) inputs and outputs, where it runs, who does backups of what, etc. Before that it was haphazard. This is short of a real ops manual and could be done better of course, and would change as other org. practices change, but it was far better than nothing, and could be created in 10 minutes from a template. Great for bringing new teammates on board. I think an organization should have something like that for the whole org, listing teams and each team having such a page, as well as listing everything essential that a new person should know rather than relying on haphazard cultural transmission. We had rules like "no new debt" that were well-adopted and began to slip out of the culture as new people joined.
• testfrequency 4y ago
  I saw this often. The heroes who are up late, solving a page or mitigating an outage are often the ones remembered and rewarded. Meanwhile, a dependable, resourceful, and independent IC who “picks up trash on the floor”, promotes good work habits, and is dead reliable - no praise, they are just “doing their job” Managers and leaders like drama, most staff and senior engineers gravitate to drama and love talking about it. Their day to to day work, often subpar and often not team players
  • hammock 4y ago
    Everyone is selling something, and stories are how you do it. Can’t have a story without some drama
    • agumonkey 4y ago
      This is why I need a different job. This pattern sucks energy out of me. I need military minded coders.
      • GoodDreams 4y ago
        I’m interested in understanding what it means to be a military minded coder.
        • californical 4y ago
          It’s likely a reference to the story in the linked article
        • alisonatwork 4y ago
          Slow is smooth. Smooth is fast.
          • agumonkey 4y ago
            Festina lente
        • 4y ago
• thefaux 4y ago
  Yes, there are often a lot of organizational barriers that inhibit doing the right thing. Most organizations like to talk a big game about how innovative they are, but they really just want to keep doing the same things that they've always done. I have been in the situation where over a period of time I inherited a lot of code that was written by my boss and absolutely critical to our business. Over time I began to see that they had made a number of poor design choices that made it very difficult to work with some of these internal frameworks. For example, there was a lot of global state passed around that kind of worked in production, but made it impossible to run tests in parallel (as well as making some of them flaky even when run serially). I introduced an internal dependency injection framework that leveraged some unusual host language features to remove the global state without painfully having to add a bunch of parameters to every function/class that would have to be passed around. This allowed the tests to run in parallel and completely removed the flaky failures. At the time I introduced the changes, they reduced the time to run all the tests from about 3 minutes to 20 seconds with no flaky failures. This was a major quality of life improvement not only for me, but for everyone else working on this part of the system. Was this work rewarded? No. What I didn't consider when undertaking the refactoring was that I was implicitly making my boss look bad by fixing the systemic problems that they'd introduced. Even worse, I used techniques that were unfamiliar to them which was a further blow to their ego, though they wouldn't have admitted this at the time. Instead, they complained that they didn't understand some of my techniques rather than seriously try to learn what I'd done and appreciate the benefits that it brought. Ultimately, it became soul crushing to realize that they were more invested in doing things the way they'd always done them then to learn how to do things better (or at least offer constructive feedback beyond "I don't superficially understand this so it must be bad"). When you are in this situation, advancement becomes almost impossible because you have now become a threat to your boss, who will never make the mistake of allowing you to be promoted to their level or beyond. I still think we should all strive to do the best work that we can because ultimately you should feel proud of the work that you've done. But this often comes with a major cost (which may ultimately be that you are forced to leave the organization).
• Sn0wCoder 4y ago
  Just wanted to say feel your pain. My team was recently disbanded to other teams because our product has no issues and we were considered too slow. Fast forward just released a shiny new front end that is buggy and slow. 4 hot fixes in a month. At my review they said see getting you on a new team was all it took to speed you up. Well ya I guess no unit testing, and no documentation. At least I got the team to start using the linter. Another benefit is I have more family time and been working on a side project to ensure I don’t forget how to do things correctly lol. Always think this place is going to be different but timelines are king and those who hit them will be rewarded. Also can’t get much faster than 4 releases in a month which is all the boss sees on paper…