6 ms·

Try to fix it one level deeper

129 points by Smaug123 2y ago

hoherd 2y ago
This seems like the code implementation way of shifting left. https://news.ycombinator.com/item?id=38187879 https://news.ycombinator.com/item?id=38187879
andai 2y ago
I was reading about NASA's software engineering practices. When they find a bug, they don't just fix the bug, they fix the engineering process that allowed the bug to occur in the first place.
- deleted 2y ago
  [deleted]
- anotherhue 2y ago
  Maintenance is never as rewarded as new features, there's probably some MBA logic behind it to do with avoiding commoditisation. It's true in software, it's true in physical infrastructure (read about the sorry state of most dams). Until we root cause that process I don't see much progress coming from this direction, on the plus side CS principles are making their way into compilers. We're a long way from C.
  - giantg2 2y ago
    "Maintenance is never as rewarded as new features," And security work is rewarded even less!
    - riknos314 2y ago
      > And security work is rewarded even less While I do recognize that this is a pervasive problem, it seems counter-intuitive to me based on the tendency of the human brain to be risk averse. It raises an interesting question of "why doesn't the risk of security breaches trigger the emotions associated with risk in those making the decision of how much to invest in security?". Downstream of that is likely "Can we communicate the security risk story in a way that more appropriately triggers the associated risk emotions?"
      - giantg2 2y ago
        The people making the decision don't have a direct negative impact. Someone's head might role, but that's usually far up the chain where the comp and connections are high enough to not care. The POs making the day to day decisions are under more pressure for new features than they are for security.
      - SAI_Peregrinus 2y ago
        What is the consequence for security breaches? Usually some negative press everyone forgets in a week. Maybe a lost sale or two, but that's hard to measure. If you're exceedingly unlucky, an inconsequential fine. At worst paying for two years of credit monitoring for your users. What's the risk? The stock price will be back up by next week.
      - amonon 2y ago
        easier to consider people to tend towards conservative rather than risk averse. if we were truly risk averse, society would be very different.
      - chii 2y ago
        the easy explanation is that the cost of a breach is externalized, so decision makers gain benefit from savings in not investing in security. Look at the crowdstrike failure as a recent example, but there's plenty more in the past.
  - hedvig23 2y ago
    Speaking of digging deeper, can you expand on that theory on why focus/man hours spent on maintenance leads to commoditization and why a company wants to avoid that?
    - anotherhue 2y ago
      Top of my head, new things have unbounded potential, existing ones have known potential. We assume the new will be better. I think it's part of the reason stocks almost always dip after positive earnings reports. No matter how positive it's always less than idealised. You might think there's a trick where you can sell maintenance as a new thing but you've just invented the unnecessary rewrite. To answer your question more directly, once something has been achieved it's safe to assume someone else can achieve it also, so the focus turns to the new thing. Why else would we develop hydrogen or neutron bombs when we already had perfectly good fission ones (they got commoditised).
    - asdff 2y ago
      Given enough iteration with the same incentives, two engineering teams might end up with the same sort of product overall. We see this with airframes. We established the prototypical airframe for the commercial airliner in the 1950s and haven't changed it in 70 years since. This is good for the airline but bad for the aircraft manufacturer. The airline can now choose between boeing or airbus or anyone else for their product. If boeing had some novel plane design that wasn't copied the world over, then the airline company would be beholden to them alone.
      - CRConrad 2y ago
        Sounds like automobiles are -- have been, until now -- an even more long-lasting example. The basic recipe has been the same for over a century: Four wheels, two to five doors, steering wheel and other controls at one of the front seats, internal combustion engine. Only now is one of those components, the engine, being changed out. And looking at that from this perspective, it's apparently not as humongous a change as many are now claiming: For one thing, it's just one of the four (or more?) parts of the basic recipe; for another, it's not all that new -- electric motors are in wide use elsewhere, and were one of the alternatives in rather wide use in cars, too, before the industry settled on the current recipe ~a hundred years ago. Anyway, the "beholden to them alone" bit doesn't seem to apply to the automotive industry, since pretty much all manufacturers are in on the switch-to-electrics idea. Which begs the question: Would there really be so much of a customer lock-in in the aeronautic space either? Just like the idea of driving cars by electric motors is unpatentable, what could Boeing (or anyone else) come up with in the basic design of an aeroplane that isn't too general an idea to be protectable, or (most probably) hasn't been tried before?
- xelxebar 2y ago
  This is such a powerful frame of mind. Bugs, software architecture, tooling choices, etc. all happen within organizational, social, political, and market machinery. A bug isn't just a technical failure, but a potential issue with the meta-structures in which the software is embedded. Code review is one example of addressing the engineering process, but I also find it very helpful to consider business and political processes as well. Granted, NASA's concerns are very different than that of most companies, but as engineers and consultants, we have leeway to choose where and how to address bugs, beyond just the technical and immediate dev habits. Soft skills matter hard.
  - asdff 2y ago
    It makes you wonder if there's been work designing software that is resilient to bugs. Maybe you can test this by writing a given function in a variety of different ways, simulate some type of bug (fat fingering is probably easiest), and compare outputs. Some of these functions might not work at all. Some might spit out the wrong result. But then there will probably be a few that are written in such a way to get very close to the true result, and maybe that variance is acceptable for your purposes. Given how we currently write code (in english in a way a human can read it) maybe its not so realistic. But if we get to the point with our generative code where you can generate good quality machine code without having it transmuted to human readable code for human verification, then this is how we would be operating: looking at distributions of results from a billion putative functions.
- toolz 2y ago
  To that example though, is NASA really the pinnacle of achievement in their field? Sure, it's not a very competitive field (e.g. compared to something like the restaurant industry) and most of their existence has been about r&d for tech there wasn't really a market for yet, but still spaceX comes along and in a fraction of the time they're landing and reusing rockets making space launches more attainable and significantly cheaper. I'm hoping that example holds up, but I'm not well versed in that area so it may be a terrible counter-example but my overarching point is this: overly engineered code often produces less value than quickly executed code. We're not in the business of making computers do things artfully just for the beauty of the rigor and correctness of our systems. We're doing it to make computers do useful thing for humanity. You may think that spending an extra year perfecting a pace-maker might end up saving lives, but what if more people die in the year before you go to market than would've ended up dying had you launched with something almost perfect, but with potential defects? Time is expensive in so many more ways than just capital spent.
  - the_other 2y ago
    SpaceX came along decades after NASA’s most famous projects. Would SpaceX have been able to do what they did if NASA hadn’t engineered to their standard earlier on? My argument (and I’m just thought experimenting here) is that without NASA’s rigor, their programmes would have failed. Public support, and thus the market for soace projects, would have dried up before SpaceX was able to “do it faster”. (Feel free to shoot this down: I wasn’t there and I havn’t read any deep histories of the conpanies. I’m just brainstorming to explore the problem space)
    - euroderf 2y ago
      So I wonder if SpaceX has a lending library of all of the relevant (quality-related) NASA documents, printed out on dead trees. For light lunchtime reading.
  - sfn42 2y ago
    The fallacy here is that you're assuming that doing things right takes more time. Doing things right takes less time in my experience. You spend a little more time up front to figure out the right way to do something, and a lot of the time that investment pays dividends. The alternative is to just choose the quickest fix every time until eventually your code is so riddled with quick fixes that nobody knows how it works and it's impossible to get anything done.
- sendfoods 2y ago
  Which blog/post/book was this? Thanks
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
brody_hamer 2y ago
I learned a similar mantra that I keep returning to: “there’s never just one problem.” - How did this bug make it to production? Where’s the missing unit test? Code review? - Could the error have been handled automatically? Or more gracefully?
niccl 2y ago
In the course of interviewing a bunch of developers, and employing a few of them, I've concluded that this ability/inclination/something to do this deeper digging is one of the things I prize most in a developer. They have to know when to go deep and when not to, though, and that's sometimes a hard balancing act. I've never found a good way of screening for the ability, and more, for when not to go deep, because everyone will come up with some example if you ask, and it's not the sort of thing that I can see highlighting in a coding test (and _certainly_ not in a leet-code test!). If anyone has any suggestions on how to uncover it during the hiring process I'd be ecstatic!
- deleted 2y ago
  [deleted]
- giantg2 2y ago
  "I've concluded that this ability/inclination/something to do this deeper digging is one of the things I prize most in a developer." Where have you been all my life? It seems most of teams I've been on value speed over future proofing bugs. The systems thinking approach is rare. If you want to test for this, you can create a PR for a fake project. Make sure the project runs but has error, code smells, etc. Have a few things like they talk about in the article, like a message of being out of disk space but missing critical message/logging infrastructure to cover other scenarios. The best part is, you can use the same PR for all levels that you're hiring for by expecting senior to get X% of the bugs, mids to get X/2% and noobs to get X/4%.
  - niccl 2y ago
    That's a really good idea. Thanks
  - jerf 2y ago
    "It seems most of teams I've been on value speed over future proofing bugs." So, obviously, if one team is future proofing bugs, and the other team just blasts out localized short-term fixes as quickly as possible, there will come a point where the first team will overtake the second, because the second team's velocity will by necessity has to slow down more than the first as the code base grows. If the crossover point is ten years hence, then it only makes sense to be the second team. However, what I find a bit horrifying as a developer is that my estimate of the crossover point keeps coming in. When I'm working by myself on greenfield code, I'd put it at about three weeks; yes, I'll go somewhat faster today if I just blast out code and skip the unit tests, but it's only weeks before I'm getting bitten by that. Bigger teams may have a somewhat farther cross over point, but it's still likely to be small single-digit months. There is of course overdoing it and being too perfectionist, and that does get some people, but the people, teams, managers, and companies who always vote for the short term code blasting simply have no idea how much performance they are leaving on the table almost immediately. Established code bases are slower to turn, naturally. But even so, I still think the constant short-term focus is vastly more expensive than those who choose it understand. And I don't even mean obvious stuff like "oh, you'll have more bugs" or "oh, it's so much harder to on board", even if that's true... no, I mean, even by the only metric you seem care about, the team that takes the time to fix fundamental issues and invests in better logging and metrics and all those things you think just slow you down can also smoke you on dev speed after a couple of months... and they'll have the solid code base, too! "Make sure the project runs but has error, code smells, etc." It is a hard problem to construct a test for this but it would be interesting to provide the candidate some code that compiles with warnings and just watch them react to the warnings. You may not learn everything you need but it'll certainly teach you something.
Cpoll 2y ago
This kind of reminds me of https://en.m.wikipedia.org/wiki/Five_whys https://en.m.wikipedia.org/wiki/Five_whys.
peter_d_sherman 2y ago
>"There’s a bug! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the bug is valuable, and it is pointing in the direction of a bigger related problem." That is an absolutely stellar quote! It's also more broadly applicable to life / problem solving / goal setting (if we replace the word 'bug' with 'problem' in the above quote): "There’s a problem! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the problem is valuable, and it is pointing in the direction of a bigger related problem." In other words, in life / problem solving / goal setting -- smaller problems can be really valuable, because they can be pointers/signs/omens/subcases/indicators of/to larger surrounding problems in larger surrounding contexts... (Just like bugs can be, in Software Engineering!) Now if only our political classes (on both sides!) could see the problems that they typically see as problems -- as effects not causes (because that's what they all are, effects), of as-of-yet unseen larger problems, of which those smaller problems are pointers to, "hints at", subcases of, "indicators of" (use whatever terminology you prefer...) Phrased another way, in life/legislation/problem solving/Software Engineering -- you always have to nail down first causes -- otherwise you're always in "Effectsville"... :-) You don't want to live in "Effectsville" -- because anything you change will be changed back to what it was previously in the shortest time possible, because everything is an effect in Effectsville! :-) Legislating something that is seen that is the effect of another, greater, as-of-yet unseen problem -- will not fix the seen problem! Finally, all problems are always valuable -- but if and only if their surrounding context is properly perceived... So, an an excellent observation by the author, in the context of Software Engineering!
raphlinus 2y ago
The title immediately brings to mind the Osterhout classic, "Always Measure One Level Deeper", [1], and I imagine was probably inspired by it. Also worth revisiting. [1]: https://cacm.acm.org/research/always-measure-one-level-deeper/ https://cacm.acm.org/research/always-measure-one-level-deepe...
- matklad 2y ago
  I was not actually aware of the paper, and it is indeed pure gold, thanks for linking it. It is _also_ extremely timely, as with my current project (TigerBeetle), we are exactly at the point of transition from first-principles back-of-the-envelope performance architecture to measurement-driven performance engineering, and all the advice here is directly applicable!
- WillAdams 2y ago
  See the Book review: A Philosophy of Software Design (2020) (johz.bearblog.dev) https://news.ycombinator.com/item?id=27686818 https://news.ycombinator.com/item?id=27686818 for more in this vein.
Terr_ 2y ago
IMO it may be worth distinguishing between: 1. Diagnosing the "real causes" one level deeper 2. Implementing a "real fix" fix one level deeper Sometimes they have huge overlap, but the first is much more consistently-desirable. For example, it might be the most-practical fix is to add some "if this happens just retry" logic, but it would be beneficial to know--and leave a comment--that it occurs because of a race condition.
KaiserPro 2y ago
You need to choose your rabbit holes carefully. In large and complex codebases, its often more pragmatic to build a guard in your local area against that bug, than following the bug all the way downthe stack. Its not optimal, and doesn't make the system better as a whole. but its the only way to get things done. That doesn't mean you should be silent though, you do need to contact the team that looks after that part of the system
cantSpellSober 2y ago
In enterprise monorepos I find this hard because "one level deeper" is often code you don't own. Fun article, good mantra!
kayvulpe 2y ago
Echoes of "Hal fixing a light bulb" from Malcolm in the Middle

129 points by Smaug123 2y ago

This seems like the code implementation way of shifting left. https://news.ycombinator.com/item?id=38187879 https://news.ycombinator.com/item?id=38187879

andai 2y ago

I was reading about NASA's software engineering practices. When they find a bug, they don't just fix the bug, they fix the engineering process that allowed the bug to occur in the first place.

deleted 2y ago
[deleted]
anotherhue 2y ago
Maintenance is never as rewarded as new features, there's probably some MBA logic behind it to do with avoiding commoditisation. It's true in software, it's true in physical infrastructure (read about the sorry state of most dams). Until we root cause that process I don't see much progress coming from this direction, on the plus side CS principles are making their way into compilers. We're a long way from C.
- giantg2 2y ago
  "Maintenance is never as rewarded as new features," And security work is rewarded even less!
  - riknos314 2y ago
    > And security work is rewarded even less While I do recognize that this is a pervasive problem, it seems counter-intuitive to me based on the tendency of the human brain to be risk averse. It raises an interesting question of "why doesn't the risk of security breaches trigger the emotions associated with risk in those making the decision of how much to invest in security?". Downstream of that is likely "Can we communicate the security risk story in a way that more appropriately triggers the associated risk emotions?"
    - giantg2 2y ago
      The people making the decision don't have a direct negative impact. Someone's head might role, but that's usually far up the chain where the comp and connections are high enough to not care. The POs making the day to day decisions are under more pressure for new features than they are for security.
    - SAI_Peregrinus 2y ago
      What is the consequence for security breaches? Usually some negative press everyone forgets in a week. Maybe a lost sale or two, but that's hard to measure. If you're exceedingly unlucky, an inconsequential fine. At worst paying for two years of credit monitoring for your users. What's the risk? The stock price will be back up by next week.
    - amonon 2y ago
      easier to consider people to tend towards conservative rather than risk averse. if we were truly risk averse, society would be very different.
    - chii 2y ago
      the easy explanation is that the cost of a breach is externalized, so decision makers gain benefit from savings in not investing in security. Look at the crowdstrike failure as a recent example, but there's plenty more in the past.
- hedvig23 2y ago
  Speaking of digging deeper, can you expand on that theory on why focus/man hours spent on maintenance leads to commoditization and why a company wants to avoid that?
  - anotherhue 2y ago
    Top of my head, new things have unbounded potential, existing ones have known potential. We assume the new will be better. I think it's part of the reason stocks almost always dip after positive earnings reports. No matter how positive it's always less than idealised. You might think there's a trick where you can sell maintenance as a new thing but you've just invented the unnecessary rewrite. To answer your question more directly, once something has been achieved it's safe to assume someone else can achieve it also, so the focus turns to the new thing. Why else would we develop hydrogen or neutron bombs when we already had perfectly good fission ones (they got commoditised).
  - asdff 2y ago
    Given enough iteration with the same incentives, two engineering teams might end up with the same sort of product overall. We see this with airframes. We established the prototypical airframe for the commercial airliner in the 1950s and haven't changed it in 70 years since. This is good for the airline but bad for the aircraft manufacturer. The airline can now choose between boeing or airbus or anyone else for their product. If boeing had some novel plane design that wasn't copied the world over, then the airline company would be beholden to them alone.
    - CRConrad 2y ago
      Sounds like automobiles are -- have been, until now -- an even more long-lasting example. The basic recipe has been the same for over a century: Four wheels, two to five doors, steering wheel and other controls at one of the front seats, internal combustion engine. Only now is one of those components, the engine, being changed out. And looking at that from this perspective, it's apparently not as humongous a change as many are now claiming: For one thing, it's just one of the four (or more?) parts of the basic recipe; for another, it's not all that new -- electric motors are in wide use elsewhere, and were one of the alternatives in rather wide use in cars, too, before the industry settled on the current recipe ~a hundred years ago. Anyway, the "beholden to them alone" bit doesn't seem to apply to the automotive industry, since pretty much all manufacturers are in on the switch-to-electrics idea. Which begs the question: Would there really be so much of a customer lock-in in the aeronautic space either? Just like the idea of driving cars by electric motors is unpatentable, what could Boeing (or anyone else) come up with in the basic design of an aeroplane that isn't too general an idea to be protectable, or (most probably) hasn't been tried before?
xelxebar 2y ago
This is such a powerful frame of mind. Bugs, software architecture, tooling choices, etc. all happen within organizational, social, political, and market machinery. A bug isn't just a technical failure, but a potential issue with the meta-structures in which the software is embedded. Code review is one example of addressing the engineering process, but I also find it very helpful to consider business and political processes as well. Granted, NASA's concerns are very different than that of most companies, but as engineers and consultants, we have leeway to choose where and how to address bugs, beyond just the technical and immediate dev habits. Soft skills matter hard.
- asdff 2y ago
  It makes you wonder if there's been work designing software that is resilient to bugs. Maybe you can test this by writing a given function in a variety of different ways, simulate some type of bug (fat fingering is probably easiest), and compare outputs. Some of these functions might not work at all. Some might spit out the wrong result. But then there will probably be a few that are written in such a way to get very close to the true result, and maybe that variance is acceptable for your purposes. Given how we currently write code (in english in a way a human can read it) maybe its not so realistic. But if we get to the point with our generative code where you can generate good quality machine code without having it transmuted to human readable code for human verification, then this is how we would be operating: looking at distributions of results from a billion putative functions.
toolz 2y ago
To that example though, is NASA really the pinnacle of achievement in their field? Sure, it's not a very competitive field (e.g. compared to something like the restaurant industry) and most of their existence has been about r&d for tech there wasn't really a market for yet, but still spaceX comes along and in a fraction of the time they're landing and reusing rockets making space launches more attainable and significantly cheaper. I'm hoping that example holds up, but I'm not well versed in that area so it may be a terrible counter-example but my overarching point is this: overly engineered code often produces less value than quickly executed code. We're not in the business of making computers do things artfully just for the beauty of the rigor and correctness of our systems. We're doing it to make computers do useful thing for humanity. You may think that spending an extra year perfecting a pace-maker might end up saving lives, but what if more people die in the year before you go to market than would've ended up dying had you launched with something almost perfect, but with potential defects? Time is expensive in so many more ways than just capital spent.
- the_other 2y ago
  SpaceX came along decades after NASA’s most famous projects. Would SpaceX have been able to do what they did if NASA hadn’t engineered to their standard earlier on? My argument (and I’m just thought experimenting here) is that without NASA’s rigor, their programmes would have failed. Public support, and thus the market for soace projects, would have dried up before SpaceX was able to “do it faster”. (Feel free to shoot this down: I wasn’t there and I havn’t read any deep histories of the conpanies. I’m just brainstorming to explore the problem space)
  - euroderf 2y ago
    So I wonder if SpaceX has a lending library of all of the relevant (quality-related) NASA documents, printed out on dead trees. For light lunchtime reading.
- sfn42 2y ago
  The fallacy here is that you're assuming that doing things right takes more time. Doing things right takes less time in my experience. You spend a little more time up front to figure out the right way to do something, and a lot of the time that investment pays dividends. The alternative is to just choose the quickest fix every time until eventually your code is so riddled with quick fixes that nobody knows how it works and it's impossible to get anything done.
sendfoods 2y ago
Which blog/post/book was this? Thanks

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

brody_hamer 2y ago

I learned a similar mantra that I keep returning to: “there’s never just one problem.” - How did this bug make it to production? Where’s the missing unit test? Code review? - Could the error have been handled automatically? Or more gracefully?

niccl 2y ago

In the course of interviewing a bunch of developers, and employing a few of them, I've concluded that this ability/inclination/something to do this deeper digging is one of the things I prize most in a developer. They have to know when to go deep and when not to, though, and that's sometimes a hard balancing act. I've never found a good way of screening for the ability, and more, for when not to go deep, because everyone will come up with some example if you ask, and it's not the sort of thing that I can see highlighting in a coding test (and _certainly_ not in a leet-code test!). If anyone has any suggestions on how to uncover it during the hiring process I'd be ecstatic!

deleted 2y ago
[deleted]
giantg2 2y ago
"I've concluded that this ability/inclination/something to do this deeper digging is one of the things I prize most in a developer." Where have you been all my life? It seems most of teams I've been on value speed over future proofing bugs. The systems thinking approach is rare. If you want to test for this, you can create a PR for a fake project. Make sure the project runs but has error, code smells, etc. Have a few things like they talk about in the article, like a message of being out of disk space but missing critical message/logging infrastructure to cover other scenarios. The best part is, you can use the same PR for all levels that you're hiring for by expecting senior to get X% of the bugs, mids to get X/2% and noobs to get X/4%.
- niccl 2y ago
  That's a really good idea. Thanks
- jerf 2y ago
  "It seems most of teams I've been on value speed over future proofing bugs." So, obviously, if one team is future proofing bugs, and the other team just blasts out localized short-term fixes as quickly as possible, there will come a point where the first team will overtake the second, because the second team's velocity will by necessity has to slow down more than the first as the code base grows. If the crossover point is ten years hence, then it only makes sense to be the second team. However, what I find a bit horrifying as a developer is that my estimate of the crossover point keeps coming in. When I'm working by myself on greenfield code, I'd put it at about three weeks; yes, I'll go somewhat faster today if I just blast out code and skip the unit tests, but it's only weeks before I'm getting bitten by that. Bigger teams may have a somewhat farther cross over point, but it's still likely to be small single-digit months. There is of course overdoing it and being too perfectionist, and that does get some people, but the people, teams, managers, and companies who always vote for the short term code blasting simply have no idea how much performance they are leaving on the table almost immediately. Established code bases are slower to turn, naturally. But even so, I still think the constant short-term focus is vastly more expensive than those who choose it understand. And I don't even mean obvious stuff like "oh, you'll have more bugs" or "oh, it's so much harder to on board", even if that's true... no, I mean, even by the only metric you seem care about, the team that takes the time to fix fundamental issues and invests in better logging and metrics and all those things you think just slow you down can also smoke you on dev speed after a couple of months... and they'll have the solid code base, too! "Make sure the project runs but has error, code smells, etc." It is a hard problem to construct a test for this but it would be interesting to provide the candidate some code that compiles with warnings and just watch them react to the warnings. You may not learn everything you need but it'll certainly teach you something.

Cpoll 2y ago

This kind of reminds me of https://en.m.wikipedia.org/wiki/Five_whys https://en.m.wikipedia.org/wiki/Five_whys.

peter_d_sherman 2y ago

>"There’s a bug! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the bug is valuable, and it is pointing in the direction of a bigger related problem." That is an absolutely stellar quote! It's also more broadly applicable to life / problem solving / goal setting (if we replace the word 'bug' with 'problem' in the above quote): "There’s a problem! And it is sort-of obvious how to fix it. But if you don’t laser-focus on that, and try to perceive the surrounding context, it turns out that the problem is valuable, and it is pointing in the direction of a bigger related problem." In other words, in life / problem solving / goal setting -- smaller problems can be really valuable, because they can be pointers/signs/omens/subcases/indicators of/to larger surrounding problems in larger surrounding contexts... (Just like bugs can be, in Software Engineering!) Now if only our political classes (on both sides!) could see the problems that they typically see as problems -- as effects not causes (because that's what they all are, effects), of as-of-yet unseen larger problems, of which those smaller problems are pointers to, "hints at", subcases of, "indicators of" (use whatever terminology you prefer...) Phrased another way, in life/legislation/problem solving/Software Engineering -- you always have to nail down first causes -- otherwise you're always in "Effectsville"... :-) You don't want to live in "Effectsville" -- because anything you change will be changed back to what it was previously in the shortest time possible, because everything is an effect in Effectsville! :-) Legislating something that is seen that is the effect of another, greater, as-of-yet unseen problem -- will not fix the seen problem! Finally, all problems are always valuable -- but if and only if their surrounding context is properly perceived... So, an an excellent observation by the author, in the context of Software Engineering!

raphlinus 2y ago

The title immediately brings to mind the Osterhout classic, "Always Measure One Level Deeper", [1], and I imagine was probably inspired by it. Also worth revisiting. [1]: https://cacm.acm.org/research/always-measure-one-level-deeper/ https://cacm.acm.org/research/always-measure-one-level-deepe...

matklad 2y ago
I was not actually aware of the paper, and it is indeed pure gold, thanks for linking it. It is _also_ extremely timely, as with my current project (TigerBeetle), we are exactly at the point of transition from first-principles back-of-the-envelope performance architecture to measurement-driven performance engineering, and all the advice here is directly applicable!
WillAdams 2y ago
See the Book review: A Philosophy of Software Design (2020) (johz.bearblog.dev) https://news.ycombinator.com/item?id=27686818 https://news.ycombinator.com/item?id=27686818 for more in this vein.

Terr_ 2y ago

IMO it may be worth distinguishing between: 1. Diagnosing the "real causes" one level deeper 2. Implementing a "real fix" fix one level deeper Sometimes they have huge overlap, but the first is much more consistently-desirable. For example, it might be the most-practical fix is to add some "if this happens just retry" logic, but it would be beneficial to know--and leave a comment--that it occurs because of a race condition.

KaiserPro 2y ago

You need to choose your rabbit holes carefully. In large and complex codebases, its often more pragmatic to build a guard in your local area against that bug, than following the bug all the way downthe stack. Its not optimal, and doesn't make the system better as a whole. but its the only way to get things done. That doesn't mean you should be silent though, you do need to contact the team that looks after that part of the system

cantSpellSober 2y ago

In enterprise monorepos I find this hard because "one level deeper" is often code you don't own. Fun article, good mantra!

kayvulpe 2y ago

Echoes of "Hal fixing a light bulb" from Malcolm in the Middle