3 ms·

Deep familiarity with a complex code base is a thing and is not easily replicated. I have seen this first hand. In my day job - I work on a very popular but com

Deep familiarity with a complex code base is a thing and is not easily replicated. I have seen this first hand. In my day job - I work on a very popular but complex open source project and in my team there is a engineer who has been there since beginning and I am like second person after him. I have noticed that when we get certain class of bugs from customers(and customers want fix NOW) that spans 2-3 layers of the project, only me and him can pick those bugs. Some decisions are there in the code that is not easy to figure out(like why it was done this way) and I have also seen engineers who are founders of this project - point design issues that I could not have thought (and I am doing this for 5 years!). Can we hire someone new to provide design reviews that spans network, storage, kernel? May be - but engineers who created this thing and have been doing this for long time, have got really good at it. To me - some of them seem god like. I know they will probably suck same as me, if they had to start developing a new iOS app or write new game for Nintendo Switch.

deleted 5y ago
[deleted]
sangnoir 5y ago
>...only me and him can pick those bugs Brittle or hard-to-grok code that other engineers avoid is a code smell. Eng leadership/senior engineers ought to prioritize refactoring that code ASAP
- jandrewrogers 5y ago
  There wasn't any implication that the code is "brittle" or hard to grok due to a defect in the software architecture. It may be approximately optimal from the perspective of software maintainability relative to functionality and performance. All code has non-local side effects, but in most software that is buried below the noise floor of inefficiency. Some types of software are intrinsically difficult to understand e.g. due to highly visible behavioral coupling across multiple unrelated parts of otherwise tidy, modular code. This is a strictly correct and intentional design in many cases but it means even a seemingly small code change can only be evaluated for correctness relative to the detailed internals across diverse code components in various runtime contexts. Reasoning about these systems has a very high cognitive load. This is one of the reasons writing competent database kernels is so difficult -- the best designs asymptotically converge on being exquisitely complex and subtle monoliths, even when the code appears clean and modular. In my experience, there are always parts of these code bases that only one or two engineers dare modify even though the code is straightforward, because any local change can't be evaluated for correctness without deep expertise in half the system internals. It really depends on the nature of the software.
  - sangnoir 5y ago
    > There wasn't any implication that the code is "brittle" or hard to grok due to a defect in the software architecture Re-reading my comment, it comes off dismissive - I apologies for that. I agree with most of your points > Some types of software are intrinsically difficult to understand e.g. due to highly visible behavioral coupling across multiple unrelated parts of otherwise tidy, modular code. This is a strictly correct and intentional design in many cases but it means even a seemingly small code change can only be evaluated for correctness relative to the detailed internals across diverse code components in various runtime contexts Sometimes - the above is the case, but other engineers are unable to maintain those hard parts because the information required to do so is in the heads of the experts (the original author, or the 2nd engineer). In those cases, docs may help, but sometimes the code exhibiting unexpected coupling and/or side-effects may need to be further modularized, OR paradoxically, consolidated and abstracted by an easier-to-keep-in-mind abstraction as the "modularization" is illusive since the seemingly independent modules covertly belong together. I may be projecting because of my past experiences: at different points I was the only one who could make changes to specific modules, one because it was brittle and had low test coverage (and prone to race conditions), and the other time, it was because there was non-obvious coupling which resulted in high cognitive load as one had to keep several layers of state in mind. On both occasions, refactoring and improving the abstractions would have made it easier for other engineers to safely work on the code. YMMV if the high-cognitive-load architecture is deliberately chosen - in my case, it was an accident of (commit) history.
  - throwaway4354 5y ago
    Yep this so much. It is not that code is brittle but problem space is hard. What seems like an obvious fix may break other vendors that depend on same behavior etc. Having said all of that. Modesty is a thing and many engineers wouldn't word things the way Manish worded in aforementioned blog.
nikolay 5y ago
I have to agree. I worked on a pretty large Java project, we hired 2-3 capable engineers, but they couldn't grasp the complexity of the problem, not necessarily the code - the code was pretty readable if you know what it does and how.

deleted 5y ago

[deleted]

sangnoir 5y ago

>...only me and him can pick those bugs Brittle or hard-to-grok code that other engineers avoid is a code smell. Eng leadership/senior engineers ought to prioritize refactoring that code ASAP

jandrewrogers 5y ago
There wasn't any implication that the code is "brittle" or hard to grok due to a defect in the software architecture. It may be approximately optimal from the perspective of software maintainability relative to functionality and performance. All code has non-local side effects, but in most software that is buried below the noise floor of inefficiency. Some types of software are intrinsically difficult to understand e.g. due to highly visible behavioral coupling across multiple unrelated parts of otherwise tidy, modular code. This is a strictly correct and intentional design in many cases but it means even a seemingly small code change can only be evaluated for correctness relative to the detailed internals across diverse code components in various runtime contexts. Reasoning about these systems has a very high cognitive load. This is one of the reasons writing competent database kernels is so difficult -- the best designs asymptotically converge on being exquisitely complex and subtle monoliths, even when the code appears clean and modular. In my experience, there are always parts of these code bases that only one or two engineers dare modify even though the code is straightforward, because any local change can't be evaluated for correctness without deep expertise in half the system internals. It really depends on the nature of the software.
- sangnoir 5y ago
  > There wasn't any implication that the code is "brittle" or hard to grok due to a defect in the software architecture Re-reading my comment, it comes off dismissive - I apologies for that. I agree with most of your points > Some types of software are intrinsically difficult to understand e.g. due to highly visible behavioral coupling across multiple unrelated parts of otherwise tidy, modular code. This is a strictly correct and intentional design in many cases but it means even a seemingly small code change can only be evaluated for correctness relative to the detailed internals across diverse code components in various runtime contexts Sometimes - the above is the case, but other engineers are unable to maintain those hard parts because the information required to do so is in the heads of the experts (the original author, or the 2nd engineer). In those cases, docs may help, but sometimes the code exhibiting unexpected coupling and/or side-effects may need to be further modularized, OR paradoxically, consolidated and abstracted by an easier-to-keep-in-mind abstraction as the "modularization" is illusive since the seemingly independent modules covertly belong together. I may be projecting because of my past experiences: at different points I was the only one who could make changes to specific modules, one because it was brittle and had low test coverage (and prone to race conditions), and the other time, it was because there was non-obvious coupling which resulted in high cognitive load as one had to keep several layers of state in mind. On both occasions, refactoring and improving the abstractions would have made it easier for other engineers to safely work on the code. YMMV if the high-cognitive-load architecture is deliberately chosen - in my case, it was an accident of (commit) history.
- throwaway4354 5y ago
  Yep this so much. It is not that code is brittle but problem space is hard. What seems like an obvious fix may break other vendors that depend on same behavior etc. Having said all of that. Modesty is a thing and many engineers wouldn't word things the way Manish worded in aforementioned blog.

nikolay 5y ago

I have to agree. I worked on a pretty large Java project, we hired 2-3 capable engineers, but they couldn't grasp the complexity of the problem, not necessarily the code - the code was pretty readable if you know what it does and how.