4 ms·
Dear Google: I don't want to sound negative (that's not the purpose of this message), but my belief is that this incident is a heart palpitation on the way to
by peter_d_sherman 8y ago
Dear Google:
I don't want to sound negative (that's not the purpose of this message), but my belief is that this incident is a heart palpitation on the way to a major coronary, stroke, or heart attack... Let me explain to you why.
You see, as your company has grown, so has the complexity of your codebase and so has the number of people who maintain that code.
The complexity of your code has increased, perhaps asymptotically (since you seem to like Big-O notation and related subject matter, from what I hear, at your interviews), yet I'd be willing to bet that the overall knowledge of the codebase that the average worker there has, has actually diminished (although perhaps not asymptotically, perhaps linearly) over time.
In other words, you've got too many chefs, and a soup with too many ingredients where every chef knows some of them, but not all of them.
Maybe Jeff Dean is the exception to that, but that's what you've got going on, basically.
When something like this happens, your responsibility is to a) Educate your rank-and-file, even at the expense of new product releases, b) Re-delegate, insuring that there aren't too many chefs to soup, c) Reduce the ingredient count (refactor), and perhaps most important d) Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom (difficult when you've got multiple source-code contributors making changes, and multiple layers of people only responsible for some aspects of the system...).
(Oh, and if it were me, I'd go as far as to buffer the entire incoming datastream so as to be able to replicate the problem, and if you can't do that (run a buffered datastream from a point in time against a point-in-time snapshot of all data at that point in time, and see exactly where in code the problem occurs), then your first goal is to be able to do exactly that. But I'm a little extreme in what I feel qualifies as acceptable testing...)
- jhgg 8y ago>Make sure there's someone, anyone, who understands ALL of the codebase from top to bottom Either you seriously over estimate the limits of ones understanding, or you severely under estimate the size of Google’s codebase/monorepo. At this point, I’m pretty sure it’s impossible to even check it out on a single developers workstation due to its size.
- peter_d_sherman 8y agohttps://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a... ...Also, compare that to the Bible's "Tower Of Babel" story: https://en.wikipedia.org/wiki/Tower_of_Babel https://en.wikipedia.org/wiki/Tower_of_Babel The Bible's story is about human language, specifically about how human languages encompass greater degrees of abstraction over time (languages built on top of languages, abstractions built on top of abstractions). But, when you get to a certain level of abstractions, the abstractions start to "leak". That is why there are communication difficulties in societies (people who are more educated typically speak a different language than people who are less educated; academia and lawyers speak their own language). This is also when/where/why/how software starts to fail. Too many levels of abstraction, and those abstractions will "leak". Don't take my word for it, use your own logic, your own common sense and think about it... Software problem A exists. Management Team: "We have a problem, can you fix it?" Programming Team: "Sure." (fixes it, but this causes problem B, which is not recognized until several weeks later) Management Team: "Now we have this different problem, can you fix it?" Programming Team: "Sure." (fixes it, but this cause problem A to resurface, which is not detected for several weeks...) Management Team: "We have a problem, can you fix it?" And the cycle repeats... Was it the programming team's fault? No, it was the nature of the beast called "complexity". The higher you go in abstractions (and this is especially true for AI), the more you must exercise ENGINEERING DISCIPLINE. You must be able to go BACKWARDS IN TIME (this is accomplished in programming by being able to go back to simpler programs and fully understand / test them before advancing to more complex ones, and PRESERVING THAT CHAIN OF UNDERSTANDING...). This is probably why advanced societies destroy themselves, too much abstraction, too much "magic" (without understanding of all of the sublevels), and no one has the exact knowledge necessary to pinpoint exactly when/where/why the exact problem occurred... But again, don't take my word for it. Use your own common sense to determine if what I'm saying has any merit...