4 ms·

I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.

• repeekad 1y ago
  The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%… In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
  • pclmulqdq 1y ago
    There's a culture of "I won't approve this unless it does something for me" at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.
    • moregrist 1y ago
      While this sounds pretty frustrating, there is at least a small upside: at least you get to the obvious-but-ignored bugfixes. Most smaller places don’t have the bandwidth and many larger ones don’t have the desire. I’m not sure if that makes up for bugs potentially introduced in the refactors, though.
      • dietr1ch 1y ago
        Well, when the owner asks for a whole test suite that didn't exist to get a fix in, what most likely happens is that you just wasted your time in a draft CL that will get lost.
        • vkou 1y ago
          They aren't asking for you to write tests because 'it benefits them', they are asking you to write tests because as a professional engineer, you should write tests, and not just yolo it. Look, sometimes you may have good reasons for why a test is impractical. You are allowed to push back, or look for a different reviewer. There's a hundred thousand people in the firm, you should be able to find one or two that will let you submit literally anything that compiles. But most of the time, the reviewer is giving you advice that you should take.
          • pclmulqdq 1y ago
            If you are turning a button to a slightly different shade of blue and it's not a button you own, the owner of the button should not be asking you to write tests for the button.
            • vkou 1y ago
              If the case is as simple as you describe, surely there's more than one owner of the code that can approve this, if one guy is being unreasonable. If there is actually only one owner that can approve changes to the package, there's something really weird and unusual about that project setup, or it's someone's internal hobby project that they wrote five years ago and semi-maintain, in which case, I have to wonder why you submitting one-liner changes to it is all that important. We're all adults, we all work together, we can all work this out. If someone absolutely insists on being an asshole, escalate. It's why you have a manager, and why they have a manager. My experience is that very few people are unreasonable assholes. There's always plenty of organizational, vision, strategy, and execution problem in any billion-dollar company, but 'people are unreasonable in code reviews' is not one I'd put in the top 10. It might be something that ruins your day once or twice, but that doesn't make it systemic.
              • dietr1ch 1y ago
                > If someone absolutely insists on being an asshole, escalate. That's doubling down on time spent on contributing back. It's usually cheaper to workaround the issue once you notice it'll be way harder than it should be (not hard at all).
  • apercu 1y ago
    > lazy domain owners Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs. Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.
  • sollewitt 1y ago
    Coordination Headwind: https://komoroske.com/slime-mold/ https://komoroske.com/slime-mold/
    • repeekad 1y ago
      Thanks, this may be the name they used it’s been a little while
  • steveBK123 1y ago
    The assumptions in that math are wrong anyway. Once you depend on 10 people, the chance that they each achieve "95% successful execution" is 0. This is only partially down to the impossibility of having every staff member on a project be A++ players. There is coordination RISK not just coordination overhead. Think planning a 2 week trip with your spouse with multiple planes/trains/hotels, museum/exhibit ticket bookings, meal reservations, etc. Inevitably something gets misunderstood/miscommunicated between the two of you and therefore mis-implemented. Now add more communication nodes to the graph and watch the error rate explode.
    • nostrademons 1y ago
      That's what the math is reflecting. Project succeeds if all of 10 people does their job well. Each person has a 95% chance of succeeding. 0.95^10 ~= 60%, and so the chance that all 10 people do their job successfully is ~60%. Those jobs also include things like management and product design, and so the coordination risk is reflected in the 5% chance that the manager drops the ball on communication. (As a manager, I suspect that chance is significantly more than 5% and that's why overall success rates are even lower.)
      • steveBK123 1y ago
        That's what I mean "only 5%" encapsulating all failure modes (comms/implementation/coordination/etc) is very low. And that under-estimation compounds to make the top level 60% much higher than it should be. A 7.5% rate takes top-level success odds below 50% - 46%. A not unrealistic 10% takes the top level down to 35%. Etc.
        • econ 1y ago
          For extra fun we are doing more than one project in a row.
  • xnx 1y ago
    The old "If you want to go fast, go alone. If you want to go far, go together."
    • nostrademons 1y ago
      Also why the optimal business strategy seems to be to go as far as you can alone and then bring on other people when you're running out of steam.
  • miki123211 1y ago
    Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in, which is deeply counterintuitive to most people. Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in a major lawsuit, regulatory fine or PR disaster that costs the company more than the project was ever expected to make. If you're a startup, the worst that can happen to your company is the value going to 0. From an investor's perspective, there's not much of a difference between burning all the money ($10m) and not finding product-market-fit (normal failure), or your company getting sued for $3b and going bankrupt (critical failure). The result is the same, the investment is lost. For a large corporation, a $3b lawsuit is far more costly than sinking $10m into a failed project. You can trade off these three possibilities against each other. Maybe forcing each release through an arduous checklist of legal review or internationalization and accessibility testing decreases success rates by 10%, but moves the "critical failure rate" from 1% to 0.5%. From a startup's perspective, this is a bad tradeoff, but if you're a barely-profitable R&D project at big co, the checklist is the right call to make. This problem is independent from all the other causes to which bureaucracy is usually attributed, like the number of layers of management, internal culture, or "organizational scar tissue." Just from a legal and brand safety perspective, the bigger your org, the more bureaucracy makes sense, no matter how efficient you can get your org to be.
    • kridsdale3 1y ago
      This is an excellent explanation of the culture of BigTech.
    • YZF 1y ago
      I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection. Just like anything else in life you want to look at the present value and then get insurance for huge risks. That said agree that a startup can take more risk but I don't think that is the major factor explaining why larger companies tend to be process heavy and slower.
  • zenogantner 1y ago
    Well, good management/tech leadership is about making sure that the risks coming from individual failure points (10 people in your example) are recognized and mitigated, and that the individuals involved can flag risks and conflicts early enough so that the overall project success probability does not go down as you describe...
• simianwords 1y ago
  Equally important is the amount of time they save because of available abstractions to use like infra, tooling etc
  • materielle 1y ago
    I think I understand what you are saying, and I agree. Google has all sorts of great infra projects that simplify complex domains. Those are solved problems now, so nobody notices it. The existence of incidental complexity isn’t evidence that the counter factual is less complexity.
• Demiurge 1y ago
  And when you’re at a smaller company 90% of your time is fighting societal complexity, limit of which also approaches infinity, but at a steeper angle. No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have no idea if your lesson is even useful. Let’s judge the story on its merits and learn what we can from it.
  • apercu 1y ago
    Look, I've never had to design, build or maintain systems at the scale of a FAANG, but that doesn't mean I haven't been involved in pretty complicated systems (e.g., 5000 different pricing and subsidy rules for 5000 different corporate clients with individually negotiated hardware subsidies (changing all the time) and service plans, commission structure, plus logistics, which involves not only shipping but shipping to specific departments for configuration before the device goes to the employee, etc. Arbitrarily, 95% of the time the issues were people problems, not technical ones.
    • Demiurge 1y ago
      I have a similar perspective. I think after a few years, it's the people things that have always been the hardest part of the job. That's probably why in the interviews, we always say things like: communication is key, culture fit, etc. On the other hand, the good part of the job is solving complex technical problem with a team.
    • bitexploder 1y ago
      You are right but it misses the flavor of the problem. I was a consultant in infosec to F500s for many years. Often solving a problem involves simply knowing the right person that has already thought about it or toiled on the problem or a similar one. But when there are 100,000 engineers it becomes an order of magnitude (or two!) more difficult and that puts forth unique challenges. You can still call them “people problems” and they often may be. However if you try to solve them the same way you might solve it at a smaller engineering org you will get and be nowhere and be poorer for the time you spent trying it. Ask me how I know lol. The technical problems are also like that. Almost everything has an analog or similar thing to what you are probably familiar with but it is scaled out, has a lot of unfamiliar edges and is often just different enough that you have to adjust your reasoning model. Things you can just do at even a typical f500 you can’t just do at big tech scale. Anyway, you are directionally correct and many of these wounds are self inflicted. But running a company like Google or Facebook is ridiculously hard and there are no easy answers, we just do our best.