6 ms·

Putting aside questions of monorepos, I am always astounded by how much big companies invest in their build/change management/review systems. Shipping becomes a

• caust1c 4y ago
  Personally, Uber doesn't come off to me as an engineering culture that values simplicity. And in fact, in this very article, the last paragraph actually states that they are aiming to increase the complexity of the change validation process?!? > Additionally, it helped our team focus on increasing the complexity and features of our change validation process. Personally, I haven't been super impressed with any of the engineering projects I've heard come out of Uber, but what they're doing obviously works to some degree so I can't be one to judge.
  • speedgoose 4y ago
    Deck.gl is a fine map visualisation library.
• stingraycharles 4y ago
  I’d say that this is just internal platform / infrastructure support, aimed at making the whole organisation be more effective. It’s absolutely adding a lot of value to the organisation, it’s just not visible from the outside. But saying that “it’s too many steps removed from the value” appears to be a misrepresentation of the value it adds.
• MPSimmons 4y ago
  Per DORA and the information in Accelerate, shipping code frequently and reliably contributes directly to good outcomes. It seems like a good investment.
  • bdcravens 4y ago
    I don't think the commenter is challenging this idea. Many companies do this with "off the shelf" CI/CD solutions without committing large amounts of money into bespoke tooling.
    • zero_shift 4y ago
      From my experience, the off the shelf tooling works poorly with monorepos. Everything is still optimised for the polyrepo microservice craze of 2017. There are tools out there, but they're not free and they involve some level of vendor lock in. I've worked on some monorepo build scripting myself.
      • MPSimmons 4y ago
        In my opinion, git in general works poorly as a monorepo. My ideal VC solution would have the branch control of git with the checkout model and hook configuration of SVN. I am not familiar enough with the version control ecosystem to know if there’s another product that does it, but I doubt it since I kind of suspect it would violate the CAP theorem.
• anonymous_fun 4y ago
  I remember reading once that the larger your engineering org, the larger the percentage of your engineers should work on tooling. As the complexity of your system grows, the ability to know if your change causes an ill effect becomes more difficult. Large systems have large and complex tooling because it becomes a necessity for survival.
  • jmholla 4y ago
    Do you remember where you read that? I'd love to read that article/book.
• baskethead 4y ago
  Investing in that is vital if you want to grow as an engineering company. I work at a company with 2000 engineers, and the infrastructure and our build/change management/review system is horrible. It takes me all day to check something in, the integration tests are flaky and waste a lot of time tracking down errors, etc. Investing in this would collectively save so much time, but the people in charge think we can't afford the investment so everyone suffers.
  • bdcravens 4y ago
    Sounds more like an organizational issue than a technical one.
• mattarm 4y ago
  It is really just a matter of scale. At big companies it is easier to see the value of having people on these problems, so the teams have a clear reason to exist. As for your "side quest off a side quest" observation, I think that is a matter of perspective, about what personally motivates you, and about how work is organized in society. I find that working at a small company makes it easier to have a feeling of focus and purpose -- everybody is focused on one thing. But, I find that working at a large (profitable, well run) company makes it easier to feel like you can actually move the needle and have the resources do large things (and yes, this includes a connection between something like a 'build tools' team and being able to ship a quality product more quickly). It is easy to miss that every race car needs a pit crew, every pit crew needs equipment and parts, getting parts requires suppliers, parts need to be built, tested, etc., all those people need to eat, all that food needs to be grown, and so on. Organizing all of that "work" into small single-purpose companies, that buy each others' stuff, is one way to go. Another way to go is to organize it into larger companies with sub-teams/groups/departments. So a company's larger mission might be "win races" but, if it is large enough, smaller support efforts in support of the larger goal can make a lot of sense.
• bzxcvbn 4y ago
  Do you also find the existence of accounting or janitorial departments in companies whose core business is neither accounting nor custodial services strange?
  • tedunangst 4y ago
    I would find it pretty strange if the janitorial department were smelting iron ore to make their own trash bins.
  • bdcravens 4y ago
    And yet accountants mostly adhere to GAAP no matter the industry, and the janitors are using the same tools their counterparts are.
• tikhonj 4y ago
  I've repeatedly seen both high quality and low quality tooling act as massive force multipliers even for small teams; the question is just which direction you want to multiply. They say an army marches on its stomach—well, a tech effort marches on its tools. I interned at a trading company with, at the time, maybe 100 developers that built almost everything in-house and yet was quite a bit more successful and productive than much larger teams I've worked on since. The larger teams had incentives and structure to "focus" on what's "important" for very narrow notions of important... and so they ended up with lower-quality internal systems, a bunch of day-to-day drag on development and, both technically and organizationally, far less adaptability. (Turns out, one of the major advantages of investing in your own tools is that it builds up deep institutional knowledge about tooling that is hard-to-impossible to maintain otherwise.) Only valuing easily-measurable work is, frankly, a modern organizational disease. It's like searching for the keys under the streetlight—and why? As a substitute for human judgement or resolving disagreements? As a way to make work more legible for executives? It leads to unforced errors and systemic problems, but we can't seem to do anything about it.
  • infogulch 4y ago
    > Only valuing easily-measurable work is ... a modern organizational disease
  • jiggawatts 4y ago
    Recently I showed a development team how to use a debug tool that can automatically collect debug snapshots from production that can be opened with one click in the IDE. It’ll jump to the line of code and show the state of everything — not just the stack but the heap too. My demo is to pick a random crash and diagnose the root cause while talking. The mean time to resolution is low single digit minutes. I showed this to an entire team, one person at a time, solving bugs as I went. I showed the junior devs, senior devs, and their manager. No interest. None. Just… silence. The tools are amazing, but the lack of motivation from the typical developers for learning to use them is even more amazing.
    • djbusby 4y ago
• est31 4y ago
  Tech companies pay for few things more than for developer time. In a small company, spending one day to speed up a 30 minute build of a codebase used by 10 developers by 1 minute is not going to unlock much developer time, but imagine you multiply it by a thousand or ten thousand developers. Suddenly you save 10 thousand minutes per build, or 166 hours. Imagine the average dev runs a build three times per day, your 1 minute saving is now saving 498 hours of developer time per day, or 62 8 hour work work days per day (and there is the saying that in FANG it's way less than 8 hour work days). These are continuous cost savings, so the value your 1 day investment created is equivalent to 62 fully paid developers. Isn't that amazing? I think tech companies should spend a lot more on speeding up their stack. I would love to do something like this (improving developer workflows in a giant organization) as my day job.
  • dan-robertson 4y ago
    The counter argument here is that that 1 minute difference doesn’t correspond to a minute of extra work in the way that ‘62 days of work saved’ conveys. But the counter-counter argument is that sufficiently fast/good tooling is qualitatively different and increases productivity by enabling new workflows (eg build it vs think carefully about whether it will compile right because builds are slow; break up changes into smaller chunks vs write big hard-to-review changes because CI is slow) such that speeding up tools or maintaining performance as the code base grows are valuable.
• jayd16 4y ago
  YAGNI is always a valid strategy but what you are and are not going to need will change with scale. An inconsequential case can turn out to be something that impacts several teams. At scale, rabbit holes can be quite relevant. At scale, you can no longer trust that your personal view of the org is complete enough. There's still plenty of waste but just because you don't need something doesn't mean it isn't in another team's hot loop. Best to come in with trust first and incredulity second.
• david38 4y ago
  People drive cars to work, school, etc. Look at the infrastructure required for that and what we spend on it. It’s massive.
  • bdcravens 4y ago
    Best practices in software tend to not be confined to a specific domain, so the amount of money spent in the industry being targeted really isn't relevant to need for custom CI vis-a-vis off-the-shelf solutions.
• rr808 4y ago
  Wait til the regulators come for your industry. Modern IT has to prove to Auditors amongst other things that no one rogue person can contribute malicious code. I've worked in companies that let people do their own thing but then let them deal with Audit. A much better company had a sophisticated custom SDLC process that meant Audit just checked you used the process.
• dilyevsky 4y ago
  > It’s too many steps removed from the value. And this kind of reasoning is why tooling in most shops absolutely sucks
• madeofpalk 4y ago
  > It’s too many steps removed from the value Uber is, partly, a software business. They need to ship software. They get direct value from shipping good software efficiently.
• novok 4y ago
  The amount of people dedicated to such projects is often a very small overall percentage of the engineering force, and they act as a leverage point. Invest in X staff, make all eng at the company 1.X more effective, therefore giving you a large return on investment. > It’s too many steps removed from the value. Don't be worried about being too many steps removed from the value. With that logic, CEOs, CTOs, etc are also too many steps removed from the value, yet they are valued very highly. Look at such infra work as multiplying the value of the entire company and in a way, being part of a "CEO team".