4 ms·

The tooling is great in some ways, but possibly not in the way you'd expect. There is a tool that will do almost anything, but the tools are often just a bit j

The tooling is great in some ways, but possibly not in the way you'd expect. There is a tool that will do almost anything, but the tools are often just a bit janky. Rather than thinking of the toolchain as some polished, tightly integrated, perfect system, think of it more like an internal open-source ecosystem of things that work together but which are made by individuals, with limited resources, that still feature bugs. This is not to downplay what we have, it's amazing in many ways, but most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them that cover almost every aspect of development. So much of the time I run into neat little systems and then discover they're a few Python scripts running under someone's personal quota. Sometimes these are 20% time projects, sometimes they are bits of tooling teams build for their own needs, sometimes they are more concerted efforts. There is a core, tightly integrated set of systems that work well and are more "productized", but most of those aren't that much more special than what you can buy at any other company. If you've used Datadog then you'd feel at home in much of the modern infra tooling. Spanner is great, but anyone can pay to use it. Our CDN is very good, but there are other CDNs and from a developer perspective there isn't a whole lot of difference.

88913527 3y ago
I have yet to be impressed by internal software tools developed by one team and consumed by another. You're likely being friendly and understating the degree of jank which is present.
- danpalmer 3y ago
  Is this at Google, or elsewhere? If it's at Google I'd be happy to try to persuade you via chat :) If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).
  - water-your-self 3y ago
    >if its google id be happy to try to persuade you via chat. I would be curious as to how you view projects that are staffed entirely 20%.
    - danpalmer 3y ago
      Feel free to reach out internally! But in general, it seems to work ok for many popular tools. I haven't attempted to do it, maybe it sucks as the maintainer?
- scarmig 3y ago
  It's janky, and when you do choose to use a best-effort tool with no dedicated support, you absolutely have to plan for it being a pain point and the likelihood of eventual migration. That's true at Google, just as it's true anywhere. But what makes Google a much better experience than most places is a willingness (and ability) to pay for internal dev tooling of every sort with dedicated headcount. And that results in a consistently better experience than pretty much anywhere (I understand Meta has a comparable culture and ability to fund it, and I wouldn't be surprised if some or many of their internal tools surpass Google's). It's also unfortunately not replicable as a strategy, because you need a giant money spigot to make it work.
water-your-self 3y ago
>most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them Why? Why reinvent the wheel 10 times?
- danpalmer 3y ago
  Most engineers create tools, Google is not special in that way. But even when there's a similar open-source tool, it may be difficult to integrate with internal systems. There's a lot of benefit to a consistent set of technologies (data formats, storage systems, etc) so that's often a reason to write internally. There is a lot of open source software in use at Google so I wouldn't say there's a lot of reinventing the wheel. And when there is an internal clone, the requirements are often necessarily different enough to warrant writing an internal version.
- asoneth 3y ago
  I did not interpret their statement as meaning they have many variants of the same tool, but that they have tools that cover a wider variety of situations. I found the same thing moving from a very large company to a smallish company. Both had good tooling, the latter just had significantly less and there were gaps in coverage. Whenever I encountered one of those areas I often found myself wishing I had access to some niche tool I had gotten used to at my prior company.
- UncleMeat 3y ago
  Google has 180,000 employees. There's an org that is responsible for internal tooling, but they are an org with priorities and challenges like any other. So sometimes Search needs a thing and Core won't do it so some people in Search build a tool. Maybe later somebody in Ads needs a similar tool but has never heard of the thing that Search built. So there is an organic force to this sort of complexity. There are systems to resist this and people whose whole job is to focus on unifying systems, but they aren't perfect. In general, I'd say that Google is way way way more uniform than most other companies because of the monorepo, centralized tooling, and (mostly) uniform development process across the company.
  - 3y ago

88913527 3y ago

I have yet to be impressed by internal software tools developed by one team and consumed by another. You're likely being friendly and understating the degree of jank which is present.

danpalmer 3y ago
Is this at Google, or elsewhere? If it's at Google I'd be happy to try to persuade you via chat :) If elsewhere, I think there are 2 kinds of tool: those developed by individuals/small groups in informal ways, some of these are bad, but much like with open-source tools, the ones that are good survive and become popular. The other type is those with more formal backing, dedicated dev/design/UX/PM resources, and those are good at surveying users to figure out what the right problem to solve is, so they're normally pretty good, and when they're not, they still accept fixes (I regularly contribute fixes to other teams codebases when I find issues).
- water-your-self 3y ago
  >if its google id be happy to try to persuade you via chat. I would be curious as to how you view projects that are staffed entirely 20%.
  - danpalmer 3y ago
    Feel free to reach out internally! But in general, it seems to work ok for many popular tools. I haven't attempted to do it, maybe it sucks as the maintainer?
scarmig 3y ago
It's janky, and when you do choose to use a best-effort tool with no dedicated support, you absolutely have to plan for it being a pain point and the likelihood of eventual migration. That's true at Google, just as it's true anywhere. But what makes Google a much better experience than most places is a willingness (and ability) to pay for internal dev tooling of every sort with dedicated headcount. And that results in a consistently better experience than pretty much anywhere (I understand Meta has a comparable culture and ability to fund it, and I wouldn't be surprised if some or many of their internal tools surpass Google's). It's also unfortunately not replicable as a strategy, because you need a giant money spigot to make it work.

water-your-self 3y ago

>most tools are exactly the same sort of thing that you'd create in any other place, we just have more of them Why? Why reinvent the wheel 10 times?

danpalmer 3y ago
Most engineers create tools, Google is not special in that way. But even when there's a similar open-source tool, it may be difficult to integrate with internal systems. There's a lot of benefit to a consistent set of technologies (data formats, storage systems, etc) so that's often a reason to write internally. There is a lot of open source software in use at Google so I wouldn't say there's a lot of reinventing the wheel. And when there is an internal clone, the requirements are often necessarily different enough to warrant writing an internal version.
asoneth 3y ago
I did not interpret their statement as meaning they have many variants of the same tool, but that they have tools that cover a wider variety of situations. I found the same thing moving from a very large company to a smallish company. Both had good tooling, the latter just had significantly less and there were gaps in coverage. Whenever I encountered one of those areas I often found myself wishing I had access to some niche tool I had gotten used to at my prior company.
UncleMeat 3y ago
Google has 180,000 employees. There's an org that is responsible for internal tooling, but they are an org with priorities and challenges like any other. So sometimes Search needs a thing and Core won't do it so some people in Search build a tool. Maybe later somebody in Ads needs a similar tool but has never heard of the thing that Search built. So there is an organic force to this sort of complexity. There are systems to resist this and people whose whole job is to focus on unifying systems, but they aren't perfect. In general, I'd say that Google is way way way more uniform than most other companies because of the monorepo, centralized tooling, and (mostly) uniform development process across the company.
- 3y ago