5 ms·

It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion. How repos are organized is a choice. Google coul

It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion. How repos are organized is a choice. Google could just as easily (financially speaking) have chosen to break up that repo and use more standard tooling. They did not choose that. Did they have (possibly perfectly valid) reasons for that? Sure. That does not render it as their only possible path. Rather than switch their source control strategy to something that would work well with existing products, they chose to build a brand new product to support their existing strategy. This is, IMO, not an obviously sane decision from an outside perspective. Why? Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant. That's how it looks from an outside perspective. The question I'm asking is not "why didn't they use git?!". The question I'm asking is, "what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?" Haven't seen an answer to that yet. People are too busy implying or outright stating that I'm stupid for asking the question in the first place.

bananapub 2y ago
> It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion. not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.
- floating-io 2y ago
  I see a lot of saying that, but no evidence. A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration? The answer to that is not obvious. (edit: response specifically regarding "astronomical" costs)
  - valicord 2y ago
    What makes you think that a multirepo system would be cheaper to maintain than a monorepo? It's just moving complexity from one place (source control software) to another (everything that interacts with the source control software). TANSTAAFL
    - floating-io 2y ago
      I never argued that I thought multirepo would be cheaper than monorepo. Or vice versa, for that matter. I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument. TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate. I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty. Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture. Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice. I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center. Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time? Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence. If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards? Yeah, I'm going to call bullshit.
valicord 2y ago
> what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products? Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.
disgruntledphd2 2y ago
So I was at FB when they attempted to add support to Git for really large monorepos. They offered cash and people to work on it, but the git team was completely uninterested and basically said that you're holding it wrong. Perhaps Google tried to do it this way and hit the same issues.
- gremlinExtra 2y ago
  Git also just doesn't play nice with monorepos of the scale of Google's/Facebooks. From a different article explaining why FB moved away from Git: "The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1] [1] https://graphite.dev/blog/why-facebook-doesnt-use-git https://graphite.dev/blog/why-facebook-doesnt-use-git
  - disgruntledphd2 2y ago
    Yeah, I remember this. The team involved did try really hard to get the Git team to listen to them, before working with Mercurial (and later building their own thing, although that was after I left).
kccqzy 2y ago
I don't think you understand how much early Google values engineer productivity. > Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant. God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool. Git sucks so much compared to Piper that it boggles the mind to even ask such a question.
- floating-io 2y ago
  I haven't used piper, so I can't comment on that. But people seem to have focused on git. Remind me not to actually use product names as examples; it inevitably triggers people into holy wars and prevents them from actually seeing the point. I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know. The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency. I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue. It's always a trade; those dollars can potentially do more good somewhere else. You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH. It's really hard to deny that Google has acute NIH. Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions. I don't think there's anything wrong with that.

bananapub 2y ago

> It's interesting how people treat the monorepo as some kind of religion that MUST be kept in this discussion. not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.

floating-io 2y ago
I see a lot of saying that, but no evidence. A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration? The answer to that is not obvious. (edit: response specifically regarding "astronomical" costs)
- valicord 2y ago
  What makes you think that a multirepo system would be cheaper to maintain than a monorepo? It's just moving complexity from one place (source control software) to another (everything that interacts with the source control software). TANSTAAFL
  - floating-io 2y ago
    I never argued that I thought multirepo would be cheaper than monorepo. Or vice versa, for that matter. I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument. TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate. I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty. Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture. Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice. I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center. Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time? Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence. If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards? Yeah, I'm going to call bullshit.

valicord 2y ago

> what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products? Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.

disgruntledphd2 2y ago

So I was at FB when they attempted to add support to Git for really large monorepos. They offered cash and people to work on it, but the git team was completely uninterested and basically said that you're holding it wrong. Perhaps Google tried to do it this way and hit the same issues.

gremlinExtra 2y ago
Git also just doesn't play nice with monorepos of the scale of Google's/Facebooks. From a different article explaining why FB moved away from Git: "The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1] [1] https://graphite.dev/blog/why-facebook-doesnt-use-git https://graphite.dev/blog/why-facebook-doesnt-use-git
- disgruntledphd2 2y ago
  Yeah, I remember this. The team involved did try really hard to get the Git team to listen to them, before working with Mercurial (and later building their own thing, although that was after I left).

kccqzy 2y ago

I don't think you understand how much early Google values engineer productivity. > Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant. God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool. Git sucks so much compared to Piper that it boggles the mind to even ask such a question.

floating-io 2y ago
I haven't used piper, so I can't comment on that. But people seem to have focused on git. Remind me not to actually use product names as examples; it inevitably triggers people into holy wars and prevents them from actually seeing the point. I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know. The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency. I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue. It's always a trade; those dollars can potentially do more good somewhere else. You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH. It's really hard to deny that Google has acute NIH. Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions. I don't think there's anything wrong with that.