3 ms·

One gigantic Perforce repo that achieved such Brobdingagian scale, Google had to roll their own ultra-giga Perforce that properly belongs in a hellish mondo alt

One gigantic Perforce repo that achieved such Brobdingagian scale, Google had to roll their own ultra-giga Perforce that properly belongs in a hellish mondo alternate dimension of largeness. “But it’s searchable”, Google says. Yes. By the devil, it is.

gravypod 7y ago
Is writing/improving tools to fit your needs considered bad? If Google needed to store an exabyte of data into MySQL and they rewrote something with a MySQL compatible APIb (PingCap) would this have been bad?
- dahfizz 7y ago
  I think the sentiment is that would be solving the wrong problem. Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that.
  - joshuamorton 7y ago
    > Why do they need to store so much data in a single DB, and could they use a different approach? Well, we have examples of the opposite (Amazon uses a multi-repo approach) and my understanding is that it works fine, until it doesn't, when its awful. This often happens when there are security vulnerabilities, and Amazon is forced to, under time pressure, walk their entire dependency graph and pin all dependencies to newer versions, fixing forward as they go. At the point where you have a lot of code, you only really have two options.
  - gravypod 7y ago
    > Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that. I see the choices as: a) rewrite one core tool infrastructure component or b) rewrite everything - currently and soon to be - built on that infrastructure. It's also a question of level of abstraction. Why does my application care about database sharing, replication, leaders, followers, etc. It should only care about connecting & querying. The complications of running a system at scale should be hidden as well as possible from the code you've written. From a monorepo perspective: why do I care where my code is stored? As long as I can control visibility, hide build complexity, and expose an API to other pieces of code I have all the functionality I need. Why should the nitty gritty of a package manager, VCSs, or other multi-repo concepts influence how I write my code?
singlow 7y ago
(Brobdingnagian) Edit: or maybe Brobdingragian, as Swift said the publishers actually misspelled it.

gravypod 7y ago

Is writing/improving tools to fit your needs considered bad? If Google needed to store an exabyte of data into MySQL and they rewrote something with a MySQL compatible APIb (PingCap) would this have been bad?

dahfizz 7y ago
I think the sentiment is that would be solving the wrong problem. Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that.
- joshuamorton 7y ago
  > Why do they need to store so much data in a single DB, and could they use a different approach? Well, we have examples of the opposite (Amazon uses a multi-repo approach) and my understanding is that it works fine, until it doesn't, when its awful. This often happens when there are security vulnerabilities, and Amazon is forced to, under time pressure, walk their entire dependency graph and pin all dependencies to newer versions, fixing forward as they go. At the point where you have a lot of code, you only really have two options.
- gravypod 7y ago
  > Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that. I see the choices as: a) rewrite one core tool infrastructure component or b) rewrite everything - currently and soon to be - built on that infrastructure. It's also a question of level of abstraction. Why does my application care about database sharing, replication, leaders, followers, etc. It should only care about connecting & querying. The complications of running a system at scale should be hidden as well as possible from the code you've written. From a monorepo perspective: why do I care where my code is stored? As long as I can control visibility, hide build complexity, and expose an API to other pieces of code I have all the functionality I need. Why should the nitty gritty of a package manager, VCSs, or other multi-repo concepts influence how I write my code?

singlow 7y ago

(Brobdingnagian) Edit: or maybe Brobdingragian, as Swift said the publishers actually misspelled it.