5 ms·

Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several lab

by dblock 15y ago

Others have tried and keep throwing more and more smart people at the problem they just shouldn't have. MSFT with Windows codebase that runs out of several labs. Crazy branching and merging infrastructure. They use source-depot, originally a clone of perforce. Google with all their source code in one Perforce repo. Facebook will be on perforce before we know it. The solution is an internal Github, not one giant project.

sek 15y ago
Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.
- deleted 15y ago
  [deleted]
- mikeocool 15y ago
  No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google http://www.infoq.com/presentations/Development-at-Google
  - sek 15y ago
    Very interesting, thank you.
  - rachelbythebay 15y ago
    The kernel? Android? Some other spooky stuff involving the pest control guy who's holding a big rubber mallet when you fail a unit test? Are you sure about that?
    - nostrademons 15y ago
      Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.
      - rachelbythebay 15y ago
        Right, so "literally the entire codebase for all their products" is incorrect. Thanks.
        jrockway 15y ago
        The open-source stuff is a rounding error. Think about all the Google products; Search, Google+, Gmail, Groups, Translate, Maps, Docs, Calendar, Checkout, Wallet, Voice, ... those are all in one repository. (Not to mention all the libraries and internal tools; those are all in there too.)
        nostrademons 15y ago
        To be fair, Android and Chrome are pretty huge projects. I know the numbers (though I don't think I can share them outside of Google), and while they're nowhere close to being a big part of the total, they're also big enough to not be considered a rounding error.
        amalter 15y ago
        Ah,rachelbythebay, you caught him for being "technically incorrect". Which, depending on how you look at it, is either the best or worst kind of correctness.
- deleted 15y ago
  [deleted]
- deleted 15y ago
  [deleted]
EricBurnett 15y ago
Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions: * Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. * Build from head. We build everything from source, statically linked. No need to worry about multiple versions of dependencies, no lag between a bug fix and it being available to any and all binaries that need it, whenever they're next updated. I don't think that an "internal Github" is going be a magic bullet here. It's more likely it would be a matter of trading one set of hard problems for another, as we all of a sudden need to figure out how to do cross-dependencies sanely, deal with multiple versions of libraries, etc, at scale. You are correct that one monolithic Perforce repo is a bit of a pain point, but that doesn't necessarily mean that the right decision is to shatter our codebase into different pieces - we'd rather make our repo scale better. For reference, we've already got hundreds of millions of lines of code, 20+ changes/minute 6 months ago (so what, 30+ now?), and plans for scaling the next 10x are in motion. If you're interested, I recommend http://google-engtools.blogspot.com/ http://google-engtools.blogspot.com/. It details a number of the problems we've run into, and our solutions for dealing with them at scale.
- mindcrime 15y ago
  Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstation? And sure a compiled language would catch naming collisions on functions or whatever, but nothing stops somebody from creating a method doQuickSort( ... ) and somebody else creating quickSortFoo(...) where they are semantically equivalent (or very nearly so). It seems to me that the problem of duplicating code, because you don't know that a method already exists to do what you're trying to do, is the same problem regardless of how your tree is laid out; and is ultimately more of a documentation / process / discipline issue. But I'd be curious to hear the counter-argument to that...

by dblock 15y ago

sek 15y ago

Google has everything in one Perforce repo? You mean the search engine, do you? I agree btw, the Github mindset is the best one. Create for every project a new repo and connect them with build tools. But why not hire 100 SOA-Consultants, they have enough money now.

deleted 15y ago
[deleted]
mikeocool 15y ago
No, literally the entire codebase for all of their products is in one Perforce repo. Ashish Kumar, manager of the Engineering Tools team, mentions it in this presentation: http://www.infoq.com/presentations/Development-at-Google http://www.infoq.com/presentations/Development-at-Google
- sek 15y ago
  Very interesting, thank you.
- rachelbythebay 15y ago
  The kernel? Android? Some other spooky stuff involving the pest control guy who's holding a big rubber mallet when you fail a unit test? Are you sure about that?
  - nostrademons 15y ago
    Kernel/Android/Chrome/basically anything open-source is different. If the code is going to be open-sourced, it can't have dependencies on proprietary code anyway.
    - rachelbythebay 15y ago
      Right, so "literally the entire codebase for all their products" is incorrect. Thanks.
      - jrockway 15y ago
        The open-source stuff is a rounding error. Think about all the Google products; Search, Google+, Gmail, Groups, Translate, Maps, Docs, Calendar, Checkout, Wallet, Voice, ... those are all in one repository. (Not to mention all the libraries and internal tools; those are all in there too.)
        nostrademons 15y ago
        To be fair, Android and Chrome are pretty huge projects. I know the numbers (though I don't think I can share them outside of Google), and while they're nowhere close to being a big part of the total, they're also big enough to not be considered a rounding error.
      - amalter 15y ago
        Ah,rachelbythebay, you caught him for being "technically incorrect". Which, depending on how you look at it, is either the best or worst kind of correctness.
deleted 15y ago
[deleted]
deleted 15y ago
[deleted]

EricBurnett 15y ago

Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions: * Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. * Build from head. We build everything from source, statically linked. No need to worry about multiple versions of dependencies, no lag between a bug fix and it being available to any and all binaries that need it, whenever they're next updated. I don't think that an "internal Github" is going be a magic bullet here. It's more likely it would be a matter of trading one set of hard problems for another, as we all of a sudden need to figure out how to do cross-dependencies sanely, deal with multiple versions of libraries, etc, at scale. You are correct that one monolithic Perforce repo is a bit of a pain point, but that doesn't necessarily mean that the right decision is to shatter our codebase into different pieces - we'd rather make our repo scale better. For reference, we've already got hundreds of millions of lines of code, 20+ changes/minute 6 months ago (so what, 30+ now?), and plans for scaling the next 10x are in motion. If you're interested, I recommend http://google-engtools.blogspot.com/ http://google-engtools.blogspot.com/. It details a number of the problems we've run into, and our solutions for dealing with them at scale.

mindcrime 15y ago
Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. I'm not convinced that the difference between a singly rooted tree and a multiple-rooted tree is going to make that much difference. I mean, think about it... if you 100k's or even millions of files, is anybody going to parse through all of that, looking for a reusable function, even if it is on their workstation? And sure a compiled language would catch naming collisions on functions or whatever, but nothing stops somebody from creating a method doQuickSort( ... ) and somebody else creating quickSortFoo(...) where they are semantically equivalent (or very nearly so). It seems to me that the problem of duplicating code, because you don't know that a method already exists to do what you're trying to do, is the same problem regardless of how your tree is laid out; and is ultimately more of a documentation / process / discipline issue. But I'd be curious to hear the counter-argument to that...