8 ms·

Is the practice of shoving all disparate pieces of proprietary software (or individual projects) in the same repo a common occurrence? I have found that pullin

Is the practice of shoving all disparate pieces of proprietary software (or individual projects) in the same repo a common occurrence? I have found that pulling unrelated changes just so that I can push my changes is an inconvenience. Furthermore, tracking the history of a particular project is confounded due to unrelated commits. I am sure that their vcs (piper?) makes this a feasible task, but for git, it seems like it would suck. The article posted by kyrra, mentions this. Given the value gained from the existing tools Google has built and the many advantages of the monolithic codebase structure, it is clear that moving to more and smaller repositories would not make sense for Google's main repository. The alternative of moving to Git or any other DVCS that would require repository splitting is not compelling for Google. It seems like they have just too much invested in this "shove it in the same repo" style. Or is this the more appropriate way to do things in a large organization?

tehlike 10y ago
It actually works quite nicely. Most of the google software is built internally. Ensuring everyone is running at head is a blessing (rarely a curse), because when you make a change you immediately see if it breaks something. You can also be sure everything gets all the bug fixes in their new release. This is good for various reasons, including the fact that it makes security audits significantly simpler. Some google orgs, like android, don't use the existing infra, and they kind of struggle and have to reinvent most wheels, because of that. Edit: added second paragraph
throwawayGogEng 10y ago
Coming from companies that use reasonably-sized git repos, I absolutely hated Google's VCS. Here's some of my painpoints with it: * No branches. If you want to make a temporary code branch, you create a CL (Google's version of a pull request), but never submit it. This means nobody else can collaborate on it with you, and it must be manually updated to HEAD. * No CL collaboration. Unlike Git branches, CLs can only contain changes from one user. * No stable branch. Since everything is essentially on one long branch, it's a real hassle when a project is broken at HEAD. Sure, integration tests should ideally prevent this. In practice, HEAD is often broken. Teams have created bash scripts and mailing lists to determine 'stable' old versions that can be checked out for development. * Single versions of libraries. Any library that is used is also checked into the VCS. However, only one version of the library can exist in the codebase, which is rarely updated. However, there are exceptions to this. At one point, Sergey mentioned bringing Google "up to industry standards" regarding VCS's. However, that would be a monumental task and I doubt it will happen.
- jmillikin 10y ago
  Everything you posted is wrong, which makes me believe you know everything you posted is wrong. Nobody writes things that inaccurate by accident. For the benefit of YC, here are corrections: > No branches. If you want to make a temporary code branch, > you create a CL (Google's version of a pull request), but > never submit it. This means nobody else can collaborate on > it with you, and it must be manually updated to HEAD. Piper has branches, which can be committed to as normal by any number of engineers. It's common to have both branches for developing certain features ("dev branches"), and branches pinned to a stable base version with cherry-picked bug fixes ("release branches"). > No CL collaboration. Unlike Git branches, CLs can only > contain changes from one user. CLs are equivalent to Git commits. Collaboration is expected to occur via a series of CLs, just like Git-based projects have large changes made via a series of smaller commits. > No stable branch. Since everything is essentially on one > long branch, it's a real hassle when a project is broken > at HEAD. Sure, integration tests should ideally prevent > this. In practice, HEAD is often broken. Teams have > created bash scripts and mailing lists to determine > 'stable' old versions that can be checked out for > development. There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases. This is similar to the Linux kernel's dev model, where stable releases are cut at known-healthy points in an evolving codebase. Important libraries define more rigorous releases, similar to Git labels, which are updated automatically every day or two. These both reduce the amount of tests that need to run, and reduce chances of errors in low-level code affecting many teams. > Single versions of libraries. Any library that is used is > also checked into the VCS. However, only one version of > the library can exist in the codebase, which is rarely > updated. However, there are exceptions to this. Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature. Three of the four languages most often used at Google (Python, C++, Go) do not allow multiple versions of a library to be linked into a single process due to symbol conflicts. This is a limitation of those languages, not of the Google repository, and they affect any company that allows use of third-party code. The standard recommendation at Google is to avoid dependency hell by sharding large binaries and using RPCs to communicate. This development model has many advantages that have been documented elsewhere.
skybrian 10y ago
There are advantages and disadvantages. Here's an article about the good parts: https://medium.freecodecamp.com/how-google-builds-a-web-framework-5eeddd691dea https://medium.freecodecamp.com/how-google-builds-a-web-fram...
milesrout 10y ago
No, it's not common. It's something that a couple of big companies have done, and while it might work for them it does NOT work for most people. It's a really, really bad way of doing development.
- arjie 10y ago
  Why is it so bad? It seems incredibly effective for them and there are very good outcomes: • no trailing dependencies so you never have to backport fixes, maintain multiple released versions, etc. • extensive integration testing of libraries since they go live instantly • forced library developer and library user collaboration • deployed code is not out of date What's the problem with only One True Version? All the complaints seem around tooling.
  - Chyzwar 10y ago
    One True Version force you to move whole company in the same pace. Even if you have legacy project you need to keep updating because your dependencies are moving. In the same time if you want to refactor or redesign API you need to wait for all your users and communicate changes since you cannot have beta/alpha release. I am guessing that in Google people they try to avoid above issue by creating and then depreciating a lot of projects instead having new major release. Older projects will become frozen because too many things depends on them. This is visible in Google products that take years to change in any way. Gmail/Search are basically the same as long I remember. Given number of engineers in Google it is hard to see any output.
  - latch 10y ago
    Because smaller shops want the benefit of monorepo without paying the cost/discipline it requires. Mono repos without excellent unit tests and integration tests, review processes and various tools, is a disaster. And, in fairness to Google and Facebook, when they talk about the benefits of monorepo, they always mention these costs (it just gets ignored by people: Headline Driven Development) And I'm sure that the number of companies doing really good integration tests in somewhere in the 1/10000 range (or worse). Every dependency, external provider, internal API, data store, taking into account versioning....it's hard. I'm a huge believer in automated testing, I've written about it and done it for almost two decades now. My views on it have continuously evolved...and comprehensive and effective integration testing is still a complete mystery to me. The only reason I'd want to work at BigTechCo would be to learn about that specific aspect of software development.
msangi 10y ago
The company I work for has a monorepo and has a team dedicated to develop lots of tooling around it to make it manageable. Do you want to check out only the app you're working on? There is a script that grabs it and its dependencies. Do you want to do continuous integration? We have plugins to our CI server that understand when the app code or one of the dependencies has been updated. Building always off HEAD is nice and it solved the issues we had with diamond dependencies but I am not completely convinced this is the right approach
brown9-2 10y ago
I am pretty sure that engineers there don't sync the entire repo to their workspace. Look up the talk that introduced Piper and Client in the Cloud (I think it was at the BUILD conference 1-2 years ago).

tehlike 10y ago

It actually works quite nicely. Most of the google software is built internally. Ensuring everyone is running at head is a blessing (rarely a curse), because when you make a change you immediately see if it breaks something. You can also be sure everything gets all the bug fixes in their new release. This is good for various reasons, including the fact that it makes security audits significantly simpler. Some google orgs, like android, don't use the existing infra, and they kind of struggle and have to reinvent most wheels, because of that. Edit: added second paragraph

throwawayGogEng 10y ago

Coming from companies that use reasonably-sized git repos, I absolutely hated Google's VCS. Here's some of my painpoints with it: * No branches. If you want to make a temporary code branch, you create a CL (Google's version of a pull request), but never submit it. This means nobody else can collaborate on it with you, and it must be manually updated to HEAD. * No CL collaboration. Unlike Git branches, CLs can only contain changes from one user. * No stable branch. Since everything is essentially on one long branch, it's a real hassle when a project is broken at HEAD. Sure, integration tests should ideally prevent this. In practice, HEAD is often broken. Teams have created bash scripts and mailing lists to determine 'stable' old versions that can be checked out for development. * Single versions of libraries. Any library that is used is also checked into the VCS. However, only one version of the library can exist in the codebase, which is rarely updated. However, there are exceptions to this. At one point, Sergey mentioned bringing Google "up to industry standards" regarding VCS's. However, that would be a monumental task and I doubt it will happen.

jmillikin 10y ago
Everything you posted is wrong, which makes me believe you know everything you posted is wrong. Nobody writes things that inaccurate by accident. For the benefit of YC, here are corrections: > No branches. If you want to make a temporary code branch, > you create a CL (Google's version of a pull request), but > never submit it. This means nobody else can collaborate on > it with you, and it must be manually updated to HEAD. Piper has branches, which can be committed to as normal by any number of engineers. It's common to have both branches for developing certain features ("dev branches"), and branches pinned to a stable base version with cherry-picked bug fixes ("release branches"). > No CL collaboration. Unlike Git branches, CLs can only > contain changes from one user. CLs are equivalent to Git commits. Collaboration is expected to occur via a series of CLs, just like Git-based projects have large changes made via a series of smaller commits. > No stable branch. Since everything is essentially on one > long branch, it's a real hassle when a project is broken > at HEAD. Sure, integration tests should ideally prevent > this. In practice, HEAD is often broken. Teams have > created bash scripts and mailing lists to determine > 'stable' old versions that can be checked out for > development. There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases. This is similar to the Linux kernel's dev model, where stable releases are cut at known-healthy points in an evolving codebase. Important libraries define more rigorous releases, similar to Git labels, which are updated automatically every day or two. These both reduce the amount of tests that need to run, and reduce chances of errors in low-level code affecting many teams. > Single versions of libraries. Any library that is used is > also checked into the VCS. However, only one version of > the library can exist in the codebase, which is rarely > updated. However, there are exceptions to this. Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature. Three of the four languages most often used at Google (Python, C++, Go) do not allow multiple versions of a library to be linked into a single process due to symbol conflicts. This is a limitation of those languages, not of the Google repository, and they affect any company that allows use of third-party code. The standard recommendation at Google is to avoid dependency hell by sharding large binaries and using RPCs to communicate. This development model has many advantages that have been documented elsewhere.

skybrian 10y ago

There are advantages and disadvantages. Here's an article about the good parts: https://medium.freecodecamp.com/how-google-builds-a-web-framework-5eeddd691dea https://medium.freecodecamp.com/how-google-builds-a-web-fram...

milesrout 10y ago

No, it's not common. It's something that a couple of big companies have done, and while it might work for them it does NOT work for most people. It's a really, really bad way of doing development.

arjie 10y ago
Why is it so bad? It seems incredibly effective for them and there are very good outcomes: • no trailing dependencies so you never have to backport fixes, maintain multiple released versions, etc. • extensive integration testing of libraries since they go live instantly • forced library developer and library user collaboration • deployed code is not out of date What's the problem with only One True Version? All the complaints seem around tooling.
- Chyzwar 10y ago
  One True Version force you to move whole company in the same pace. Even if you have legacy project you need to keep updating because your dependencies are moving. In the same time if you want to refactor or redesign API you need to wait for all your users and communicate changes since you cannot have beta/alpha release. I am guessing that in Google people they try to avoid above issue by creating and then depreciating a lot of projects instead having new major release. Older projects will become frozen because too many things depends on them. This is visible in Google products that take years to change in any way. Gmail/Search are basically the same as long I remember. Given number of engineers in Google it is hard to see any output.
- latch 10y ago
  Because smaller shops want the benefit of monorepo without paying the cost/discipline it requires. Mono repos without excellent unit tests and integration tests, review processes and various tools, is a disaster. And, in fairness to Google and Facebook, when they talk about the benefits of monorepo, they always mention these costs (it just gets ignored by people: Headline Driven Development) And I'm sure that the number of companies doing really good integration tests in somewhere in the 1/10000 range (or worse). Every dependency, external provider, internal API, data store, taking into account versioning....it's hard. I'm a huge believer in automated testing, I've written about it and done it for almost two decades now. My views on it have continuously evolved...and comprehensive and effective integration testing is still a complete mystery to me. The only reason I'd want to work at BigTechCo would be to learn about that specific aspect of software development.

msangi 10y ago

The company I work for has a monorepo and has a team dedicated to develop lots of tooling around it to make it manageable. Do you want to check out only the app you're working on? There is a script that grabs it and its dependencies. Do you want to do continuous integration? We have plugins to our CI server that understand when the app code or one of the dependencies has been updated. Building always off HEAD is nice and it solved the issues we had with diamond dependencies but I am not completely convinced this is the right approach

brown9-2 10y ago

I am pretty sure that engineers there don't sync the entire repo to their workspace. Look up the talk that introduced Piper and Client in the Cloud (I think it was at the BUILD conference 1-2 years ago).