4 ms·

Being able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!)

by nevir 11y ago

Being able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!) is phenomenal as well.

Touche 11y ago
How is this a side effect of it being in the same repository/
- nevir 11y ago
  Say I make a change to a commonly used library (let's say deprecating a function, and replacing it with another): * I can see literally _every use_ of the old function. * I can run the tests for everyone who uses that function. * * this is automated; the build/test tooling can figure out the transitive set of build/test targets that are affected by such a change. * I can (relatively) easily update _every use_ of the deprecated call with the new hotness * I can do that all within the same commit (or set of commits, realistically) --- None of this is impossible with multiple repos, it's just a lot more difficult to coordinate
  - dekhn 11y ago
    Technically, you can't see every use of a function because dynamic dispatch mechanisms aren't available at source code time. That's why you would normally run tests of all your library's dependents (having this dependency graph is one of the most important parts of the piper/blaze system).
  - omouse 11y ago
    Yep, this is the key factor, you get to shorten your deprecation cycle to little or none. In contrast, with multiple repos, even for small projects, you will get a few days up to a month of lag time.
  - luckydude 11y ago
    We've been really crappy at marketing this but we have a multiple repos answer that is semantically close to identical to a monolithic repo. Any work flow or command line works identically in both (there are a few corner cases that don't work but almost 100% of the stuff works like you'd expect). It's sort of like a partial clone, you can clone any subset you want, work on it, add in other parts, the tool takes care of making sure the stuff you add in is lined up (if you cloned a week ago and you add in another repo, it's rolled back so it matches time/commit wise). If you want to search for something it's the same command in a collection of repos as it is in a monolithic repo: bk -U grep refactor_me but truth in advertising, that searches only what you have populated. The google answer is "ship grep to the data" so they'll search everything. Google wins if you have their datacenter. We win on UI, it's the same everywhere. Our design is much more tightly coupled than Git's submodules. We manage the subrepos so that they are in sync just like files are in sync. What I mean by that is if you have two files modified in the same commit, you can't pull that commit and only get one of the files updated, they are both updated because that's what happened in that commit. We've provided the same semantics for collections of repositories. Git doesn't, getting those semantics is an exercise left to the user. We get better performance because you can clone as little as one repository, what we call the product. It would be easy and fun to put a Unix distro in this system and have the top repo just have a makefile and you type make X11 and it populates the compiler, libc, the X11 sources, and builds them. It's commercial so maybe that's uninteresting but if you want submodules that work and scale in performance, we might be worth a look (sorry for the marketing, if that's against the rules here then I'll learn how to delete a post). Read more here, comments welcome, it's beta copy: http://www.bitkeeper.com/nested http://www.bitkeeper.com/nested
- wtetzner 11y ago
  If it's in the same repository, and you update a library, you can update all of the uses of that library in the same commit.
- kyrra 11y ago
  All dependencies between directories are specified as part of the bazel[0] BUILD files. Since dependencies between files/packages are all laid out in BUILD files, you can have a graph of all dependencies between all packages. So if you change something in 1 package it can figure out all packages that should be effected. Bazel test[1] doesn't provide dependency testing as far as I know, but it creates the framework to support doing it. [0] http://bazel.io/docs/build-encyclopedia.html http://bazel.io/docs/build-encyclopedia.html [1] http://bazel.io/docs/test-encyclopedia.html http://bazel.io/docs/test-encyclopedia.html
- dekhn 11y ago
  It's a form of efficiency due to vertical integration. I agree as others mentioned that it would be possible to do this without a single code base but it would be very unwieldly.
QuercusMax 11y ago
I just submitted a change today that affected 266,259 different build targets within google3. I was slightly terrified to make a change of this magnitude, having been at Google for less than three months, but I ran all the affected tests multiple times and it doesn't seem to have broken anything. (Fingers crossed!) Working on the google codebase is pretty awesome. This week I've made changes/improvements to libraries owned by three different teams (not counting my own), in C++ and python, when my main project is all in Java. It's super fun. The code search tool is great - it's ridiculously fast, and makes navigating through the codebase very easy.

by nevir 11y ago

Being able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!) is phenomenal as well.

Touche 11y ago

How is this a side effect of it being in the same repository/

nevir 11y ago
Say I make a change to a commonly used library (let's say deprecating a function, and replacing it with another): * I can see literally _every use_ of the old function. * I can run the tests for everyone who uses that function. * * this is automated; the build/test tooling can figure out the transitive set of build/test targets that are affected by such a change. * I can (relatively) easily update _every use_ of the deprecated call with the new hotness * I can do that all within the same commit (or set of commits, realistically) --- None of this is impossible with multiple repos, it's just a lot more difficult to coordinate
- dekhn 11y ago
  Technically, you can't see every use of a function because dynamic dispatch mechanisms aren't available at source code time. That's why you would normally run tests of all your library's dependents (having this dependency graph is one of the most important parts of the piper/blaze system).
- omouse 11y ago
  Yep, this is the key factor, you get to shorten your deprecation cycle to little or none. In contrast, with multiple repos, even for small projects, you will get a few days up to a month of lag time.
- luckydude 11y ago
  We've been really crappy at marketing this but we have a multiple repos answer that is semantically close to identical to a monolithic repo. Any work flow or command line works identically in both (there are a few corner cases that don't work but almost 100% of the stuff works like you'd expect). It's sort of like a partial clone, you can clone any subset you want, work on it, add in other parts, the tool takes care of making sure the stuff you add in is lined up (if you cloned a week ago and you add in another repo, it's rolled back so it matches time/commit wise). If you want to search for something it's the same command in a collection of repos as it is in a monolithic repo: bk -U grep refactor_me but truth in advertising, that searches only what you have populated. The google answer is "ship grep to the data" so they'll search everything. Google wins if you have their datacenter. We win on UI, it's the same everywhere. Our design is much more tightly coupled than Git's submodules. We manage the subrepos so that they are in sync just like files are in sync. What I mean by that is if you have two files modified in the same commit, you can't pull that commit and only get one of the files updated, they are both updated because that's what happened in that commit. We've provided the same semantics for collections of repositories. Git doesn't, getting those semantics is an exercise left to the user. We get better performance because you can clone as little as one repository, what we call the product. It would be easy and fun to put a Unix distro in this system and have the top repo just have a makefile and you type make X11 and it populates the compiler, libc, the X11 sources, and builds them. It's commercial so maybe that's uninteresting but if you want submodules that work and scale in performance, we might be worth a look (sorry for the marketing, if that's against the rules here then I'll learn how to delete a post). Read more here, comments welcome, it's beta copy: http://www.bitkeeper.com/nested http://www.bitkeeper.com/nested
wtetzner 11y ago
If it's in the same repository, and you update a library, you can update all of the uses of that library in the same commit.
kyrra 11y ago
All dependencies between directories are specified as part of the bazel[0] BUILD files. Since dependencies between files/packages are all laid out in BUILD files, you can have a graph of all dependencies between all packages. So if you change something in 1 package it can figure out all packages that should be effected. Bazel test[1] doesn't provide dependency testing as far as I know, but it creates the framework to support doing it. [0] http://bazel.io/docs/build-encyclopedia.html http://bazel.io/docs/build-encyclopedia.html [1] http://bazel.io/docs/test-encyclopedia.html http://bazel.io/docs/test-encyclopedia.html
dekhn 11y ago
It's a form of efficiency due to vertical integration. I agree as others mentioned that it would be possible to do this without a single code base but it would be very unwieldly.

QuercusMax 11y ago

I just submitted a change today that affected 266,259 different build targets within google3. I was slightly terrified to make a change of this magnitude, having been at Google for less than three months, but I ran all the affected tests multiple times and it doesn't seem to have broken anything. (Fingers crossed!) Working on the google codebase is pretty awesome. This week I've made changes/improvements to libraries owned by three different teams (not counting my own), in C++ and python, when my main project is all in Java. It's super fun. The code search tool is great - it's ridiculously fast, and makes navigating through the codebase very easy.