4 ms·
I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'
by dekhn 11y ago
I'm a google software engineer and it's nice to see this public article about our software control system. I think it has plusses and minuses, but one thing I'll say is that when you're in the coding flow, working on a single code base with thousands of engineers can be an intensely awesome experience.
Part of my job- although it's not listed as a responsibility- is updating a few key scientific python packages. When I do this, I get immediate feedback on which tests get broken and I fix those problems for other teams along side my upgrades. This sort of continuous integration has completely changed how I view modern software development and testing.
- nevir 11y agoBeing able to make sweeping changes to a shared piece of code, and ensure that everyone's up to date (Hi, Rosie!) and not broken by your change (yay TAP train!) is phenomenal as well.
- Touche 11y agoHow is this a side effect of it being in the same repository/
- nevir 11y agoSay I make a change to a commonly used library (let's say deprecating a function, and replacing it with another): * I can see literally _every use_ of the old function. * I can run the tests for everyone who uses that function. * * this is automated; the build/test tooling can figure out the transitive set of build/test targets that are affected by such a change. * I can (relatively) easily update _every use_ of the deprecated call with the new hotness * I can do that all within the same commit (or set of commits, realistically) --- None of this is impossible with multiple repos, it's just a lot more difficult to coordinate
- dekhn 11y agoTechnically, you can't see every use of a function because dynamic dispatch mechanisms aren't available at source code time. That's why you would normally run tests of all your library's dependents (having this dependency graph is one of the most important parts of the piper/blaze system).
- omouse 11y agoYep, this is the key factor, you get to shorten your deprecation cycle to little or none. In contrast, with multiple repos, even for small projects, you will get a few days up to a month of lag time.
- luckydude 11y agoWe've been really crappy at marketing this but we have a multiple repos answer that is semantically close to identical to a monolithic repo. Any work flow or command line works identically in both (there are a few corner cases that don't work but almost 100% of the stuff works like you'd expect). It's sort of like a partial clone, you can clone any subset you want, work on it, add in other parts, the tool takes care of making sure the stuff you add in is lined up (if you cloned a week ago and you add in another repo, it's rolled back so it matches time/commit wise). If you want to search for something it's the same command in a collection of repos as it is in a monolithic repo: bk -U grep refactor_me but truth in advertising, that searches only what you have populated. The google answer is "ship grep to the data" so they'll search everything. Google wins if you have their datacenter. We win on UI, it's the same everywhere. Our design is much more tightly coupled than Git's submodules. We manage the subrepos so that they are in sync just like files are in sync. What I mean by that is if you have two files modified in the same commit, you can't pull that commit and only get one of the files updated, they are both updated because that's what happened in that commit. We've provided the same semantics for collections of repositories. Git doesn't, getting those semantics is an exercise left to the user. We get better performance because you can clone as little as one repository, what we call the product. It would be easy and fun to put a Unix distro in this system and have the top repo just have a makefile and you type make X11 and it populates the compiler, libc, the X11 sources, and builds them. It's commercial so maybe that's uninteresting but if you want submodules that work and scale in performance, we might be worth a look (sorry for the marketing, if that's against the rules here then I'll learn how to delete a post). Read more here, comments welcome, it's beta copy: http://www.bitkeeper.com/nested http://www.bitkeeper.com/nested
- wtetzner 11y agoIf it's in the same repository, and you update a library, you can update all of the uses of that library in the same commit.
- kyrra 11y agoAll dependencies between directories are specified as part of the bazel[0] BUILD files. Since dependencies between files/packages are all laid out in BUILD files, you can have a graph of all dependencies between all packages. So if you change something in 1 package it can figure out all packages that should be effected. Bazel test[1] doesn't provide dependency testing as far as I know, but it creates the framework to support doing it. [0] http://bazel.io/docs/build-encyclopedia.html http://bazel.io/docs/build-encyclopedia.html [1] http://bazel.io/docs/test-encyclopedia.html http://bazel.io/docs/test-encyclopedia.html
- dekhn 11y agoIt's a form of efficiency due to vertical integration. I agree as others mentioned that it would be possible to do this without a single code base but it would be very unwieldly.
- QuercusMax 11y agoI just submitted a change today that affected 266,259 different build targets within google3. I was slightly terrified to make a change of this magnitude, having been at Google for less than three months, but I ran all the affected tests multiple times and it doesn't seem to have broken anything. (Fingers crossed!) Working on the google codebase is pretty awesome. This week I've made changes/improvements to libraries owned by three different teams (not counting my own), in C++ and python, when my main project is all in Java. It's super fun. The code search tool is great - it's ridiculously fast, and makes navigating through the codebase very easy.
- Strikingwolf 11y agoIt sounds like a really awesome workflow, although parts of it really only work on very large teams. I bet the version control system would be very cool though, hopefully it gets a github equivalent and has better branching than mercurial
- amelius 11y agoCould you tell something about the level of documentation? For instance, do you have to write a paragraph of documentation for every function that you add to the system? How about adding preconditions and postconditions, and other invariants? Also, is the code that you add to the repository always inspected by other people? Is that done systematically?
- lrem 11y agoIt is mandatory that each code change is inspected for correctness, language style and approved by a code owner (all 3 may be the same person, or require multiple, depending on the situation).
- amelius 11y agoBut I guess the original programmer can never be the same person as those other 3 roles? :)
- deleted 11y ago[deleted]
- mvgoogler 11y ago> Could you tell something about the level of documentation? For instance, > do you have to write a paragraph of documentation for every function that > you add to the system? How about adding preconditions and postconditions, > and other invariants? It varies a lot by project and by how widely used the code is. There are few (if any) strictly enforced rules for documentation or invariant checking. You basically have to convince at least one other engineer that what you have is sufficient. The documentation is _generally_ pretty decent for core libraries, but sometimes you just have to read the code.
- tajen 11y agoIf I recoup with latest news: The ranking algorithm is in one repo, Chrome and Android in another, and all the rest is in Piper? What is all the rest? Gmail, Calendar, the defunct Reader, the car, Maps, and OnHub... How does that fit with Alphabet? Is Piper a cross-functional initiative of Alphabet now?
- kyrra 11y agoLook at the companies that are non-Google entities (the ones under Alphabet that aren't under Google): * Google X - moonshot projects that aren't software centric. * Google Fiber - mainly a infrstructure setup. May share some stuff in piper. * Google Ventures - investment arm, not code related. * Google Capital - more investment stuff. * Calico - R&D for biotech. Anything that would be software centric will probably still live under Google inc. As well, you are forgetting so many products that Google has. There are various lists of them out there [0][1]. [0] http://www.google.com/about/products/ http://www.google.com/about/products/ [1] https://en.wikipedia.org/wiki/List_of_Google_products https://en.wikipedia.org/wiki/List_of_Google_products
- QuercusMax 11y agoDon't forget Google Life Sciences -- we have lots of software. (The team I'm on is 100% software, and we're around 25 people split among 3 sub-teams.) All our code is in piper, and depends heavily on the rest of the google infrastructure.
- ocdtrekkie 11y agoIs Google going to split this out of their infrastructure, or is this whole separation between "Alphabet" companies and "Google" really just a marketing ploy for no change at all?
- QuercusMax 11y agoAt least as far as Google Life Sciences is concerned, there are multiple reasons that are known internally (at least within GLS) that I cannot comment on at this time. An astute observer may be able to come up with some plausible reasons (at least for GLS) as to why such an arrangement would be desirable. As far as splitting up the infrastructure, we're still going to depend on Google infrastructure and use their services where it makes sense.