43 ms·

> Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies.

> Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies. This sounds like horror to me: it's essentially a forced update to the latest version of all in-house dependencies. Interesting article though. It feels like there's a broader lesson here about not getting obsessed with what for some reason have become best practices, and really taking the time to think independently about the pros and cons of various approaches.

lpolovets 10y ago
It's been a long, long time since I worked at Google, but IIRC the build system was like Maven: you could depend on the latest version of something, or a specific version, and only the dependencies you needed would get pulled from the global repo when you checked out code. If your code depended on a specific version of some piece of code, then you'd be immune to future updates to that piece of code unless you updated your dependency config. (Hopefully I'm remembering all of this correctly..)
- thirtyseven 10y ago
  Nope, that's not quite how it worked. Except for a few critical libraries that were pinned to specific versions, every build built against HEAD. However, a presubmit hook would run tests on the transitive closure of code depending on the changed code, and if anything broke, it was your responsibility as the committer to fix it or roll it back. This is really what kept things stable. If you maintained a a common library and wanted to update it, you had to either keep things backwards compatible, write an automated refactor across the codebase (which wasn't too hard because of awesome tooling), or you would get dozens of angry emails when everything broke. If you relied on a library, you had to be sure to write solid e2e tests or your code might silently break and it would mostly be your fault. I definitely miss the big G, so many resources were dedicated to engineer productivity and tooling. It's what makes the monorepo work.
  - lpolovets 10y ago
    Thanks! I appreciate the correction. I miss a lot of the resources, too. I also miss the spirit of dedicating serious attention to code quality. Things like Testing Fix-it and Documentation Fix-it days were awesome.
    - squeaky-clean 10y ago
      > Things like Testing Fix-it and Documentation Fix-it days were awesome. Can you elaborate on this? Are they just days where the team goes "Ok, no need to stress about new features today, let's just catch on on test coverage / documentation" ? If so, that sounds pretty wonderful.
      - lpolovets 10y ago
        It was company-wide. Every month or two, there would be a "documentation fix-it day" or something similar. The theme would vary: sometimes it was docs, sometimes unit tests, sometimes localization, etc. If memory serves correctly, the idea was that unless you had a high priority/production bug, you'd spend your day working on the theme of the fix-it.
  - krashidov 10y ago
    Why did you leave?
    - iainmerrick 10y ago
      "Oh yeah, and this is our awesome in-house build system. You're really going to enjoy working on ads!"
      - packetslave 10y ago
        Yes, working in a product area that's directly responsible for generating $N*10^9 of revenue per year is just terrible. I'm sure they don't do ANYTHING interesting, technology-wise.
- thirtyseven 10y ago
  Nope, that's not quite how it worked. Except for a few critical libraries that were pinned to specific versions, every build built against HEAD. However, a presubmit hook would run tests on the transitive closure of code depending on the changed code, and if anything broke, it was your responsibility as the committer to fix it or roll it back. This is really what kept things stable. If you maintained a a common library and wanted to update it, you had to either keep things backwards compatible, write an automated refactor across the codebase (which wasn't too hard because of awesome tooling), or you would get dozens of angry emails when everything broke. If you relied on a library, you had to be sure to write solid e2e tests or your code might silently break and it would mostly be your fault. I definitely miss the big G, so many resources were dedicated to engineer productivity and tooling. It's what makes the monorepo work.
- kyrra 10y ago
  FYI, Google open sourced their build system: http://www.bazel.io http://www.bazel.io
s17tnet 10y ago
No, it isn't if artifacts/projects declare dependencies against fixed, stable version archived somewhere (e.g. artifactory). Just I wonder the repository performance and the every day workflow.
- nevir 10y ago
  Pretty much everything builds against head, and very little is in versioned archives. There's a ton of fantastic artifact caching (that gets invalidated whenever source changes; which is frequently), distributing building, and some really nifty source code juggling
mining 10y ago
Mmm, two things of note: 1) As has been mentioned, I'm fairly sure you can fix on a given dependency. 2) Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen.
- 746F7475 10y ago
  >Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen. This doesn't prevent build breaking changes, it just makes the team using the library aware what is broken and they can have fix ready when they want to upgrade their dependencies.
  - Morgawr 10y ago
    Afaik you cannot submit code that breaks any test in the repo. I mean, there are ways to force it, but unless it's a special case you really don't want to.
  - 746F7475 10y ago
    Judging by the downvotes, obviously some people have no idea how maintaining libraries work.
Qwertious 10y ago
I think the way you should think of it is that there is only one "canonical" version - otherwise the "canonical" version is fragmented, and the fragments can contradict eachother. And god damn, I hate Mark Shuttleworth for using that name for his company.
- Asooka 10y ago
  https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html Essentially there are two rules here: don't post or upvote crap links, and don't be rude or dumb in comment threads. The most important principle on HN, though, is to make thoughtful comments. Thoughtful in both senses: civil and substantial. So the principle here is: don't say anything you wouldn't say face to face. Emphasis mine. That attack on Mark Shuttleworth was absolutely uncalled for and in stark conflict with HN's community guidelines.
skybrian 10y ago
It's a "forced" update, but the downstream dependencies have veto power: if their tests fail, the change gets rolled back. Of course in practice, it depends on how important the app is. If you break gmail it's definitely getting rolled back. If it's just a one-off abandoned tool somewhere doing something they're not supposed to do anyway, or some tests that flake all the time, maybe it will be ignored. If you are upstream of a lot of important code, you might not be able to move as quickly as you like and that's because you have a lot of responsibility. Migrations need to be planned carefully. Unlike in open source, you cannot make an incompatible change, bump the major version number, and call it a day. Migrating the downstream dependencies is your job. Version numbers are mostly meaningless and compatibility is judged by actually compiling code and running tests. You soon learn that if you can fix a bug without changing a public API it's a whole lot easier. But if you need to change an API, there's a way to do it.

lpolovets 10y ago

It's been a long, long time since I worked at Google, but IIRC the build system was like Maven: you could depend on the latest version of something, or a specific version, and only the dependencies you needed would get pulled from the global repo when you checked out code. If your code depended on a specific version of some piece of code, then you'd be immune to future updates to that piece of code unless you updated your dependency config. (Hopefully I'm remembering all of this correctly..)

thirtyseven 10y ago
Nope, that's not quite how it worked. Except for a few critical libraries that were pinned to specific versions, every build built against HEAD. However, a presubmit hook would run tests on the transitive closure of code depending on the changed code, and if anything broke, it was your responsibility as the committer to fix it or roll it back. This is really what kept things stable. If you maintained a a common library and wanted to update it, you had to either keep things backwards compatible, write an automated refactor across the codebase (which wasn't too hard because of awesome tooling), or you would get dozens of angry emails when everything broke. If you relied on a library, you had to be sure to write solid e2e tests or your code might silently break and it would mostly be your fault. I definitely miss the big G, so many resources were dedicated to engineer productivity and tooling. It's what makes the monorepo work.
- lpolovets 10y ago
  Thanks! I appreciate the correction. I miss a lot of the resources, too. I also miss the spirit of dedicating serious attention to code quality. Things like Testing Fix-it and Documentation Fix-it days were awesome.
  - squeaky-clean 10y ago
    > Things like Testing Fix-it and Documentation Fix-it days were awesome. Can you elaborate on this? Are they just days where the team goes "Ok, no need to stress about new features today, let's just catch on on test coverage / documentation" ? If so, that sounds pretty wonderful.
    - lpolovets 10y ago
      It was company-wide. Every month or two, there would be a "documentation fix-it day" or something similar. The theme would vary: sometimes it was docs, sometimes unit tests, sometimes localization, etc. If memory serves correctly, the idea was that unless you had a high priority/production bug, you'd spend your day working on the theme of the fix-it.
- krashidov 10y ago
  Why did you leave?
  - iainmerrick 10y ago
    "Oh yeah, and this is our awesome in-house build system. You're really going to enjoy working on ads!"
    - packetslave 10y ago
      Yes, working in a product area that's directly responsible for generating $N*10^9 of revenue per year is just terrible. I'm sure they don't do ANYTHING interesting, technology-wise.
thirtyseven 10y ago
Nope, that's not quite how it worked. Except for a few critical libraries that were pinned to specific versions, every build built against HEAD. However, a presubmit hook would run tests on the transitive closure of code depending on the changed code, and if anything broke, it was your responsibility as the committer to fix it or roll it back. This is really what kept things stable. If you maintained a a common library and wanted to update it, you had to either keep things backwards compatible, write an automated refactor across the codebase (which wasn't too hard because of awesome tooling), or you would get dozens of angry emails when everything broke. If you relied on a library, you had to be sure to write solid e2e tests or your code might silently break and it would mostly be your fault. I definitely miss the big G, so many resources were dedicated to engineer productivity and tooling. It's what makes the monorepo work.
kyrra 10y ago
FYI, Google open sourced their build system: http://www.bazel.io http://www.bazel.io

s17tnet 10y ago

No, it isn't if artifacts/projects declare dependencies against fixed, stable version archived somewhere (e.g. artifactory). Just I wonder the repository performance and the every day workflow.

nevir 10y ago
Pretty much everything builds against head, and very little is in versioned archives. There's a ton of fantastic artifact caching (that gets invalidated whenever source changes; which is frequently), distributing building, and some really nifty source code juggling

mining 10y ago

Mmm, two things of note: 1) As has been mentioned, I'm fairly sure you can fix on a given dependency. 2) Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen.

746F7475 10y ago
>Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen. This doesn't prevent build breaking changes, it just makes the team using the library aware what is broken and they can have fix ready when they want to upgrade their dependencies.
- Morgawr 10y ago
  Afaik you cannot submit code that breaks any test in the repo. I mean, there are ways to force it, but unless it's a special case you really don't want to.
- 746F7475 10y ago
  Judging by the downvotes, obviously some people have no idea how maintaining libraries work.

Qwertious 10y ago

I think the way you should think of it is that there is only one "canonical" version - otherwise the "canonical" version is fragmented, and the fragments can contradict eachother. And god damn, I hate Mark Shuttleworth for using that name for his company.

Asooka 10y ago
https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html Essentially there are two rules here: don't post or upvote crap links, and don't be rude or dumb in comment threads. The most important principle on HN, though, is to make thoughtful comments. Thoughtful in both senses: civil and substantial. So the principle here is: don't say anything you wouldn't say face to face. Emphasis mine. That attack on Mark Shuttleworth was absolutely uncalled for and in stark conflict with HN's community guidelines.

skybrian 10y ago

It's a "forced" update, but the downstream dependencies have veto power: if their tests fail, the change gets rolled back. Of course in practice, it depends on how important the app is. If you break gmail it's definitely getting rolled back. If it's just a one-off abandoned tool somewhere doing something they're not supposed to do anyway, or some tests that flake all the time, maybe it will be ignored. If you are upstream of a lot of important code, you might not be able to move as quickly as you like and that's because you have a lot of responsibility. Migrations need to be planned carefully. Unlike in open source, you cannot make an incompatible change, bump the major version number, and call it a day. Migrating the downstream dependencies is your job. Version numbers are mostly meaningless and compatibility is judged by actually compiling code and running tests. You soon learn that if you can fix a bug without changing a public API it's a whole lot easier. But if you need to change an API, there's a way to do it.