6 ms·

> The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something...The second version is upgrade early, upgra

> The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something...The second version is upgrade early, upgrade often...Google is an excellent example of a company that does this. This is misleading. My understanding of Google's internal build systems is that they ruthlessly lock down the version of every single dependency, up to and including the compiler binary itself. They then provide tooling on top of that to make it easier to upgrade those locked down versions regularly. The core problem is that when your codebase gets to the kind of scale that Google's has, if you can't reproduce the entire universe of your dependencies, there is no way any historical commit of anything will ever build. That makes it difficult to do basic things like maintain release branches or bisect bugs. > if you want to lock down version numbers for a specific release, have an automated tool supply the right ones for you. And make it trivial to upgrade early, and upgrade often. This part sounds like a more accurate description of what Google and others do, yes.

verroq 7y ago
Yeah but only google is google. You are not google, your code doesn’t need to google scale and you don’t need to go to their extremes to manage dependencies. They do it because they are forced to, doesn’t mean it is right or their way is how it should work for everyone.
- OJFord 7y ago
  This isn't really a 'wow, look at the crazy stuff Google needs' thing. Any tiny open source project benefits from a reproducible build (when you come back to it months later) and also new versions (with fixed vulnerabilities, and compatibility with the new thing you're trying to do).
  - verroq 7y ago
    No not really, any tiny open source project isn’t worth the hassle of making a reproducible build for.
    - OJFord 7y ago
      Well, I look at reproducible as a scale (and incidentally, with an increase in effort as you slide along it, too). A certain amount of reproducibility - a container, pinned dependencies - gives such large reward for how easy it is to achieve that it absolutely is worth it for a tiny open source project. Worrying about the possibility of unavailable package registries and revoked signing keys, on the other hand, probably isn't. It's a trade-off. But you certainly don't need to be Google-scale for some of it to be very worth your while.
  - michaelmior 7y ago
    I think this depends on your definition of "reproducible build." If you're talking about builds being bit for bit identical, that might not be worthwhile given the complexities of doing so with most build tools. But if you mean the same versions being used for dependencies, then absolutely.
    - OJFord 7y ago
      Yes, I agree completely, as replied to sibling: https://news.ycombinator.com/item?id=20032980 https://news.ycombinator.com/item?id=20032980
- oconnor663 7y ago
  I haven't worked at Google, but I have worked at Facebook, and I can say with some confidence that in this respect Facebook is Google too :) For sure there are tradeoffs for big projects that don't make sense for small ones. But there are also times where big projects need a tool that's "just better" than what small projects need, and once that tool has been built it can make sense for everyone to use it. I think good, strong, convenient version pinning is an example of the latter, when the tools are available. That was the inspiration for the peru tool (https://github.com/buildinspace/peru https://github.com/buildinspace/peru).
  - weberc2 7y ago
    I agree with this, but I think to the extent that such tools are lacking (or at least that the overhead is prohibitively high for smaller projects), the parent is correct. Thanks for tipping me off to peru; hadn't seen that before.
OJFord 7y ago
For an easy open source example of such tooling, see Pyup. We use it to do exactly that: pin down every dependency to an exact version, but automatically build and test with newly released versions of each one. (And then merge the upgrade, after fixing any issue.)
- toomuchtodo 7y ago
  I’d imagine this is easier now with dependabot joining Github, being free for all, and implementing a proper CI test system for your repos. Logically, the next step is supporting such infra for containers. Automate all the mundane regression/security/functionality testing while driving dependency upgrades forward.
- jrochkind1 7y ago
  Or the original ruby bundler, which locks down exact versions in a `Gemfile.lock`, but lets you easily update to latest version(s) with `bundle update`, which will update the `Gemfile.lock`. Actually, it goes further, `bundle update` doesn't update just to "latest version", but to latest version allowed by your direct or transitive version restrictions. I believe `yarn` ends up working similar in JS? To me, this is definitely the best practice pattern for dependency management. You definitely need to ruthlessly lock down the exact versions used, in a file that's checked into the repo -- so all builds will use the exact same versions, whether deployment builds or CI builds or whatever. But you also need tooling that lets you easily update the versions, and change the file recording the exact versions that's in the repo. I'm not sure how/if you can do that reliably and easily with the sorts of dependencies discussed in the OP or in Dockerfiles in general... but it seems clear to me it's the goal.
  - deleted 7y ago
    [deleted]
dalore 7y ago
Yes they have a huge mono repository and tooling to update projects in it to specific versions. You don't get a choice really. You can go home one night with your project on say Java 7 and then wake up and find someone has migrated it to Java 8 because they've decided it's Java 8 now.
- paulddraper 7y ago
  But that change only happened once all the tests for "your project" passed on Java 8.
  - HeWhoLurksLate 7y ago
    So now you're saying that I have to write tests? /s
    - dodobirdlord 7y ago
      There is in fact something of a philosophy at Google that it's both your problem and your fault if a dependency upgrades in a way that breaks your project without breaking any of your tests.
  - koala_man 7y ago
    This is the crucial difference. Library developers at Google know all their reverse dependencies, and can easily build, test, notify, or fix all of them. You can't do that with external FOSS libraries. The closest thing we have is deprecation log messages and blog posts with migration guides.
    - MaulingMonkey 7y ago
      > The closest thing we have is deprecation log messages and blog posts with migration guides. Rust has crater, which can at least build/test/notify over a large chunk of the rust FOSS ecosystem. It won't pick up every project, granted, and I haven't heard of anyone really using it outside of compiler/stdlib development itself, but it's an example of something a bit closer to what google has.
eitland 7y ago
If I remember correctly Angular comes with unpinned dependencies.
- paulddraper 7y ago
  The package.json file specifies unpinned dependencies. The package-lock.json or yarn.lock or similar specifies the pinned dependencies.
  - eitland 7y ago
    Correct. But neither the package-lock.json or the yarn.lock file is part of what you get when you create an angular project using the angular cli, meaning that the versions aren't pinned from googles side.
    - paulddraper 7y ago
      A. Nobody wants that. The reality is you're going to be using dozens of other libraries and will welcome wiggle room on how many versions your output includes. You want the list to be consistent and versioned, but you don't need the exact same one as Google. B. If you really want to know what Angular is being tested with, see https://github.com/angular/angular/blob/master/yarn.lock https://github.com/angular/angular/blob/master/yarn.lock
paulddraper 7y ago
A larger problem is that Docker is nearly inherently unreproducible. Downloading and installing system packages lists, etc. For this reason, Google doesn't use Docker at all. It writes the OCI images more or less directly. https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker
- merb 7y ago
  well also docker's sha hash for each layer is just a random sha and not a sha of the actual content. also it includes a timestamp. thus docker is not reporducible. but google actually has kaniko and jib which correct that problem.
  - jonjonsonjr 7y ago
    Your first point is incorrect. That was true of v1 docker images, but layers have been content-addressable for a while now. Your second point is absolutely correct - we strip timestamps from everything which tends to confuse folks :)
    - baylisscg 7y ago
      Thou shalt be able to recognise the Unix Epoch 0
    - merb 7y ago
      > true of v1 docker images yeah I missed that one. but basically it was still a pain. But yeah google's tools are actually awesome. I mean I even own a "fork" (or basically a plugin) for sbt which brings jib to sbt: https://github.com/schmitch/sbt-jib https://github.com/schmitch/sbt-jib It's just so much easier to build java/scala images with jib than it is with plain docker.
xorcist 7y ago
> Google is an excellent example of a company that does this. Which part of Google would that be? My impression is the complete opposite, dependencies are not only locked down and sometimes even maintained internally.

verroq 7y ago

Yeah but only google is google. You are not google, your code doesn’t need to google scale and you don’t need to go to their extremes to manage dependencies. They do it because they are forced to, doesn’t mean it is right or their way is how it should work for everyone.

OJFord 7y ago
This isn't really a 'wow, look at the crazy stuff Google needs' thing. Any tiny open source project benefits from a reproducible build (when you come back to it months later) and also new versions (with fixed vulnerabilities, and compatibility with the new thing you're trying to do).
- verroq 7y ago
  No not really, any tiny open source project isn’t worth the hassle of making a reproducible build for.
  - OJFord 7y ago
    Well, I look at reproducible as a scale (and incidentally, with an increase in effort as you slide along it, too). A certain amount of reproducibility - a container, pinned dependencies - gives such large reward for how easy it is to achieve that it absolutely is worth it for a tiny open source project. Worrying about the possibility of unavailable package registries and revoked signing keys, on the other hand, probably isn't. It's a trade-off. But you certainly don't need to be Google-scale for some of it to be very worth your while.
- michaelmior 7y ago
  I think this depends on your definition of "reproducible build." If you're talking about builds being bit for bit identical, that might not be worthwhile given the complexities of doing so with most build tools. But if you mean the same versions being used for dependencies, then absolutely.
  - OJFord 7y ago
    Yes, I agree completely, as replied to sibling: https://news.ycombinator.com/item?id=20032980 https://news.ycombinator.com/item?id=20032980
oconnor663 7y ago
I haven't worked at Google, but I have worked at Facebook, and I can say with some confidence that in this respect Facebook is Google too :) For sure there are tradeoffs for big projects that don't make sense for small ones. But there are also times where big projects need a tool that's "just better" than what small projects need, and once that tool has been built it can make sense for everyone to use it. I think good, strong, convenient version pinning is an example of the latter, when the tools are available. That was the inspiration for the peru tool (https://github.com/buildinspace/peru https://github.com/buildinspace/peru).
- weberc2 7y ago
  I agree with this, but I think to the extent that such tools are lacking (or at least that the overhead is prohibitively high for smaller projects), the parent is correct. Thanks for tipping me off to peru; hadn't seen that before.

OJFord 7y ago

For an easy open source example of such tooling, see Pyup. We use it to do exactly that: pin down every dependency to an exact version, but automatically build and test with newly released versions of each one. (And then merge the upgrade, after fixing any issue.)

toomuchtodo 7y ago
I’d imagine this is easier now with dependabot joining Github, being free for all, and implementing a proper CI test system for your repos. Logically, the next step is supporting such infra for containers. Automate all the mundane regression/security/functionality testing while driving dependency upgrades forward.
jrochkind1 7y ago
Or the original ruby bundler, which locks down exact versions in a `Gemfile.lock`, but lets you easily update to latest version(s) with `bundle update`, which will update the `Gemfile.lock`. Actually, it goes further, `bundle update` doesn't update just to "latest version", but to latest version allowed by your direct or transitive version restrictions. I believe `yarn` ends up working similar in JS? To me, this is definitely the best practice pattern for dependency management. You definitely need to ruthlessly lock down the exact versions used, in a file that's checked into the repo -- so all builds will use the exact same versions, whether deployment builds or CI builds or whatever. But you also need tooling that lets you easily update the versions, and change the file recording the exact versions that's in the repo. I'm not sure how/if you can do that reliably and easily with the sorts of dependencies discussed in the OP or in Dockerfiles in general... but it seems clear to me it's the goal.
- deleted 7y ago
  [deleted]

dalore 7y ago

Yes they have a huge mono repository and tooling to update projects in it to specific versions. You don't get a choice really. You can go home one night with your project on say Java 7 and then wake up and find someone has migrated it to Java 8 because they've decided it's Java 8 now.

paulddraper 7y ago
But that change only happened once all the tests for "your project" passed on Java 8.
- HeWhoLurksLate 7y ago
  So now you're saying that I have to write tests? /s
  - dodobirdlord 7y ago
    There is in fact something of a philosophy at Google that it's both your problem and your fault if a dependency upgrades in a way that breaks your project without breaking any of your tests.
- koala_man 7y ago
  This is the crucial difference. Library developers at Google know all their reverse dependencies, and can easily build, test, notify, or fix all of them. You can't do that with external FOSS libraries. The closest thing we have is deprecation log messages and blog posts with migration guides.
  - MaulingMonkey 7y ago
    > The closest thing we have is deprecation log messages and blog posts with migration guides. Rust has crater, which can at least build/test/notify over a large chunk of the rust FOSS ecosystem. It won't pick up every project, granted, and I haven't heard of anyone really using it outside of compiler/stdlib development itself, but it's an example of something a bit closer to what google has.

eitland 7y ago

If I remember correctly Angular comes with unpinned dependencies.

paulddraper 7y ago
The package.json file specifies unpinned dependencies. The package-lock.json or yarn.lock or similar specifies the pinned dependencies.
- eitland 7y ago
  Correct. But neither the package-lock.json or the yarn.lock file is part of what you get when you create an angular project using the angular cli, meaning that the versions aren't pinned from googles side.
  - paulddraper 7y ago
    A. Nobody wants that. The reality is you're going to be using dozens of other libraries and will welcome wiggle room on how many versions your output includes. You want the list to be consistent and versioned, but you don't need the exact same one as Google. B. If you really want to know what Angular is being tested with, see https://github.com/angular/angular/blob/master/yarn.lock https://github.com/angular/angular/blob/master/yarn.lock

paulddraper 7y ago

A larger problem is that Docker is nearly inherently unreproducible. Downloading and installing system packages lists, etc. For this reason, Google doesn't use Docker at all. It writes the OCI images more or less directly. https://github.com/bazelbuild/rules_docker https://github.com/bazelbuild/rules_docker

merb 7y ago
well also docker's sha hash for each layer is just a random sha and not a sha of the actual content. also it includes a timestamp. thus docker is not reporducible. but google actually has kaniko and jib which correct that problem.
- jonjonsonjr 7y ago
  Your first point is incorrect. That was true of v1 docker images, but layers have been content-addressable for a while now. Your second point is absolutely correct - we strip timestamps from everything which tends to confuse folks :)
  - baylisscg 7y ago
    Thou shalt be able to recognise the Unix Epoch 0
  - merb 7y ago
    > true of v1 docker images yeah I missed that one. but basically it was still a pain. But yeah google's tools are actually awesome. I mean I even own a "fork" (or basically a plugin) for sbt which brings jib to sbt: https://github.com/schmitch/sbt-jib https://github.com/schmitch/sbt-jib It's just so much easier to build java/scala images with jib than it is with plain docker.

xorcist 7y ago

> Google is an excellent example of a company that does this. Which part of Google would that be? My impression is the complete opposite, dependencies are not only locked down and sometimes even maintained internally.