3 ms·

I think we are using two different meaning of "reproducible". To me, saying that a build is reproducible means that anyone is able to independently build a bit

I think we are using two different meaning of "reproducible". To me, saying that a build is reproducible means that anyone is able to independently build a bit-for-bit exact copy of an artifact given only its description (source code + build scripts, Dockerfile, ...), more or less in the sense of [0]. For this definition, even content-addressability is not enough. [0] https://reproducible-builds.org/ https://reproducible-builds.org/

yebyen 9y ago
Well, I think you disagree on the meaning of the word "artifact." GP cares that all production deployments are identical to the development environment that went through system tests. To ops, a reproducible build is one that has been uploaded to the registry, because it can be ported over to the production cluster and it will be bit-for-bit identical. It is not necessary to reproduce the same build bit-for-bit because you have kept a copy and distributed it. You are not rebuilding the same thing twice because that is an inefficient use of resources. Nobody really cares that the timestamps are different when the builds started at different times of day, it is evident therefore that you will not produce bit-for-bit identical builds as long as your filesystem includes these kinds of metadata and details. If you built it again, it is to make a change, so there is no point in insisting on bit-for-bit reproducibility in your build process (unless you are actually trying to optimize for immutable infrastructure, in which case more power to you, and it might be a particularly desirable trait to have. Not for nothing!)
- nqzero 9y ago
  that's a mirror, not a reproducible build
  - yebyen 9y ago
    No, you're right, I read the "reproducible builds" page and it's a good formal definition for the term. I just think that if you ask ten average devops people on the street what "reproducible" means in the context of their devops jobs, eight of them at minimum are not going to know this definition, or insist on a bit-for-bit representation that arises directly from pure source code like what meets the definition at http://reproducible-builds.org http://reproducible-builds.org We're going to think you mean "has a BDR plan in place." Or maybe I've underestimated the reach of this concept.
- FooBarWidget 9y ago
  > Nobody really cares that the timestamps are different when the builds started at different times of day, it is evident therefore that you will not produce bit-for-bit identical builds as long as your filesystem includes these kinds of metadata and details. Do you think different timestamps is all that you need to worry about? How about the fact that some source tarballs that you depend on may have moved location? Or that your Dockerfile contains 'apt-get upgrade' but some upgraded package breaks your build whereas it did not before? Or that your Dockerfile curls some binary from the Internet but now that server uses an SSL cipher suite that is now incompatible with your OpenSSL version? All problems that I encountered.
  - yebyen 9y ago
    Hey FooBarWidget! :D I was not speaking in terms of formal correctness in terms of the definition of reproducible builds, I was speaking of practical implementation for a production deployment. All of the issues you mentioned are (at least temporarily) resolvable by keeping a mirror of the builds so that you can reproduce the production environment without rebuilding if it goes away. I'm just defending the (apparently ops) person who was going by the original dictionary definition of "reproducible" which predates the "reproducible builds" definition of reproducible. If my manager asks me to make sure that my environment is reproducible, I assure you she is not talking about the formal definition that is being used here. I'm not saying that one shouldn't care about these things, but I am saying that many won't care about them. If you're doing a new build and the build results in upgrading to a newer version of a package, then that is a new build. It won't match. If you're doing a rebuild as an exercise to see if the output matches the previous build, then you're concerned about different things than the ops person will be. If my production instances are deleted by mistake, I'll be redeploying from the backup images, I won't be rebuilding my sources from scratch unless the backups also failed. I agree that it is a sort of problem that people usually don't build from scratch, it's just not one of the kinds of problems that my manager will ever be likely to ask me to solve.

yebyen 9y ago

Well, I think you disagree on the meaning of the word "artifact." GP cares that all production deployments are identical to the development environment that went through system tests. To ops, a reproducible build is one that has been uploaded to the registry, because it can be ported over to the production cluster and it will be bit-for-bit identical. It is not necessary to reproduce the same build bit-for-bit because you have kept a copy and distributed it. You are not rebuilding the same thing twice because that is an inefficient use of resources. Nobody really cares that the timestamps are different when the builds started at different times of day, it is evident therefore that you will not produce bit-for-bit identical builds as long as your filesystem includes these kinds of metadata and details. If you built it again, it is to make a change, so there is no point in insisting on bit-for-bit reproducibility in your build process (unless you are actually trying to optimize for immutable infrastructure, in which case more power to you, and it might be a particularly desirable trait to have. Not for nothing!)

nqzero 9y ago
that's a mirror, not a reproducible build
- yebyen 9y ago
  No, you're right, I read the "reproducible builds" page and it's a good formal definition for the term. I just think that if you ask ten average devops people on the street what "reproducible" means in the context of their devops jobs, eight of them at minimum are not going to know this definition, or insist on a bit-for-bit representation that arises directly from pure source code like what meets the definition at http://reproducible-builds.org http://reproducible-builds.org We're going to think you mean "has a BDR plan in place." Or maybe I've underestimated the reach of this concept.
FooBarWidget 9y ago
> Nobody really cares that the timestamps are different when the builds started at different times of day, it is evident therefore that you will not produce bit-for-bit identical builds as long as your filesystem includes these kinds of metadata and details. Do you think different timestamps is all that you need to worry about? How about the fact that some source tarballs that you depend on may have moved location? Or that your Dockerfile contains 'apt-get upgrade' but some upgraded package breaks your build whereas it did not before? Or that your Dockerfile curls some binary from the Internet but now that server uses an SSL cipher suite that is now incompatible with your OpenSSL version? All problems that I encountered.
- yebyen 9y ago
  Hey FooBarWidget! :D I was not speaking in terms of formal correctness in terms of the definition of reproducible builds, I was speaking of practical implementation for a production deployment. All of the issues you mentioned are (at least temporarily) resolvable by keeping a mirror of the builds so that you can reproduce the production environment without rebuilding if it goes away. I'm just defending the (apparently ops) person who was going by the original dictionary definition of "reproducible" which predates the "reproducible builds" definition of reproducible. If my manager asks me to make sure that my environment is reproducible, I assure you she is not talking about the formal definition that is being used here. I'm not saying that one shouldn't care about these things, but I am saying that many won't care about them. If you're doing a new build and the build results in upgrading to a newer version of a package, then that is a new build. It won't match. If you're doing a rebuild as an exercise to see if the output matches the previous build, then you're concerned about different things than the ops person will be. If my production instances are deleted by mistake, I'll be redeploying from the backup images, I won't be rebuilding my sources from scratch unless the backups also failed. I agree that it is a sort of problem that people usually don't build from scratch, it's just not one of the kinds of problems that my manager will ever be likely to ask me to solve.