7 ms·

One of my biggest pet peeves with NPM is that the packages aren't reproducible. Even if somebody links a package to their Git repo, who knows what commit correl

One of my biggest pet peeves with NPM is that the packages aren't reproducible. Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code. (especially if there is any type of build step.) I'm not 100% certain, but I believe this is an area where Golang is doing better (code references a Git repo directly). Are there any other languages that focus on "reproducible builds" for packages? (Ideally via Git) Edit: I just want to add that this is a major frustration for me with Java too. Everything on Maven barely has any metadata to go back and audit the source code!

vbezhenar 4y ago
What's wrong with maven? You have version in pom.xml, you can retrieve corresponding git tag and audit it.
- chriswarbo 4y ago
  Version numbers are just a form of documentation, giving the developer's best guess of whether the API broke or not. In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g. - Making changes without updating the version number (bad practice, but easily done) - Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit. - Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.). - Building with different versions of Maven, Java, etc. - The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache. Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both. Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything. FYI at my workplace we avoid some of these issues by: - Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc. - All builds are performed without network access. This prevents Maven resolving version numbers differently. - The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other. - Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
  - vbezhenar 4y ago
    Those issues are common for almost all package management system. Asking user to use hashes instead of versions is absurd. Those who need more control can always build packages on their environment.
    - chriswarbo 4y ago
      > Those issues are common for almost all package management system Yes, the situation's pretty bad. On the plus side, I'm increasingly of the opinion that "package management" is unnecessary; and usually just a symptom of having a poor build system.
- yakubin 4y ago
  Git tags can be changed, i.e. which commits they point to may change over time.
TheDong 4y ago
> Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code That's not what reproducible means in this context. Being able to go from a given release artifact to a git repo is not what the "reproducible builds" project/movement/whatever is about. Reproducible builds are more about the packaging of something than about specific language ecosystems, so asking about languages that focus on reproducible builds makes less sense than asking about package ecosystems, build systems, and linux distros that do. Let's look at an example: docker is written in go, and is marked as reproducible in debian, but unreproducible in archlinux: https://ismypackagereproducibleyet.org/?pkg=docker&query=query https://ismypackagereproducibleyet.org/?pkg=docker&query=que... (named docker.io on debian https://ismypackagereproducibleyet.org/?pkg=docker.io&query=query https://ismypackagereproducibleyet.org/?pkg=docker.io&query=...) Why is that? Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1ff0d78a665f1893e329a6ea892dd755f9dffc/debian/rules#L46 https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f... Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef9069afe742adc97f44e91654e80bcd829/trunk/PKGBUILD#L91-L94 https://github.com/archlinux/svntogit-community/blob/cf04bef... Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible. Reproducible builds are about capturing enough information about how something is built, and specific enough build instructions, that two people can produce the same bit-for-bit identical output, and therefore verify that someone didn't compile a malicious backdoored binary, and also increase the chance that each build consistently works or doesn't work, regardless of who compiles it when.
- pabs3 4y ago
  Your last paragraph sounds like something that the GP poster seems to want from NPM. The NPM packages are downstream pre-built packages created (hopefully) from upstream git tags, but currently nothing verifies that and few people seem to care.
- KronisLV 4y ago
pabs3 4y ago
Thats the same with Rust/Cargo, Python/PyPI, Ruby/Gems and every other language packaging ecosystem too. Even old autotools tarballs are an example of this, who knows what gets into the tarball.
- Liquid_Fire 4y ago
  Doesn't Cargo always build from source? It still doesn't mean it's reproducible, but at least you can see what you're getting.
  - pabs3 4y ago
    No, the Rust crate doesn't have to include the source. I've seen crates where the source is just files generated by a Python script and some other data, and the Python and data weren't in the crate, just in the upstream VCS.
    - newpavlov 4y ago
      But generated Rust code itself was included in the crate, correct? Otherwise, it wouldn't be possible to use such crate. If so, I don't see any problems for reproducibility.
      - pabs3 4y ago
        There is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
        Liquid_Fire 4y ago
        But you don't need to know that for the reproducibility of your product. That's a problem for the maintainer of that crate, but you know that you're getting the same Rust code every time.
        4y ago
newpavlov 4y ago
>Are there any other languages that focus on "reproducible builds" for packages? I think you misunderstand what "reproducible" means in the context of OP, but I will try to answer your question in the context of Rust. When you upload a crate to crates.io it gets uniquely identified by its cryptographic hash. It's not possible to replace code for an already uploaded crate version. Now, when you get dependencies for your build you may get newer (semver-compatible) version, but you always can retrieve the older version. To prevent such automatic updates, Rust applications often commit Cargo.lock to their repos, which contains hashes of all dependencies. So unless you manually run `cargo update` you will get exactly the same dependency tree, which with some minor build tweaks makes properly reproducible builds possible. Arguably, it's a much better system than relying on VCS as Go does.
- pabs3 4y ago
  I've seen Rust crates where the code was generated from a Python script and some other data and neither of those things are in the crate, which is potentially a license violation for copyleft crates. Or crates where the tests were in VCS but not in the crate. Or there were prebuilt files in the crate. I think always going with the VCS is a better option.
  - newpavlov 4y ago
    >code was generated from a Python script and some other data and neither of those things are in the crate Yes, it's (IMO unfortunately) possible to have non-reproducible code generation during compile time. The most prominent example is code generation based on DB schema. But such cases are quite rare and "going with the VCS" does not help here in any way. If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here. >Or crates where the tests were in VCS but not in the crate Tests are not used in building the final application, so they are irrelevant for reproducibility. >prebuilt files in the crate While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
    - 4y ago
jillesvangurp 4y ago
It's pretty doable to create reproducible builds with gradle or maven: https://reproducible-builds.org/docs/jvm/ https://reproducible-builds.org/docs/jvm/ Also, releases on maven central typically include a sources.jar along with the binaries. It's required to provide that as well as javadoc to publish on maven central, I think. They also require jars to be signed and there are some manual checks before they activate your account to verify e.g. domain ownership and metadata of your project. It's not perfect but it's better than what many other packagemanagers do. It's actually a bit of a PITA to setup; I've wasted quite a bit of time getting some stuff published there just trying to figure out their convoluted processes and tools and error messages. This stuff is way too hard in it's current form. Not all projects that use these build tools are fully reproducible but that is probably pretty easy to fix if people raise awareness of issues around this topic. There's even a website that tracks reproducibility for some common libraries on maven central: https://github.com/jvm-repo-rebuild/reproducible-central#readme https://github.com/jvm-repo-rebuild/reproducible-central#rea... If you want to use artififacts produced straight from git hashes, tags, or branches, jitpack.io is pretty neat.
Ayesh 4y ago
Composer (PHP) also does this. There is a package lock file that directly binds to a specific git commit, and all "composer install" builds are guaranteed to be identical for the same lock file (unless someone pushes something else under the same sha1, but that's Git's problem). Packages are downloaded directly from GitHub or whatever repo configured.
cxr 4y ago
> One of my biggest pet peeves with NPM is that the packages aren't reproducible[...] Are there any other languages that focus on "reproducible builds" for packages? NPM isn't a language. (Nor is NodeJS, for that matter.) It doesn't even occupy some semi-official position in its relationship to the actual language. Often, the NPM/NodeJS way even stands in stark contrast—working against the spirit and sometimes even the letter of the relevant standards and language features. (It is pretty remarkable, I admit, how the opinionated set of practices associated with NPM/NodeJS have managed to reach the place it has in the associated cultural dominance hierarchy, to the point that people often forget about this stuff.) > (Ideally via Git) Yeah, and it's super easy: you can make your builds reproducible by committing your source code to Git. All of it. (I.e., the libraries you're using, too.) People get irrationally upset when you suggest it, though; it really feels like it shouldn't be that easy, so they don't let it.
- Apes 4y ago
  > it's super easy I have a suspicion that you're not putting the source for the specific versions of glibc and Linux you used into every one of your projects.
  - cxr 4y ago
    This is not nearly as clever a response as it sounds. It's moving the goalposts. I don't produce anything where Linux's relationship to the application is as something that lives inside the build (unlike the types of third-party NPM dependencies we're talking about here) and I have a suspicion that there isn't any Linux inside the artifacts that your build step produces, either. If that's not the case for you, then yeah, I'll bite the bullet. You absolutely can either: (a) include that source code in your repo or (b) stop building things like that. If you find that going with the former option isn't easy, then that's because it never was—whether you're checking in all the source code or not.
  - ar_lan 4y ago
    A Nix package can do this easily!
  - 4y ago
voxic11 4y ago
Maybe something like SLSA could solve your issue https://slsa.dev/ https://slsa.dev/ the idea is to create a metadata standard so that packages can provide build providence information. So public build systems like github actions can sign a manifest explaining what build actions were executed and what the input and output of those actions are. That way you could verify a package was built from a specific set of sources with a specific set of build actions without requiring reproducibility. This assumes you trust the public build system not to lie though.

vbezhenar 4y ago

What's wrong with maven? You have version in pom.xml, you can retrieve corresponding git tag and audit it.

chriswarbo 4y ago
Version numbers are just a form of documentation, giving the developer's best guess of whether the API broke or not. In particular, Maven's use of groupId:artifactId:version is inadequate as an identifier, since many artifacts can have this same "identifier", e.g. - Making changes without updating the version number (bad practice, but easily done) - Building with different versions of dependencies. Maven allows version ranges, and resolves them based on the contents of third-party servers, so this can change over time even if we're building from the same git commit. - Building with different property settings, e.g. due to a different profile getting 'activated' due to some ambient aspect of the system (OS, presence/absence of some file path, env vars, etc.). - Building with different versions of Maven, Java, etc. - The contents of Maven's cache also affects builds. If a file path exists in the cache, Maven will use it regardless of its contents. Also, undeclared dependencies can sometimes be picked up from the cache, which would otherwise fail if they weren't in the cache. Maven also uses that scheme as a location for fetching artifacts, which is not only inadequate but also insecure (we can put arbitrary code at a certain path and Maven will happily fetch it). Maven does perform SHA512 verification, but those hashes are fetched from the same location as the artifacts, so subverting one allows subverting both. Also, in case you weren't aware, git tags are just a convenience. Commit IDs are SHA1 hashes, so they're unforgeable and tamper-proof; whereas git tags can be deleted and recreated to point at anything. FYI at my workplace we avoid some of these issues by: - Performing all builds using Nix, which uses a sandbox with versions of Maven, JVM, etc. specified with their SHA256 hashes. This avoids differences due to tooling, profiles, env vars, caches, etc. - All builds are performed without network access. This prevents Maven resolving version numbers differently. - The SHA256 of the build environment (including all tooling, env vars, properties, source code, etc.) gets appended to the version number, which distinguishes these different artifacts from each other. - Our projects specify the expected hash of their Maven dependencies (essentially like a lock file), which avoids having to trust third-parties for the hashes (a trust-on-first-use policy).
- vbezhenar 4y ago
  Those issues are common for almost all package management system. Asking user to use hashes instead of versions is absurd. Those who need more control can always build packages on their environment.
  - chriswarbo 4y ago
    > Those issues are common for almost all package management system Yes, the situation's pretty bad. On the plus side, I'm increasingly of the opinion that "package management" is unnecessary; and usually just a symptom of having a poor build system.
yakubin 4y ago
Git tags can be changed, i.e. which commits they point to may change over time.

TheDong 4y ago

> Even if somebody links a package to their Git repo, who knows what commit correlates to a given release? Even if somebody pushes a "tag" for the release, there is still no guarantee that it's actually the same code That's not what reproducible means in this context. Being able to go from a given release artifact to a git repo is not what the "reproducible builds" project/movement/whatever is about. Reproducible builds are more about the packaging of something than about specific language ecosystems, so asking about languages that focus on reproducible builds makes less sense than asking about package ecosystems, build systems, and linux distros that do. Let's look at an example: docker is written in go, and is marked as reproducible in debian, but unreproducible in archlinux: https://ismypackagereproducibleyet.org/?pkg=docker&query=query https://ismypackagereproducibleyet.org/?pkg=docker&query=que... (named docker.io on debian https://ismypackagereproducibleyet.org/?pkg=docker.io&query=query https://ismypackagereproducibleyet.org/?pkg=docker.io&query=...) Why is that? Well, debian sets the "version.BuildTime" option when it compiles it to a constant value each time it compiles it: https://salsa.debian.org/go-team/packages/docker/-/blob/4f1ff0d78a665f1893e329a6ea892dd755f9dffc/debian/rules#L46 https://salsa.debian.org/go-team/packages/docker/-/blob/4f1f... Archlinux's PKGBUILD doesn't https://github.com/archlinux/svntogit-community/blob/cf04bef9069afe742adc97f44e91654e80bcd829/trunk/PKGBUILD#L91-L94 https://github.com/archlinux/svntogit-community/blob/cf04bef... Because of that, "docker version" on arch linux outputs that it was built on the actual day it was built, whenever that was, and each time it's built that string changes. Hence, not reproducible. Reproducible builds are about capturing enough information about how something is built, and specific enough build instructions, that two people can produce the same bit-for-bit identical output, and therefore verify that someone didn't compile a malicious backdoored binary, and also increase the chance that each build consistently works or doesn't work, regardless of who compiles it when.

pabs3 4y ago
Your last paragraph sounds like something that the GP poster seems to want from NPM. The NPM packages are downstream pre-built packages created (hopefully) from upstream git tags, but currently nothing verifies that and few people seem to care.
KronisLV 4y ago

pabs3 4y ago

Thats the same with Rust/Cargo, Python/PyPI, Ruby/Gems and every other language packaging ecosystem too. Even old autotools tarballs are an example of this, who knows what gets into the tarball.

Liquid_Fire 4y ago
Doesn't Cargo always build from source? It still doesn't mean it's reproducible, but at least you can see what you're getting.
- pabs3 4y ago
  No, the Rust crate doesn't have to include the source. I've seen crates where the source is just files generated by a Python script and some other data, and the Python and data weren't in the crate, just in the upstream VCS.
  - newpavlov 4y ago
    But generated Rust code itself was included in the crate, correct? Otherwise, it wouldn't be possible to use such crate. If so, I don't see any problems for reproducibility.
    - pabs3 4y ago
      There is no source in the crate, just generated Rust code, so you can't know if the build process to create that Rust code is reproducible or not.
      - Liquid_Fire 4y ago
        But you don't need to know that for the reproducibility of your product. That's a problem for the maintainer of that crate, but you know that you're getting the same Rust code every time.
        4y ago

newpavlov 4y ago

>Are there any other languages that focus on "reproducible builds" for packages? I think you misunderstand what "reproducible" means in the context of OP, but I will try to answer your question in the context of Rust. When you upload a crate to crates.io it gets uniquely identified by its cryptographic hash. It's not possible to replace code for an already uploaded crate version. Now, when you get dependencies for your build you may get newer (semver-compatible) version, but you always can retrieve the older version. To prevent such automatic updates, Rust applications often commit Cargo.lock to their repos, which contains hashes of all dependencies. So unless you manually run `cargo update` you will get exactly the same dependency tree, which with some minor build tweaks makes properly reproducible builds possible. Arguably, it's a much better system than relying on VCS as Go does.

pabs3 4y ago
I've seen Rust crates where the code was generated from a Python script and some other data and neither of those things are in the crate, which is potentially a license violation for copyleft crates. Or crates where the tests were in VCS but not in the crate. Or there were prebuilt files in the crate. I think always going with the VCS is a better option.
- newpavlov 4y ago
  >code was generated from a Python script and some other data and neither of those things are in the crate Yes, it's (IMO unfortunately) possible to have non-reproducible code generation during compile time. The most prominent example is code generation based on DB schema. But such cases are quite rare and "going with the VCS" does not help here in any way. If you mean that Rust code gets generated by a Python script and then gets uploaded to crates.io without the "origin" scrips, then I don't see any problems here. >Or crates where the tests were in VCS but not in the crate Tests are not used in building the final application, so they are irrelevant for reproducibility. >prebuilt files in the crate While I think it's a bad practice (though I haven't seen such crates myself), it does not hinder build reproducibility.
  - 4y ago

jillesvangurp 4y ago

It's pretty doable to create reproducible builds with gradle or maven: https://reproducible-builds.org/docs/jvm/ https://reproducible-builds.org/docs/jvm/ Also, releases on maven central typically include a sources.jar along with the binaries. It's required to provide that as well as javadoc to publish on maven central, I think. They also require jars to be signed and there are some manual checks before they activate your account to verify e.g. domain ownership and metadata of your project. It's not perfect but it's better than what many other packagemanagers do. It's actually a bit of a PITA to setup; I've wasted quite a bit of time getting some stuff published there just trying to figure out their convoluted processes and tools and error messages. This stuff is way too hard in it's current form. Not all projects that use these build tools are fully reproducible but that is probably pretty easy to fix if people raise awareness of issues around this topic. There's even a website that tracks reproducibility for some common libraries on maven central: https://github.com/jvm-repo-rebuild/reproducible-central#readme https://github.com/jvm-repo-rebuild/reproducible-central#rea... If you want to use artififacts produced straight from git hashes, tags, or branches, jitpack.io is pretty neat.

Ayesh 4y ago

Composer (PHP) also does this. There is a package lock file that directly binds to a specific git commit, and all "composer install" builds are guaranteed to be identical for the same lock file (unless someone pushes something else under the same sha1, but that's Git's problem). Packages are downloaded directly from GitHub or whatever repo configured.

cxr 4y ago

> One of my biggest pet peeves with NPM is that the packages aren't reproducible[...] Are there any other languages that focus on "reproducible builds" for packages? NPM isn't a language. (Nor is NodeJS, for that matter.) It doesn't even occupy some semi-official position in its relationship to the actual language. Often, the NPM/NodeJS way even stands in stark contrast—working against the spirit and sometimes even the letter of the relevant standards and language features. (It is pretty remarkable, I admit, how the opinionated set of practices associated with NPM/NodeJS have managed to reach the place it has in the associated cultural dominance hierarchy, to the point that people often forget about this stuff.) > (Ideally via Git) Yeah, and it's super easy: you can make your builds reproducible by committing your source code to Git. All of it. (I.e., the libraries you're using, too.) People get irrationally upset when you suggest it, though; it really feels like it shouldn't be that easy, so they don't let it.

Apes 4y ago
> it's super easy I have a suspicion that you're not putting the source for the specific versions of glibc and Linux you used into every one of your projects.
- cxr 4y ago
  This is not nearly as clever a response as it sounds. It's moving the goalposts. I don't produce anything where Linux's relationship to the application is as something that lives inside the build (unlike the types of third-party NPM dependencies we're talking about here) and I have a suspicion that there isn't any Linux inside the artifacts that your build step produces, either. If that's not the case for you, then yeah, I'll bite the bullet. You absolutely can either: (a) include that source code in your repo or (b) stop building things like that. If you find that going with the former option isn't easy, then that's because it never was—whether you're checking in all the source code or not.
- ar_lan 4y ago
  A Nix package can do this easily!
- 4y ago

voxic11 4y ago

Maybe something like SLSA could solve your issue https://slsa.dev/ https://slsa.dev/ the idea is to create a metadata standard so that packages can provide build providence information. So public build systems like github actions can sign a manifest explaining what build actions were executed and what the input and output of those actions are. That way you could verify a package was built from a specific set of sources with a specific set of build actions without requiring reproducibility. This assumes you trust the public build system not to lie though.