7 ms·

I've already read this refrain: it seems to make sense, but I cannot really justify why "build systems should only build" should be the case What's the advanta

I've already read this refrain: it seems to make sense, but I cannot really justify why "build systems should only build" should be the case What's the advantage over a build system that downloads the dependencies, but which gives you the ability to prefetch all the dependencies (so you can reliably do work without connectivity after prefetching)?

datalist 10y ago
Separation of concerns On one hand frameworks and code in general becomes more and more bloated and incomprehensible these days, due to an overzealous approach in the aforementioned. On the other hand tools are getting cramped with features which were never in their original scope. The leftpad issue was only one example of that lets-inject-third-party-code-on-the-fly mentality.
dozzie 10y ago
To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources. And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.
- berdario 10y ago
  > To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
  - datalist 10y ago
    > if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
    - berdario 10y ago
      > I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.) I think we're misunderstanding each other. What I meant is that if you have foo-1.2 in your dependencies.conf, and you have previously downloaded foo-1.1 it doesn't make sense to compile, because the result of the compilation will most likely not be what you want
      - aninhumer 10y ago
        Sure, but that doesn't mean you need to combine downloading and building. You can just have the build say "foo-1.2 not found. Latest version is foo-1.1", and the developer can decide how to respond.
        berdario 10y ago
        Well, there are 3 cases: - "usual" case (if you agree): dependencies will be downloaded from the Internet, single command download+build is useful - in a bank/SC environment: the developer configured a proxy/local mirror of the package repositories, and the dependencies will be downloaded automatically, single command download+build is useful - in a bank/SC environment: manual process to obtain and add locally the dependencies, the build/dev system has limited networking: automatically download fails, single command download+build is not useful but not harmful either Having a "prefetch" and "offline-build" commands are perfectly fine, but I don't see the reason why the default shouldn't be a "build" command that does prefetch+offline-build
- chriswarbo 10y ago
  > Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Another part was using location-addressable storage for dependencies, rather than content-addressable. If dependencies used something like magnet links, IPFS, etc. then nobody would really care if some particular machine stopped serving some particular file.
  - forgottenpass 10y ago
    nobody would really care if some particular machine stopped serving some particular file. Yes they would. Because such systems can only serve files that exist on at least one node, people who want to rely on such a system to serve a given file must run their own node to host every file they care about the availability of. The topological properties of distribution are different, but the existence of the local cache to provide dependability is not avoided.
easytiger 10y ago
Try living off the grid (ala any bank) and see how easy it is for online dependant build systems. Java maven stuff is bad enough already
- Freak_NL 10y ago
  Maven facilitates downloading all dependencies in one go, exactly for this scenario: mvn dependency:copy-dependencies And when you build a project once, all downloaded dependencies are cached locally indefinitely. That is, you can furnish a Maven build with offline dependencies. Normally you do want to simply have your build download (and cache locally) a new version of a dependency, but an internet connection is not required if you have the dependencies available locally. Maven will always look in your local repository first, and will not try to download anything, except if you are depending on a SNAPSHOT version of something (which implies that the build should always check for a newer version).
  - forgottenpass 10y ago
    exactly for this scenario Not really. Building from local sources only or "living off the grid" as parent poster put it typically isn't just about temporary loss of Internet connectivity, but instead serves as a shorthand for a more complicated set of requirements. Generally speaking there will be (at least) disaster recovery requirements such that a computer with a fresh OS install and access to the local very-well-backed-up archives can produce a build. And the local archives typically have exactly one entry point so that legal and process controls can be applied as they go in. So the problem isn't how to cache dependencies for a working directory, but how to get, archive them, and distribute them to a development team easily. A lot of package mangers approach supporting this kind of use case, but still need a fair bit of customization and manual steps if not new tooling built around them. And all that infrastructure also has to meet qualification criteria (through testing). It gets to be enough that any tool that decides to add in their own half-assed download manager will be quite annoying to anyone that doesn't use "YOLO" as their quality management system.
    - berdario 10y ago

datalist 10y ago

Separation of concerns On one hand frameworks and code in general becomes more and more bloated and incomprehensible these days, due to an overzealous approach in the aforementioned. On the other hand tools are getting cramped with features which were never in their original scope. The leftpad issue was only one example of that lets-inject-third-party-code-on-the-fly mentality.

dozzie 10y ago

To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources. And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.

berdario 10y ago
> To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
- datalist 10y ago
  > if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
  - berdario 10y ago
    > I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.) I think we're misunderstanding each other. What I meant is that if you have foo-1.2 in your dependencies.conf, and you have previously downloaded foo-1.1 it doesn't make sense to compile, because the result of the compilation will most likely not be what you want
    - aninhumer 10y ago
      Sure, but that doesn't mean you need to combine downloading and building. You can just have the build say "foo-1.2 not found. Latest version is foo-1.1", and the developer can decide how to respond.
      - berdario 10y ago
        Well, there are 3 cases: - "usual" case (if you agree): dependencies will be downloaded from the Internet, single command download+build is useful - in a bank/SC environment: the developer configured a proxy/local mirror of the package repositories, and the dependencies will be downloaded automatically, single command download+build is useful - in a bank/SC environment: manual process to obtain and add locally the dependencies, the build/dev system has limited networking: automatically download fails, single command download+build is not useful but not harmful either Having a "prefetch" and "offline-build" commands are perfectly fine, but I don't see the reason why the default shouldn't be a "build" command that does prefetch+offline-build
chriswarbo 10y ago
> Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Another part was using location-addressable storage for dependencies, rather than content-addressable. If dependencies used something like magnet links, IPFS, etc. then nobody would really care if some particular machine stopped serving some particular file.
- forgottenpass 10y ago
  nobody would really care if some particular machine stopped serving some particular file. Yes they would. Because such systems can only serve files that exist on at least one node, people who want to rely on such a system to serve a given file must run their own node to host every file they care about the availability of. The topological properties of distribution are different, but the existence of the local cache to provide dependability is not avoided.

easytiger 10y ago

Try living off the grid (ala any bank) and see how easy it is for online dependant build systems. Java maven stuff is bad enough already

Freak_NL 10y ago
Maven facilitates downloading all dependencies in one go, exactly for this scenario: mvn dependency:copy-dependencies And when you build a project once, all downloaded dependencies are cached locally indefinitely. That is, you can furnish a Maven build with offline dependencies. Normally you do want to simply have your build download (and cache locally) a new version of a dependency, but an internet connection is not required if you have the dependencies available locally. Maven will always look in your local repository first, and will not try to download anything, except if you are depending on a SNAPSHOT version of something (which implies that the build should always check for a newer version).
- forgottenpass 10y ago
  exactly for this scenario Not really. Building from local sources only or "living off the grid" as parent poster put it typically isn't just about temporary loss of Internet connectivity, but instead serves as a shorthand for a more complicated set of requirements. Generally speaking there will be (at least) disaster recovery requirements such that a computer with a fresh OS install and access to the local very-well-backed-up archives can produce a build. And the local archives typically have exactly one entry point so that legal and process controls can be applied as they go in. So the problem isn't how to cache dependencies for a working directory, but how to get, archive them, and distribute them to a development team easily. A lot of package mangers approach supporting this kind of use case, but still need a fair bit of customization and manual steps if not new tooling built around them. And all that infrastructure also has to meet qualification criteria (through testing). It gets to be enough that any tool that decides to add in their own half-assed download manager will be quite annoying to anyone that doesn't use "YOLO" as their quality management system.
  - berdario 10y ago