5 ms·

Actually, no, please don't. Build system should not download shit from the internets. Build system should only build.

by dozzie 10y ago

datalist 10y ago
That cannot be emphasised enough. Unfortunately it falls on deaf ears. If a compile does not download at least 56 megabytes these days you can consider it a failure ;).
berdario 10y ago
I've already read this refrain: it seems to make sense, but I cannot really justify why "build systems should only build" should be the case What's the advantage over a build system that downloads the dependencies, but which gives you the ability to prefetch all the dependencies (so you can reliably do work without connectivity after prefetching)?
- datalist 10y ago
  Separation of concerns On one hand frameworks and code in general becomes more and more bloated and incomprehensible these days, due to an overzealous approach in the aforementioned. On the other hand tools are getting cramped with features which were never in their original scope. The leftpad issue was only one example of that lets-inject-third-party-code-on-the-fly mentality.
- dozzie 10y ago
  To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources. And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.
  - berdario 10y ago
    > To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
    - datalist 10y ago
      > if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
- easytiger 10y ago
  Try living off the grid (ala any bank) and see how easy it is for online dependant build systems. Java maven stuff is bad enough already
  - Freak_NL 10y ago
    Maven facilitates downloading all dependencies in one go, exactly for this scenario: mvn dependency:copy-dependencies And when you build a project once, all downloaded dependencies are cached locally indefinitely. That is, you can furnish a Maven build with offline dependencies. Normally you do want to simply have your build download (and cache locally) a new version of a dependency, but an internet connection is not required if you have the dependencies available locally. Maven will always look in your local repository first, and will not try to download anything, except if you are depending on a SNAPSHOT version of something (which implies that the build should always check for a newer version).
    - forgottenpass 10y ago
      exactly for this scenario Not really. Building from local sources only or "living off the grid" as parent poster put it typically isn't just about temporary loss of Internet connectivity, but instead serves as a shorthand for a more complicated set of requirements. Generally speaking there will be (at least) disaster recovery requirements such that a computer with a fresh OS install and access to the local very-well-backed-up archives can produce a build. And the local archives typically have exactly one entry point so that legal and process controls can be applied as they go in. So the problem isn't how to cache dependencies for a working directory, but how to get, archive them, and distribute them to a development team easily. A lot of package mangers approach supporting this kind of use case, but still need a fair bit of customization and manual steps if not new tooling built around them. And all that infrastructure also has to meet qualification criteria (through testing). It gets to be enough that any tool that decides to add in their own half-assed download manager will be quite annoying to anyone that doesn't use "YOLO" as their quality management system.
      - berdario 10y ago
phyzome 10y ago
How do you picture those files getting onto your computer?
- datalist 10y ago
  The same way they always got there.

Actually, no, please don't. Build system should not download shit from the internets. Build system should only build.

by dozzie 10y ago

Actually, no, please don't. Build system should not download shit from the internets. Build system should only build.

That cannot be emphasised enough. Unfortunately it falls on deaf ears. If a compile does not download at least 56 megabytes these days you can consider it a failure ;).

berdario 10y ago

I've already read this refrain: it seems to make sense, but I cannot really justify why "build systems should only build" should be the case What's the advantage over a build system that downloads the dependencies, but which gives you the ability to prefetch all the dependencies (so you can reliably do work without connectivity after prefetching)?

datalist 10y ago
Separation of concerns On one hand frameworks and code in general becomes more and more bloated and incomprehensible these days, due to an overzealous approach in the aforementioned. On the other hand tools are getting cramped with features which were never in their original scope. The leftpad issue was only one example of that lets-inject-third-party-code-on-the-fly mentality.
dozzie 10y ago
To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources. And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.
- berdario 10y ago
  > To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
  - datalist 10y ago
    > if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
easytiger 10y ago
Try living off the grid (ala any bank) and see how easy it is for online dependant build systems. Java maven stuff is bad enough already
- Freak_NL 10y ago
  Maven facilitates downloading all dependencies in one go, exactly for this scenario: mvn dependency:copy-dependencies And when you build a project once, all downloaded dependencies are cached locally indefinitely. That is, you can furnish a Maven build with offline dependencies. Normally you do want to simply have your build download (and cache locally) a new version of a dependency, but an internet connection is not required if you have the dependencies available locally. Maven will always look in your local repository first, and will not try to download anything, except if you are depending on a SNAPSHOT version of something (which implies that the build should always check for a newer version).
  - forgottenpass 10y ago
    exactly for this scenario Not really. Building from local sources only or "living off the grid" as parent poster put it typically isn't just about temporary loss of Internet connectivity, but instead serves as a shorthand for a more complicated set of requirements. Generally speaking there will be (at least) disaster recovery requirements such that a computer with a fresh OS install and access to the local very-well-backed-up archives can produce a build. And the local archives typically have exactly one entry point so that legal and process controls can be applied as they go in. So the problem isn't how to cache dependencies for a working directory, but how to get, archive them, and distribute them to a development team easily. A lot of package mangers approach supporting this kind of use case, but still need a fair bit of customization and manual steps if not new tooling built around them. And all that infrastructure also has to meet qualification criteria (through testing). It gets to be enough that any tool that decides to add in their own half-assed download manager will be quite annoying to anyone that doesn't use "YOLO" as their quality management system.
    - berdario 10y ago

phyzome 10y ago

How do you picture those files getting onto your computer?

datalist 10y ago
The same way they always got there.