4 ms·

To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows

by dozzie 10y ago

To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. This separation allows to build the project off-line, which is not an uncommon scenario. There are environments that have the direct internet access prohibited e.g. by a company policy. There are environments that have it difficult (e.g. are behind a proxy). There are distribution package builders (RPM, DEB), which have a policy of only working on local sources. And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process.

berdario 10y ago
> To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)
- datalist 10y ago
  > if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
  - berdario 10y ago
    > I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.) I think we're misunderstanding each other. What I meant is that if you have foo-1.2 in your dependencies.conf, and you have previously downloaded foo-1.1 it doesn't make sense to compile, because the result of the compilation will most likely not be what you want
    - aninhumer 10y ago
      Sure, but that doesn't mean you need to combine downloading and building. You can just have the build say "foo-1.2 not found. Latest version is foo-1.1", and the developer can decide how to respond.
      - berdario 10y ago
        Well, there are 3 cases: - "usual" case (if you agree): dependencies will be downloaded from the Internet, single command download+build is useful - in a bank/SC environment: the developer configured a proxy/local mirror of the package repositories, and the dependencies will be downloaded automatically, single command download+build is useful - in a bank/SC environment: manual process to obtain and add locally the dependencies, the build/dev system has limited networking: automatically download fails, single command download+build is not useful but not harmful either Having a "prefetch" and "offline-build" commands are perfectly fine, but I don't see the reason why the default shouldn't be a "build" command that does prefetch+offline-build
chriswarbo 10y ago
> Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Another part was using location-addressable storage for dependencies, rather than content-addressable. If dependencies used something like magnet links, IPFS, etc. then nobody would really care if some particular machine stopped serving some particular file.
- forgottenpass 10y ago
  nobody would really care if some particular machine stopped serving some particular file. Yes they would. Because such systems can only serve files that exist on at least one node, people who want to rely on such a system to serve a given file must run their own node to host every file they care about the availability of. The topological properties of distribution are different, but the existence of the local cache to provide dependability is not avoided.

by dozzie 10y ago

berdario 10y ago

> To clarify: I'm not against managing (downloading) dependencies. I just want two separate steps: [download things] and [compile things]. Fair enough, but if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them, and so I think it's reasonable to make the 2nd step depend on the 1st one (and thus make it execute automatically) I understand that keeping the 2 of them too-close might inadvertently conflate the 2 concerns (the build step becomes impossible to run without the download step) even if the design goals explictly thought of them as being able to be run independently, so this might be an argument for "builds should only build", but I'm not sold on it yet > And then there is build reproducibility. If external network is involved, the whole reproducibility idea goes out of the window. Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Well, the build systems that I have in mind are Stack ( http://haskellstack.org/ http://haskellstack.org/ ) and Nix ( http://nixos.org/nix/ http://nixos.org/nix/ ) Both of them have an heavy emphasis on reproducibility, and yet both of them automatically download dependencies. (the Hydra Nix continuous integration systems OTOH run build and test steps with limited/no network connectivity, to enforce that separation of concerns) (the microdependencies btw wasn't a problem that caused build non-reproducibility, microdependencies only made it worse when the shit hit the fan) Also, there are 2 different reproducible build failures: it doesn't build (bad) or it builds but the artifact is not identical (worse)... downloading dependencies from the internet can only cause problems of the first type, if you have a sane way of fetching them sane means: either you have the hashes of the dependencies to check, or you can trust the archive to always give you the same files when you ask it for a snapshot... and for security reasons you'd still check the crypto signature on the hashes to verify that nothing has been tampered in the mirror from which you downloaded (Unfortunately, not only Npm but also Hackage afaik have that anti-feature that allows you to reupload a "bugfix" with the same version number... so we cannot always trust that the systems that we're using are sane, which OTOH is impossible also due to DMCA or other law/court shenanigans: if your dependency violates someone's copyright, it might end up disappearing from the snapshots)

datalist 10y ago
> if you have outdated dependency artifacts, it doesn't make sense to compile without fetching them I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.). This entire forced update approach is already unpleasant enough in the context of applications (most notably on Android and IOS) but becomes unbearable for software development. While one might argue it could still make some sense for the former, as average users might stay forever with old versions and potential security issues, that argument should not be valid in a "professional" context - where the "users" know about the implications - as it should be given with software engineering.
- berdario 10y ago
  > I am sorry, I'd contest that. There are often cases where one might not want to use the most current version of some library (incompatibility, undesired features, etc.) I think we're misunderstanding each other. What I meant is that if you have foo-1.2 in your dependencies.conf, and you have previously downloaded foo-1.1 it doesn't make sense to compile, because the result of the compilation will most likely not be what you want
  - aninhumer 10y ago
    Sure, but that doesn't mean you need to combine downloading and building. You can just have the build say "foo-1.2 not found. Latest version is foo-1.1", and the developer can decide how to respond.
    - berdario 10y ago
      Well, there are 3 cases: - "usual" case (if you agree): dependencies will be downloaded from the Internet, single command download+build is useful - in a bank/SC environment: the developer configured a proxy/local mirror of the package repositories, and the dependencies will be downloaded automatically, single command download+build is useful - in a bank/SC environment: manual process to obtain and add locally the dependencies, the build/dev system has limited networking: automatically download fails, single command download+build is not useful but not harmful either Having a "prefetch" and "offline-build" commands are perfectly fine, but I don't see the reason why the default shouldn't be a "build" command that does prefetch+offline-build

chriswarbo 10y ago

> Remember the left-pad farce? Part of the cause was idiotic split to microdependencies, but part was that everybody used external network for their build process. Another part was using location-addressable storage for dependencies, rather than content-addressable. If dependencies used something like magnet links, IPFS, etc. then nobody would really care if some particular machine stopped serving some particular file.

forgottenpass 10y ago
nobody would really care if some particular machine stopped serving some particular file. Yes they would. Because such systems can only serve files that exist on at least one node, people who want to rely on such a system to serve a given file must run their own node to host every file they care about the availability of. The topological properties of distribution are different, but the existence of the local cache to provide dependability is not avoided.