9 ms·

Why people think it's a good idea for the build systems to reach out and grab code from the internet is beyond me.

by gspr 5y ago

hoseja 5y ago
It's not a build system, it's a meta build system.
- mathstuf 5y ago
  [ Full disclosure: CMake developer. ] Eh. CMake is a build system. It supports multiple build tools, but it's not like one can just tell CMake "here's some literal Make code, please put it into the generated files". The generated builds are still restricted to the semantics that CMake itself provides via its interface (e.g., no dyndep support in `add_custom_command` because only the Ninja generator supports that reasonably well; it's possible with Make, but I don't know how easy it would be to make it work for arbitrary custom commands).
  - iamcreasy 5y ago
    [Not a C/C++ dev] I am under the assumption CMake do not perform build itself, it produces the build file for Make, Ninja etc. and have those tools run the build file to produce the final binary. If that's correct, wouldn't it make CMake a meta build system?
    - mathstuf 5y ago
      Right, I'm being more semantic here. CMake is a build system with the concepts of targets, linking, compiling, languages, toolchains, etc. The fact that it gets "rendered" into a `build.ninja`, `.vcxproj`, `.xcodebuild`, or `Makefile` is (largely) ambivalent (I call these "build tools" or "build executors"). Other instances would be `gn` or Meson. They're the build system; ninja is "just" the build tool they use to actually implement their semantics. Still others like build2, Boost.build, or the Bazel family are both: the build system and the build tool. One cannot use CMake to write arbitrary Makefiles using all the fancy features available through macro expansion, rule templates, etc. All you have are CMake's semantics that can then be executed by a tool that implements POSIX makefile execution behaviors. A (probably poor) analogy is a network resource. It can be provided over HTTP, Gopher, or whatever. The semantics of the resource are the same, but the transport mechanism is just how it interacts with the rest of the world.
jasonpeacock 5y ago
How else would you add/manage external dependencies for your project? Do you manually install them on your local host? That's not reproducible and doesn't support multiple versions for different projects. Do you use an uber build system like Yocto to wrap your CMake build, and let it pull the packages? Do you manually copy all the external deps into your own project repo, and then manually keep them updated with the latest changes?
- krapht 5y ago
  I think parent is saying to use a package manager.
  - dagmx 5y ago
    That would be the same issue though, just moved to a different spot
    - nerdponx 5y ago
      Not necessarily. Package managers can cache things, especially if you're fetching the same version over and over. Also, separation of concerns is a good thing IMO.
      - howinteresting 5y ago
        Cargo etc also cache dependencies in a central location. "Separation of concerns" is ill-defined — you can use cargo as just a package manager, just a build system, or both. Most people use it as both but there are definitely places (e.g. bigcos) that use just one of the aspects of cargo.
    - gspr 5y ago
      Separation of tasks. The build system builds. Another system (like a package manager, if you are so inclined, or not if not) manages packages.
      - 5y ago
krapht 5y ago
People also curl and execute shell scripts from the internet every day.
- gspr 5y ago
  Yes. And it's a terrible idea.
  - handrous 5y ago
    It's no worse than several other not-very-secure-but-nonetheless-common ways to install software. Would a MS-provided copy-paste-curl-sh-to-install, say, be any worse than a link to an MSI? Either way I'm trusting that MS isn't sending me ransomware or a rootkit or whatever. If that MSI just executes curl under the hood, exactly like the sh command would have, is it better (nb this is what thin web installers for big programs do, basically)? If I add a PPA for some company and apt-install something from it, unless I audit all the files before proceeding, I'm trusting that they won't format my disk without my wanting them to. How terrible an idea it is depends on who's providing it, same as most other software.
    - mappu 5y ago
      I do agree - There are some specific pitfalls with curl|bash: (A) the web server may serve different code for curl's user-agent than what you audited by clicking the link in a browser, leading to a false sense of security; (B) if the network transfer is interrupted, bash will partially execute an incomplete script; and (C) the recurring prevalence of high-profile bugs like https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671 You can mitigate (B) by wrapping the whole script in a function, and you can mitigate (C) with `set -euo pipefail` and `trap`, but that doesn't seem to be the kind of bash scripts that people write in practice. MSI (and deb/rpm) definitely can arbitrary bad things at install time too, but at least their built-in file extraction features are entirely declarative, and they always have an entrypoint for the system uninstaller.
mr_tristan 5y ago
It's the pragmatic solution to a hard problem. Supporting distribution for all the different kinds of Linux package mangers alone isn't easy, and you take in how macOS and Windows do things, I'm not even sure where you start. It's kind of sad, but there's multiple walled gardens in play here. I mean, Java's been around for 25 years, and it's just starting to get tooling for making OS installers for applications (jpackage). So, programming toolchains just invent package management that is easy for everyone to use. Everyone needs a problem solved today for multiple platforms, and it's not like OSes are working together to make this easy. Personally, I've come to prefer vendored dependencies with the sources, and building scripts around vendoring, separating the build from dependency distribution. But not a lot of programmers are comfortable with this. It's not a common practice, and the most popular version control system (git) isn't very great with binary management.
- gspr 5y ago
  The point isn't the whole debate about vendoring. I have strong opinions on that matter, and they are different from yours, but independently of whether one wants vendoring or not: It seems to me to be highly advantageous to decouple building from dependency management (nomatter if said dependencies are vendored or not).
intc 5y ago
This is a worrisome trend. Now one needs to manually check what the build system will fetch (and from where) before starting the build. Many times most of the dependencies are available through the package manager of ones system. A good way in my opinion is just to provide a clear list of the dependencies and make the build system complain if it's not there so one can install it (if didn't bother manually to check that the listed dependencies are present). Of course there are very large and complex projects which may need more sophisticated solutions - Perhaps some sort code notarisation would serve well? Yet even there the dependency management should be clearly separated from the build system and it should prioritize using system provided packages when possible. There are probably other considerations and implications too. Like preference of static linking over dynamic etc..

Why people think it's a good idea for the build systems to reach out and grab code from the internet is beyond me.

by gspr 5y ago

Why people think it's a good idea for the build systems to reach out and grab code from the internet is beyond me.

hoseja 5y ago

It's not a build system, it's a meta build system.

mathstuf 5y ago
[ Full disclosure: CMake developer. ] Eh. CMake is a build system. It supports multiple build tools, but it's not like one can just tell CMake "here's some literal Make code, please put it into the generated files". The generated builds are still restricted to the semantics that CMake itself provides via its interface (e.g., no dyndep support in `add_custom_command` because only the Ninja generator supports that reasonably well; it's possible with Make, but I don't know how easy it would be to make it work for arbitrary custom commands).
- iamcreasy 5y ago
  [Not a C/C++ dev] I am under the assumption CMake do not perform build itself, it produces the build file for Make, Ninja etc. and have those tools run the build file to produce the final binary. If that's correct, wouldn't it make CMake a meta build system?
  - mathstuf 5y ago
    Right, I'm being more semantic here. CMake is a build system with the concepts of targets, linking, compiling, languages, toolchains, etc. The fact that it gets "rendered" into a `build.ninja`, `.vcxproj`, `.xcodebuild`, or `Makefile` is (largely) ambivalent (I call these "build tools" or "build executors"). Other instances would be `gn` or Meson. They're the build system; ninja is "just" the build tool they use to actually implement their semantics. Still others like build2, Boost.build, or the Bazel family are both: the build system and the build tool. One cannot use CMake to write arbitrary Makefiles using all the fancy features available through macro expansion, rule templates, etc. All you have are CMake's semantics that can then be executed by a tool that implements POSIX makefile execution behaviors. A (probably poor) analogy is a network resource. It can be provided over HTTP, Gopher, or whatever. The semantics of the resource are the same, but the transport mechanism is just how it interacts with the rest of the world.

jasonpeacock 5y ago

How else would you add/manage external dependencies for your project? Do you manually install them on your local host? That's not reproducible and doesn't support multiple versions for different projects. Do you use an uber build system like Yocto to wrap your CMake build, and let it pull the packages? Do you manually copy all the external deps into your own project repo, and then manually keep them updated with the latest changes?

krapht 5y ago
I think parent is saying to use a package manager.
- dagmx 5y ago
  That would be the same issue though, just moved to a different spot
  - nerdponx 5y ago
    Not necessarily. Package managers can cache things, especially if you're fetching the same version over and over. Also, separation of concerns is a good thing IMO.
    - howinteresting 5y ago
      Cargo etc also cache dependencies in a central location. "Separation of concerns" is ill-defined — you can use cargo as just a package manager, just a build system, or both. Most people use it as both but there are definitely places (e.g. bigcos) that use just one of the aspects of cargo.
  - gspr 5y ago
    Separation of tasks. The build system builds. Another system (like a package manager, if you are so inclined, or not if not) manages packages.
    - 5y ago

krapht 5y ago

People also curl and execute shell scripts from the internet every day.

gspr 5y ago
Yes. And it's a terrible idea.
- handrous 5y ago
  It's no worse than several other not-very-secure-but-nonetheless-common ways to install software. Would a MS-provided copy-paste-curl-sh-to-install, say, be any worse than a link to an MSI? Either way I'm trusting that MS isn't sending me ransomware or a rootkit or whatever. If that MSI just executes curl under the hood, exactly like the sh command would have, is it better (nb this is what thin web installers for big programs do, basically)? If I add a PPA for some company and apt-install something from it, unless I audit all the files before proceeding, I'm trusting that they won't format my disk without my wanting them to. How terrible an idea it is depends on who's providing it, same as most other software.
  - mappu 5y ago
    I do agree - There are some specific pitfalls with curl|bash: (A) the web server may serve different code for curl's user-agent than what you audited by clicking the link in a browser, leading to a false sense of security; (B) if the network transfer is interrupted, bash will partially execute an incomplete script; and (C) the recurring prevalence of high-profile bugs like https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671 You can mitigate (B) by wrapping the whole script in a function, and you can mitigate (C) with `set -euo pipefail` and `trap`, but that doesn't seem to be the kind of bash scripts that people write in practice. MSI (and deb/rpm) definitely can arbitrary bad things at install time too, but at least their built-in file extraction features are entirely declarative, and they always have an entrypoint for the system uninstaller.

mr_tristan 5y ago

It's the pragmatic solution to a hard problem. Supporting distribution for all the different kinds of Linux package mangers alone isn't easy, and you take in how macOS and Windows do things, I'm not even sure where you start. It's kind of sad, but there's multiple walled gardens in play here. I mean, Java's been around for 25 years, and it's just starting to get tooling for making OS installers for applications (jpackage). So, programming toolchains just invent package management that is easy for everyone to use. Everyone needs a problem solved today for multiple platforms, and it's not like OSes are working together to make this easy. Personally, I've come to prefer vendored dependencies with the sources, and building scripts around vendoring, separating the build from dependency distribution. But not a lot of programmers are comfortable with this. It's not a common practice, and the most popular version control system (git) isn't very great with binary management.

gspr 5y ago
The point isn't the whole debate about vendoring. I have strong opinions on that matter, and they are different from yours, but independently of whether one wants vendoring or not: It seems to me to be highly advantageous to decouple building from dependency management (nomatter if said dependencies are vendored or not).

intc 5y ago

This is a worrisome trend. Now one needs to manually check what the build system will fetch (and from where) before starting the build. Many times most of the dependencies are available through the package manager of ones system. A good way in my opinion is just to provide a clear list of the dependencies and make the build system complain if it's not there so one can install it (if didn't bother manually to check that the listed dependencies are present). Of course there are very large and complex projects which may need more sophisticated solutions - Perhaps some sort code notarisation would serve well? Yet even there the dependency management should be clearly separated from the build system and it should prioritize using system provided packages when possible. There are probably other considerations and implications too. Like preference of static linking over dynamic etc..