8 ms·
Yeah. Not sure about BSD, but I was wading into building the GNU coreutils and other GNU packages just yesterday. Fresh hell, they all seem to be build dependen
by throwaway894345 7y ago
Yeah. Not sure about BSD, but I was wading into building the GNU coreutils and other GNU packages just yesterday. Fresh hell, they all seem to be build dependencies of each other. ‘sed’ is its own build dependency.
The whole C ecosystem is a joke with respect to builds—all dependencies are implicit; you’re just expected to have the exact dependencies installed on your system at the exact versions and in the exact locations on the filesystem that the build tooling is willing to look. Autotools and CMake are absolutely terrible; they make Python’s build ecosystem look sane and well-designed.
So no, “security” isn’t the most compelling use case (for me, anyway), it’s moving past these dumpster fire build systems as quickly as possible so mere mortals can build their own software. Specifically, hasten the day where my entire application dependency tree doesn’t bottom out in some Autotools, CMake, shell script, project.
- toast0 7y ago> Yeah. Not sure about BSD, but I was wading into building the GNU coreutils and other GNU packages just yesterday. Fresh hell, they all seem to be build dependencies of each other. ‘sed’ is its own build dependency. This is an area where the BSDs are pretty good. There may not be a dependency system here exactly, but because XBSD base builds XBSD base, everything you need is in one checkout. Generally with just (BSD) Make driving the build. Programs outside of base are genrerally built with Portfiles which specify dependencies such that (BSD) Make can satisfy them. But somebody has to write the Portfile, and it's often not the authors of the program.
- sjakobi 7y agoRegarding build tools for C, you might find this one interesting: https://github.com/vmchale/cpkg https://github.com/vmchale/cpkg
- throwaway894345 7y agoYeah, there are lots of good build systems for C (didn’t know about this one; will have a look; thanks for sharing), but the problem is that they have tiny adoption, especially among “core” ecosystem packages, like coreutils and friends. The problem isn’t technical, it’s political and cultural. This ecosystem is hostile toward its users for no discernible reason. If anyone knows of a distribution that aims to use only software with sane build systems, let me know, I would be very curious.
- cycloptic 7y agoBootstrapping and too many binary dependencies at the early stage is a known problem in GNU Guix and there are people working to fix it, such as the GNU Mes project to bootstrap the system using a minimal compiler, shell and set of utilities written in Scheme. Despite this, there are still many steps to bootstrapping a full system and the complete process is not simple. It's not for "no reason", in general you will not be able to get away from having a complicated build system at the lowest level. Otherwise you would need to be prepared to give up on nice things like cross-compiling and supporting more than one compiler, CPU architecture, etc.
- Ericson2314 7y agoActually focusing on cross generally usually forces when to be bootstrapping better. The problem is most distros / deployed systems are managed and bootstrapped imperatively, which there is nothing pushing back against complicating the bootstrapping. Thus, entropy wins and it gets more complex. As someone who as actually started about reproducing BSD bootstraps declaritively (https://github.com/NixOS/nixpkgs/pull/82131/files https://github.com/NixOS/nixpkgs/pull/82131/files) they are way more entangled than they need to be, so Theo de Raadt has no ground to stand on.
- yawaramin 7y agoNix is becoming more of a name in dependency management systems: https://blog.galowicz.de/2019/04/17/tutorial_nix_cpp_setup/ https://blog.galowicz.de/2019/04/17/tutorial_nix_cpp_setup/ Its claims to fame are support for multiple languages, reproducible builds, and system-wide package caching.
- throwaway894345 7y agoYeah, Nix is really neat. It's the right direction for sure, but it's mostly used as a wrapper around autotools and cmake projects by third parties and not so much as a first-class build tool by the maintainers of C/C++ libs (this isn't going to change because Nix prefers to be "innovative" and unfamiliar in just about every design decision--its core approach to reproducible builds is right on, but the execution leaves a lot to be desired if you want to convince a community of developers to adopt it as a build tool). So anyone writing a Nix expression to package an autotools or cmake project (me, in this case) still has to deal with their nonsense.
- zozbot234 7y ago> Fresh hell, they all seem to be build dependencies of each other. ‘sed’ is its own build dependency. Except that people have run into the exact same issue with stuff like rustc and cargo. Bootstrapping an entirely new platform is just hard, no way around it.
- throwaway894345 7y agoComparing coreutils to a self-hosting compiler is a strained analogy. Bootstrapping a compiler is about as hard as it gets; there’s no reason coreutils needs to depend on itself.
- zozbot234 7y agoThere's "no reason" why the rustc build should depend on cargo, but apparently it does. And this makes it way harder than it should be to keep rustc support up-to-date in any distro, which in turn impacts stuff like Firefox updates.
- steveklabnik 7y ago> There's "no reason" why the rustc build should depend on cargo, rustc is a Rust project. Cargo is a build system for Rust projects. > this makes it way harder than it should be to keep rustc support up-to-date in any distro I am not aware of distro maintainers complaining about this, or at least, not any time in the last few years. Do you have some context to share? We want distros to have a good experience.
- GrayShade 7y agoDebian unstable (IIRC) updated Firefox a couple weeks late because cargo got a couple of new dependencies and nobody added them to the Debian repos. This is because they forbid static linking, so there are hundreds of packages of popular Rust crates compiled as shared libraries. Even worse, some of them have multiple variants, for various combinations of features.
- Uehreka 7y agoAgreed on a hundred thousand. I used to feel sheepish defending npm and the whole node package ecosystem against its critics, but once I started trying to run deep learning applications or WebRTC media servers I quickly realized that some of the critics are probably coming from a much worse package management system that they’ve merely grown familiar with. Want to have two pieces of software on one computer that rely on... idk, two different versions of libwebsockets? Yeah, you’ll need either docker/overlayfs, or some filesystem or pkgconfig hack that I’m sure a comment reply will helpfully mention if you want two have to versions of a library installed for two different executables.
- Arch-TK 7y ago"two different versions of libwebsockets" If your programs depend on different MAJOR versions of libwebsockets and libwebsockets' build system doesn't easily allow that to happen then complain to the developer of libwebsockets about it. If the developer of libwebsockets is not interested in solving this problem then complain to the developers of the programs which depend on libwebsockets that they should find a better library which doesn't suck. If they don't listen, rewrite the tools you depend on to not suck. If, on the other hand, the programs depend on different minor versions of libwebsockets and libwebsockets broke API in such a way that just using the highest common version doesn't work then complain to the developer of libwebsockets that they should stop doing terrible things like breaking API and then complain to the developers of those applications that they shouldn't let this kind of breakage slide and should move to a different library. If you think this is unrealistic then I think your push towards insane let's-pull-all-the-dependencies-into-one-binary/package build systems is unrealistic. You want a sane build system? The change starts when you stop tolerating idiots who cause things not to work. We don't tolerate engineers who build crappy bridges which break and kill people. You shouldn't be tolerating programmers who build crappy libraries which break API or don't allow two major-versions to be easily simultaneously installed and linked against.
- Uehreka 7y agoMy issue is that the default behavior is to install these libraries globally, even when they could probably be installed locally for the thing you’re trying to build. (I picked libwebsockets out of the air, I don’t necessarily have a complaint with that particular library) On my old MacBook Pro I have dozens of node projects that I’ve downloaded over the years just to mess around. If I ever want to clean up, I can delete a project’s folder, and know that everything it installed is now gone (because it would be in its node_modules folder). On my linux desktop if I want to mess around with a cool project that’s written in C/C++ I immediately pull out Docker and start trying to build up an image, since I don’t want to pollute my global library folders with libraries that I might be downloading just to try out this one thing. I can then delete the docker image if I want to clean up. This works, but it feels like a workaround to the main problem of libraries being global.
- Arch-TK 7y agoPython's build ecosystem IS sane and well-designed. It's the pinnacle of package handling so far. A separate package manager which works together with a separate build system which still works when a distribution wants to package a python package. What build systems like cargo seem to think is awesome is probably the most backwards idea of build systems I've ever seen. Static linking and automatic downloading of dependencies. Even the ability to directly pull from a github repo. No wonder almost nothing written in rust ever gets packaged. Dependencies are hard, following a spec like semver seems to have confused people. Developers are lazy and don't want to think about settling down their API. Solution? Just give up and encourage bad behaviour. Thanks guys. Great solution. I think I'll stick with the previous one.
- steveklabnik 7y ago> No wonder almost nothing written in rust ever gets packaged. rustc is also built with cargo, and it is packaged on most distros. A lot of distros are including packages of other tools too.
- roblabla 7y ago> Python's build ecosystem IS sane and well-designed. It's the pinnacle of package handling so far. Big oof. Python packaging has a lot of problems, and defaulting to a single shared location when installing, that might conflict with your default package manager, is very high on the "What the fuck are you even doing" list of issues. You also very easily get into version hell. Unless you start using virtualenv, at which point you're basically doing the same thing cargo is. > automatic downloading of dependencies How is that a problem? Cargo has a lockfile to ensure the dependency downloaded is the exact same, various flags to prevent cargo from updating the lockfile (--frozen and --locked), and even the ability to vendor dependencies with cargo vendor[0]. It defaults to the most convenient things for developers because packages will be worked on many more times than they are packaged. > Even the ability to directly pull from a github repo. So does python/pip. And most package managers I've worked with. Crates.io will refuse packages that have github dependencies though, so this only comes up when packaging something that isn't really ready to be packaged. As for linking, there are several rust-specific problems to dynamic linking: 1. Rust currently has no stable ABI (apart from the C ABI). This means updating rustc (and maybe even updating llvm) may cause the ABI to change, and libraries to become incompatible. There is currently no plan to fix this on Rust's part, and in fact some opposition, since it would prevent some optimizations from happening (such as niche-filling optimizations). 2. Generic functions and impls would not be part of this, as monomorphised variants would have to end up in the final binary. I believe C++ has the same problem with templates. A lot of Rust APIs are generic. This could theoretically be fixed (at least partially) through automatic promotion to dynamic dispatch, or other similar schemes. [0]: https://doc.rust-lang.org/cargo/commands/cargo-vendor.html https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
- adev_ 7y ago> So no, “security” isn’t the most compelling use case (for me, anyway), it’s moving past these dumpster fire build systems as quickly as possible so mere mortals can build their own software. I will be mean, I am sorry but this kind of attitude piss me off. If you are not able to run a ./configure / make / make install & 3 apt-get properly. Then you should just stay far from keyboard and not try to build anything. Whatever the language and the build system. These "dumpfires" have been able for 20 years to build program scaling from 10 lines to 100M lines... On platforms that are maybe older than you (AIX, HP-UX...) and ranging from micro-controlers of few KB of RAM to the biggest supercomputers in the world... They work for for kernel mode and user mode... and are also able to build an entire OS distribution like OpenBSD. They are indeed not perfect.... but they deserve at least a little bit of respect, even from the script kiddies not able to understand them. And honestly, an other point where Theo is right: I bet that none of their (your) '3.0 Modern successor' will be able to replace them in the next 15 years.
- staticassertion 7y ago"If you are not able to run a ./configure / make / make install & 3 apt-get properly." Nearly choked on my water reading this. Incredible how many developers are stockholme syndrome'd by their godawful tooling experience.
- adev_ 7y ago> Nearly choked on my water reading this. Incredible how many developers are stockholme syndrome'd by their godawful tooling experience. I tend to name a "magic bulshit bloat tool" that will makes me download 350MB of crap to compile an 'hello world' a "godawful tooling experience". We are all free to have our own definition.
- staticassertion 7y agoYou can tell me that other build tools have problems, but the "You must know how to run some os/project-specific commands to write a program or you're a bad developer" is the funniest kind of gatekeeping to me. But keep calling people "script kiddies" that's cool.
- geokon 7y agoWith CMake you can build statically in a very reproducible cross platform way if you use Hunter and toolchain files. the tools are all there - just they're not popular and aren't widely used I wrote a bit of an old intro to the workflow: https://geokon-gh.github.io/hunterintro.html https://geokon-gh.github.io/hunterintro.html This is an example of a project that's well setup using this pattern: https://github.com/elucideye/drishti https://github.com/elucideye/drishti And having things work dynamically ... well that's what package maintainers due. It seems inherently very fragile. Developers only build test and develop against whatever dependency versions they have locally. you can't test against every version under the sun
- akira2501 7y ago> Fresh hell, they all seem to be build dependencies of each other. ‘sed’ is its own build dependency. Well, they are the "core" utilities after all. This becomes less of an issue the further out in the ecosystem you get. > at the exact versions and in the exact locations on the filesystem that the build tooling is willing to look. This isn't strictly true.. you just need _compatible_ versions to be available, and most build tools and even the compiler itself will check several commonly used locations for code. There are a few corner cases, but by and large, this isn't as big a problem as it might be for other languages that make writing incompatible interfaces orders of magnitude easier than with C. > Specifically, hasten the day where my entire application dependency tree doesn’t bottom out in some Autotools, CMake, shell script, project. Each those solve the same problem at different levels. Perhaps this is because there is no _one true_ way to replace them, and the advantage is that each project uses the tool most suited to their case. Perhaps the built-in tools with these "modern" languages are a good general tool for the majority of developers, perhaps they aren't. If they aren't, then now you have the same problem all over again in a new ecosystem.
- throwaway894345 7y agoI don’t know man, I’ve written a lot of C and C++ and other languages and the sheer effort to get your average C project to compile is much greater than others. The dependencies aren’t listed anywhere, they are rarely compatible with the versions available via your package manager (so now you’re building more things from source), ./configure regularly runs into problems (“couldn’t guess which platform you were on”, “your version of grep is incompatible”, etc). It’s just a big mess of yak shaving. If you think I’m making too big a deal, I invite you to spend some time in the Go or Rust ecosystems where you just “cargo build” or “go build” and you’re done. You don’t need to understand anything about the project, there are no shitty DSLs to grok (looking at you, CMake). Things just work.