4 ms·

One could develop a whole package manager based on this. Add some standard build profiles to something like Gentoo's portage to maximize cache hits and you coul

One could develop a whole package manager based on this. Add some standard build profiles to something like Gentoo's portage to maximize cache hits and you could have the best of both worlds: a fast source-based distribution. Specifying a custom build profile for a package would just mean it would take more time to install but would otherwise be transparent. One thing missing from this is a trust framework where cache writers would sign their compiled code. There could also be a verification layer.

wmanley 4y ago
I wrote a little bit about this a couple of years ago: https://blog.williammanley.net/2020/05/25/unlock-software-freedom-one-by-using-better-tools.html https://blog.williammanley.net/2020/05/25/unlock-software-fr... gittup[1] implements part of this idea, but without the distributed bit. bazel[2] implements the distributed bit (minus trust), but not the distro bit. What's really lacking is momentum around the idea to get a sufficient number of people behind it. [1]: https://github.com/gittup/gittup https://github.com/gittup/gittup [2]: https://bazel.build/ https://bazel.build/
yjftsjthsd-h 4y ago
Nix is also almost there - it happily uses binaries if they match the exact build inputs but compiles if no binaries match - but it's not nearly as fine grained (you'd need something like 1 package per object file).
- ghoward 4y ago
  And the saddest part is that Nix could be exactly there and solve another one of their problems at the same time: content-addressed outputs.
heavenlyhash 4y ago
Warpforge -- project website: http://warpforge.io http://warpforge.io / https://github.com/warptools/warpforge https://github.com/warptools/warpforge -- is a project I work on that's heading a bit in this direction. Hashes in + build instruction = hashes out. It's really a powerful primitive indeed. Building big sharable package systems is indeed a second half of the problem. Figuring out how to make the system have the hashy goodness AND be usefully collaborative with async swarms of people working independently and yet sharing work: tricky. We're trying to do it, though! We have our first few packages now published, here: https://catalog.warpsys.org/ https://catalog.warpsys.org/ You can also see _how_ we build things, because we publish the full rebuild instructions with each release: for example, here's how we packaged our bash: https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3Vgei44et6RzTA785sEZGwuFV75vCazjhR11RH5veFdMTx7F5cg2c4NA5HXPK8Zv5TQ.html https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3V... I'm in #warpforge on matrix with some collaborators if anyone's interested in joining us for a chat and some hacking :)
M95D 4y ago
Gentoo supports building and installing binary packages from portage: You can compile once and then distribute the resulting package to other computers where they can be installed by Portage. Portage even verifies that the USE flags are the same!
Cloudef 4y ago
It's called nix
nextaccountic 4y ago
NixOS uses a cache that works like this; it's source-based like Gentoo but for packages in cache it will download the cache rather than rebuilding. (it will rebuild if you configure the package in such a way to invalidate the cache, or install a package not in cache) Besides the global cache for the whole distro, you can also set up caches for other software. For example, if you build your projects with Nix, you can have a cache for your projects (so that new contributors won't need to recompile everything from scratch). That's the premise behind https://www.cachix.org/ https://www.cachix.org/ The only difference is that Nix caches aren't fine grained like ccache and sccache

wmanley 4y ago

I wrote a little bit about this a couple of years ago: https://blog.williammanley.net/2020/05/25/unlock-software-freedom-one-by-using-better-tools.html https://blog.williammanley.net/2020/05/25/unlock-software-fr... gittup[1] implements part of this idea, but without the distributed bit. bazel[2] implements the distributed bit (minus trust), but not the distro bit. What's really lacking is momentum around the idea to get a sufficient number of people behind it. [1]: https://github.com/gittup/gittup https://github.com/gittup/gittup [2]: https://bazel.build/ https://bazel.build/

yjftsjthsd-h 4y ago

Nix is also almost there - it happily uses binaries if they match the exact build inputs but compiles if no binaries match - but it's not nearly as fine grained (you'd need something like 1 package per object file).

ghoward 4y ago
And the saddest part is that Nix could be exactly there and solve another one of their problems at the same time: content-addressed outputs.

heavenlyhash 4y ago

Warpforge -- project website: http://warpforge.io http://warpforge.io / https://github.com/warptools/warpforge https://github.com/warptools/warpforge -- is a project I work on that's heading a bit in this direction. Hashes in + build instruction = hashes out. It's really a powerful primitive indeed. Building big sharable package systems is indeed a second half of the problem. Figuring out how to make the system have the hashy goodness AND be usefully collaborative with async swarms of people working independently and yet sharing work: tricky. We're trying to do it, though! We have our first few packages now published, here: https://catalog.warpsys.org/ https://catalog.warpsys.org/ You can also see _how_ we build things, because we publish the full rebuild instructions with each release: for example, here's how we packaged our bash: https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3Vgei44et6RzTA785sEZGwuFV75vCazjhR11RH5veFdMTx7F5cg2c4NA5HXPK8Zv5TQ.html https://catalog.warpsys.org/warpsys.org/bash/_replays/zM5K3V... I'm in #warpforge on matrix with some collaborators if anyone's interested in joining us for a chat and some hacking :)

M95D 4y ago

Gentoo supports building and installing binary packages from portage: You can compile once and then distribute the resulting package to other computers where they can be installed by Portage. Portage even verifies that the USE flags are the same!

Cloudef 4y ago

It's called nix

nextaccountic 4y ago

NixOS uses a cache that works like this; it's source-based like Gentoo but for packages in cache it will download the cache rather than rebuilding. (it will rebuild if you configure the package in such a way to invalidate the cache, or install a package not in cache) Besides the global cache for the whole distro, you can also set up caches for other software. For example, if you build your projects with Nix, you can have a cache for your projects (so that new contributors won't need to recompile everything from scratch). That's the premise behind https://www.cachix.org/ https://www.cachix.org/ The only difference is that Nix caches aren't fine grained like ccache and sccache