3 ms·

There are a few options for Gentoo not discussed in the post, possibly because for Gentoo they would be a larger amount of work due to the design of their syste

by codys 2y ago

There are a few options for Gentoo not discussed in the post, possibly because for Gentoo they would be a larger amount of work due to the design of their system: 1. Allow building against packages without installing them. The core issue here is that Gentoo package build and install happen as a single step: one can't "build a bunch of things that depend on one another" and then "atomically install all the build items into place". This means that Gentoo can easily be partially broken when one is doing updates when an ABI change occurs (modulo so versioning, see next option). This issue with 64-bit time_t is an example of an ABI change that folks are very aware of and is very widespread. It's also an example of something that causes an ABI change that isn't handled by the normal `.so` versioning scheme (see next option). 2. Extend the normal `.so` versioning to capture changes to the ABI of packages caused by packages they depend on. Normally, every `.so` (shared object/library) embeds a version number within it, and is also installed with that version number in the file name (`libfoo.so.1.0.0`, for example, would be the real `.so` file, and would have a symlink from `libfoo.so` to tell the linker which `.so` to use). This shared object version is normally managed by the package itself internally (iow: every package decides on their ABI version number to enable them to track their own internal ABI breakages). This allows Gentoo to upgrade without breaking everything while an update is going on as long as every package manages their `.so` version perfectly correctly (not a given, but does help in many cases). There is a process in Gentoo to remove old `.so.x.y.z` files that are no longer used after an install completes. What we'd need to do to support 64-bit time_t is add another component to this version that can be controlled by the inherited ABI of dependencies of each `.so`. This is very similar in result to the "use a different libdir" option from the post, but while it has the potential to set things up to enable the same kinds of ABI changes to be made in the future, it's likely that fixing this would be more invasive than using a different libdir.

account42 2y ago
Gentoo already does retain old libraries until all dependends are updated so encoding the abi change in places where ABI changes are supposed to be encoded (architecture and SONAME) would solve this problem.
akira2501 2y ago
> Allow building against packages without installing them. A partial staged update system would work the best. I should be able to schedule the build of several new packages, have them built into a sandbox, and have new compilations use the sandbox first and then fallback to the system through a union, and then once everything is build finally package up all the pieces and then move them out of the sandbox and into the system at large. You could make all gentoo updates transactional that way which would be a huge boon in many other ways.
- account42 2y ago
  You definitely don't want all upgrades to be transactional, at least not without being able to reuse partial results. E.g. having to rebuild an llvm update because some other package failed to build would suck.
  - codys 2y ago
    Presumably one wouldn't throw away the built packages if some other build fails just because we're delaying installing the packages to `/` until later.
aaronmdjones 2y ago
Gentoo already allows option 1 by specifying a directory to ultimately install the finished image to (normally it is set to /) [1]. One can do a complete rebuild of everything in @system and @world, install it to the specified subdirectory, and then sync it all over in one shot. Preferably you would do this from a live session, although in theory you could also bind-mount / onto a subdirectory of the place you reinstalled everything to, chroot into it, and then sync it (what is now /) into the bind-mounted real upper /. [1] https://devmanual.gentoo.org/ebuild-writing/variables/#root https://devmanual.gentoo.org/ebuild-writing/variables/#root
- codys 2y ago
  It's true that Gentoo has some pieces of what they need to build an implementation of option 1 (build packages that depend on other packages before completeing an install), but it's currently the case that that is not what happens when one runs `emerge` (the gentoo packaging build/installing tool), as you've noted one would need to write scripts that wrap emerge (or do the same work manually) to attempt to accomplish this today. I suspect that using bind mounts and overlays (to allow the "building packages" chroot a view of the "installed" root) could be used to accomplish this, or alternately some filesystem snapshotting features if we're thinking about this from the "external to emerge" angle. (It's my understanding that for Gentoo to do this ABI change, though, they need some solution integrated into emerge). To some extent, this kind of potential model also reminds me of systems that integrate their package updating with filesystem snapshotting to allow rollbacks and actually-atomic upgrades of many files. I think one or more of the solaris distributions did this?
  - aaronmdjones 2y ago
    > but it's currently the case that that is not what happens when one runs `emerge` (the gentoo packaging build/installing tool) This would be a simple matter of changing /etc/portage/make.profile to point to a profile that uses the new ABI and setting (and exporting) the ROOT variable in the shell you type "emerge" into (or add this ROOT=/whatever/ line to /etc/portage/make.conf), is what I was getting at. There's no need to write any wrappers.

by codys 2y ago

account42 2y ago

Gentoo already does retain old libraries until all dependends are updated so encoding the abi change in places where ABI changes are supposed to be encoded (architecture and SONAME) would solve this problem.

akira2501 2y ago

> Allow building against packages without installing them. A partial staged update system would work the best. I should be able to schedule the build of several new packages, have them built into a sandbox, and have new compilations use the sandbox first and then fallback to the system through a union, and then once everything is build finally package up all the pieces and then move them out of the sandbox and into the system at large. You could make all gentoo updates transactional that way which would be a huge boon in many other ways.

account42 2y ago
You definitely don't want all upgrades to be transactional, at least not without being able to reuse partial results. E.g. having to rebuild an llvm update because some other package failed to build would suck.
- codys 2y ago
  Presumably one wouldn't throw away the built packages if some other build fails just because we're delaying installing the packages to `/` until later.

aaronmdjones 2y ago

Gentoo already allows option 1 by specifying a directory to ultimately install the finished image to (normally it is set to /) [1]. One can do a complete rebuild of everything in @system and @world, install it to the specified subdirectory, and then sync it all over in one shot. Preferably you would do this from a live session, although in theory you could also bind-mount / onto a subdirectory of the place you reinstalled everything to, chroot into it, and then sync it (what is now /) into the bind-mounted real upper /. [1] https://devmanual.gentoo.org/ebuild-writing/variables/#root https://devmanual.gentoo.org/ebuild-writing/variables/#root

codys 2y ago
It's true that Gentoo has some pieces of what they need to build an implementation of option 1 (build packages that depend on other packages before completeing an install), but it's currently the case that that is not what happens when one runs `emerge` (the gentoo packaging build/installing tool), as you've noted one would need to write scripts that wrap emerge (or do the same work manually) to attempt to accomplish this today. I suspect that using bind mounts and overlays (to allow the "building packages" chroot a view of the "installed" root) could be used to accomplish this, or alternately some filesystem snapshotting features if we're thinking about this from the "external to emerge" angle. (It's my understanding that for Gentoo to do this ABI change, though, they need some solution integrated into emerge). To some extent, this kind of potential model also reminds me of systems that integrate their package updating with filesystem snapshotting to allow rollbacks and actually-atomic upgrades of many files. I think one or more of the solaris distributions did this?
- aaronmdjones 2y ago
  > but it's currently the case that that is not what happens when one runs `emerge` (the gentoo packaging build/installing tool) This would be a simple matter of changing /etc/portage/make.profile to point to a profile that uses the new ABI and setting (and exporting) the ROOT variable in the shell you type "emerge" into (or add this ROOT=/whatever/ line to /etc/portage/make.conf), is what I was getting at. There's no need to write any wrappers.