4 ms·

This is a neat idea - and not without precedent. Installing each package into it's own directory, and maintaining symlinks into that directory structure, was a

This is a neat idea - and not without precedent. Installing each package into it's own directory, and maintaining symlinks into that directory structure, was an approach explored for large system installations in the 90s and 00s. NIST depot, Xheir, CMU, and Nix all did this in varying ways. The advantage of this approach was that packages could be installed once on an NFS, scoped by version/configuration, and mounted into the filesystem of every work station on a campus. The workstations then just needed to maintain references to the files on the NFS for their particular software configurations. There is also some interesting exploration of composable configuration templates that allow workstations to inherit and extend a particular configuration (i.e. the physics department might have a base template). Nix is easily the most successful in this space. It even uses something similar to the NFS approach! It can maintain a remote cache of observed configurations to pull from instead of redoing work. Reading these LISA papers is a lot of fun. If you work in system administration (i.e. maintaining an Artifactory instance for your company) and have a day to spare - I'd very much recommend reading them, starting with the Nix paper! To get you started: Nix - https://www.usenix.org/conference/lisa-04/nix-safe-and-policy-free-system-software-deployment https://www.usenix.org/conference/lisa-04/nix-safe-and-polic... Depot Lite - https://www.usenix.org/conference/lisa-viii/depot-lite-mechanism-managing-software https://www.usenix.org/conference/lisa-viii/depot-lite-mecha...

dark-star 5y ago
Wait, when did we come to the point where "working in system administration" is considered synonymous to "maintaining an Artifactory instance"?
- dagw 5y ago
  About 3-5 years ago, give or take.
- r3trohack3r 5y ago
  At least in my previous jobs, maintaining the artifactory instance included maintaining package integrity in the repositories. Answering questions about shadowing in Java, unpublishing in Node, upstream timelines for patched debs, and tracking deprecation was part of that the job - and the teams worked to make machines do as much of that job as possible. Packages bootstrap filesystems for installations. We’ve parted out system administration at scale - multiple paths you can take.
pxc 5y ago
Nix is great! But GoboLinux is actually a few years older than Nix, isn't it? First release in 2003. :) Interestingly, GoboLinux Runners are basically the same concept as NixOS' FHSUserEnvs, where user namespaces are used to mount a subset of the existing shared libraries and other dependencies to make them visible (and the only ones visible) to a running program in a compatible way. This spares them some of the filesystem overhead of running a full OS container with its own package manager. So there's a bit of further convergence there, too. I love the cleanliness and legibility of the GoboLinux file hierarchy, and I think the simple, Mac-ish app bundling approach it takes is something many users might like and prefer over the rough edges of Nix or the complexity of Flatpak. A GoboLinux-like presentation could be great for a stable, impure NixOS downstream some day. Thanks for the link to the Depot Lite paper! I'd never heard of it.
- r3trohack3r 5y ago
  Very much welcome! I can geek out about package management all day. If you do want to dive into this stuff, I’d recommend reading both the Nix and Depot Lite papers of course, but their references section is also a gold mine of fascinating systems.
  - pxc 5y ago
    I spent some time with the Depot Lite paper, and it's wild how much much there are little hints (less strict) of many of the exact principles the are used in Nix. Turning software installations read-only after a few weeks was one that stuck out the most, along with the basic FHS adjustments. I've read through a few if the Nix papers before, but they're probably worth revisiting by now after using Nix for a few years and experimenting with Guix a bit. > I can geek out about package management all day. Same! I've been fascinated with package management probably since I was about 12 or 13, when I first tried desktop Linux. Package managers are incredibly powerful systems tackling an equally incredibly tough problem space. Multi-language package managers (i.e., ports systems and Linux distro package managers) have always especially impressed me. (Imo even the 'bad' Linux distro package managers are pretty good, as much as I can be harsh on them in certain aspects when comparing them.) Getting into Nix has made me think more deeply about the designs of the many language-specific package managers, since each one needs/gets a different treatment in Nixpkgs according to its unique properties. It's similarly provided an illuminating alternative to the container approach to traditional deployment troubles— one that conserves more of what is good about the traditional package management paradigms on Linux and elsewhere. Have you looked at Michael Stapelberg's work on package management? I think it's the most exciting work on package management outside the Nix and Guix worlds: https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/ https://michael.stapelberg.ch/posts/2019-08-17-introducing-d... Luet also looks interesting, but I've not really played with it yet. Its approach seems to be to add more package-like granularity to existing container systems: https://github.com/mudler/luet https://github.com/mudler/luet
chris_wot 5y ago
That’s kind of funny, because Microsoft do something similar with WinSxS. Only worse. What they do is store packages in the winsxs folder. They then do an NTFS hard link to where the component is installed, usually system32. If you ever used Vista and Windows 7 and started to run out of disk space over time, that’s because Microsoft never used to remove the old components after they were updated. In Windows 7, the initial plan after SP1 was to allow people to use a special switch on DISM.exe to remove old service pack files. Microsoft removed the idea of service packs and instead did continuous patching with rollups. It seems to taken them quite some time before they twigged to the fact that the old updates were using up a lot of disk space, because a long time later (almost at the end of the Windows 7 lifecycle) they added a Windows update to the disk cleanup wizard. Even worse, it would only remove the components after you rebooted but before you could login to Windows. If you ran it, it could sometimes take literally hours to get the computer to finish removing files from the component store.
rackjack 5y ago
What is a LISA paper?
- dsr_ 5y ago
  Large Installations Systems Administration, a now defunct conference.
  - rackjack 5y ago
    Thanks.
  - zuj 5y ago
    defunct ? Isn't this the same ? https://www.usenix.org/conferences/byname/5 https://www.usenix.org/conferences/byname/5
    - dsr_ 5y ago
      defunct. https://www.usenix.org/conferences/upcoming https://www.usenix.org/conferences/upcoming Note the non-existence of a 2022 or 2023 LISA.

dark-star 5y ago

Wait, when did we come to the point where "working in system administration" is considered synonymous to "maintaining an Artifactory instance"?

dagw 5y ago
About 3-5 years ago, give or take.
r3trohack3r 5y ago
At least in my previous jobs, maintaining the artifactory instance included maintaining package integrity in the repositories. Answering questions about shadowing in Java, unpublishing in Node, upstream timelines for patched debs, and tracking deprecation was part of that the job - and the teams worked to make machines do as much of that job as possible. Packages bootstrap filesystems for installations. We’ve parted out system administration at scale - multiple paths you can take.

pxc 5y ago

Nix is great! But GoboLinux is actually a few years older than Nix, isn't it? First release in 2003. :) Interestingly, GoboLinux Runners are basically the same concept as NixOS' FHSUserEnvs, where user namespaces are used to mount a subset of the existing shared libraries and other dependencies to make them visible (and the only ones visible) to a running program in a compatible way. This spares them some of the filesystem overhead of running a full OS container with its own package manager. So there's a bit of further convergence there, too. I love the cleanliness and legibility of the GoboLinux file hierarchy, and I think the simple, Mac-ish app bundling approach it takes is something many users might like and prefer over the rough edges of Nix or the complexity of Flatpak. A GoboLinux-like presentation could be great for a stable, impure NixOS downstream some day. Thanks for the link to the Depot Lite paper! I'd never heard of it.

r3trohack3r 5y ago
Very much welcome! I can geek out about package management all day. If you do want to dive into this stuff, I’d recommend reading both the Nix and Depot Lite papers of course, but their references section is also a gold mine of fascinating systems.
- pxc 5y ago
  I spent some time with the Depot Lite paper, and it's wild how much much there are little hints (less strict) of many of the exact principles the are used in Nix. Turning software installations read-only after a few weeks was one that stuck out the most, along with the basic FHS adjustments. I've read through a few if the Nix papers before, but they're probably worth revisiting by now after using Nix for a few years and experimenting with Guix a bit. > I can geek out about package management all day. Same! I've been fascinated with package management probably since I was about 12 or 13, when I first tried desktop Linux. Package managers are incredibly powerful systems tackling an equally incredibly tough problem space. Multi-language package managers (i.e., ports systems and Linux distro package managers) have always especially impressed me. (Imo even the 'bad' Linux distro package managers are pretty good, as much as I can be harsh on them in certain aspects when comparing them.) Getting into Nix has made me think more deeply about the designs of the many language-specific package managers, since each one needs/gets a different treatment in Nixpkgs according to its unique properties. It's similarly provided an illuminating alternative to the container approach to traditional deployment troubles— one that conserves more of what is good about the traditional package management paradigms on Linux and elsewhere. Have you looked at Michael Stapelberg's work on package management? I think it's the most exciting work on package management outside the Nix and Guix worlds: https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/ https://michael.stapelberg.ch/posts/2019-08-17-introducing-d... Luet also looks interesting, but I've not really played with it yet. Its approach seems to be to add more package-like granularity to existing container systems: https://github.com/mudler/luet https://github.com/mudler/luet

chris_wot 5y ago

That’s kind of funny, because Microsoft do something similar with WinSxS. Only worse. What they do is store packages in the winsxs folder. They then do an NTFS hard link to where the component is installed, usually system32. If you ever used Vista and Windows 7 and started to run out of disk space over time, that’s because Microsoft never used to remove the old components after they were updated. In Windows 7, the initial plan after SP1 was to allow people to use a special switch on DISM.exe to remove old service pack files. Microsoft removed the idea of service packs and instead did continuous patching with rollups. It seems to taken them quite some time before they twigged to the fact that the old updates were using up a lot of disk space, because a long time later (almost at the end of the Windows 7 lifecycle) they added a Windows update to the disk cleanup wizard. Even worse, it would only remove the components after you rebooted but before you could login to Windows. If you ran it, it could sometimes take literally hours to get the computer to finish removing files from the component store.

rackjack 5y ago

What is a LISA paper?

dsr_ 5y ago
Large Installations Systems Administration, a now defunct conference.
- rackjack 5y ago
  Thanks.
- zuj 5y ago
  defunct ? Isn't this the same ? https://www.usenix.org/conferences/byname/5 https://www.usenix.org/conferences/byname/5
  - dsr_ 5y ago
    defunct. https://www.usenix.org/conferences/upcoming https://www.usenix.org/conferences/upcoming Note the non-existence of a 2022 or 2023 LISA.