10 ms·

Writing a Package Manager

163 points by celiktom 3y ago

Richard6544 3y ago
[dead]
celiktom 3y ago
Writing a package manager is not one of the most common programming tasks. After all, there are many out-of-the-box ones available.
- 3np 3y ago
  These copy-pasted one-liners aren't useful. Skip them.
- 000ooo000 3y ago
  Imagine what'd we be using for package management today if package manager >= #2 author said "you know what, one already exists".
  - jsunderland323 3y ago
    This is the package manager paradox. The problem is multiple package managers but to address it we build more package managers.
  - queuebert 3y ago
    I think we'd all be using 'tar'.
forrestthewoods 3y ago
Wonderful post! I've written a couple of packager manager like things over the years. Although they've all been internal-only which gives some flexibility. > I like the idea of allowing both project and global scope Yikes! Hard no from me. Globals are pure evil. Avoid like the plague. Environment variables too. Kill them all with fire. > what if the user wants to reinstall the packages on another machine or CI server? That’s where the lockfile comes in I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control! That's a rant for another day. Treating the packages folder as source of truth is basically the equivalent I think? > I understand that dropping dependencies altogether may not be something you are ready to accept. Nice. A corollary to this is that if all your packages are internal then you can simply disallow dependency graphs that want different versions of the same package. Simply bump and fix-up all version number dependencies for all packages. The basically means "all packages play nicely on latest version". Which when all the packages live in a monorepo is perfectly plausible. Totally doesn't work for public package managers, but that's a different story.
- verdverm 3y ago
  It totally works for Go with MVS, which bumps a lot of versions behind the scenes for you.
- RajT88 3y ago
  > Environment variables too. Kill them all with fire. I feel this way about host file entries. It is shocking how common they remain, and not just in QA/Dev.
- nonameiguess 3y ago
  > I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control! These are binary dependencies. You're checking shared object files into source control? Note that he's also doing this to add functionality to a binary installation of sqlite, which is presumably running somewhere like /usr/bin/sqlite. This isn't a custom application he's developing. The extensions are in his home directory. "Checking them into version control" would entail making his entire filesystem a Git repo. If you're willing to pay whatever IBM charges these days for ClearCase, something like that might actually work. Regular source control like Git, though? I don't think so.
  - forrestthewoods 3y ago
    Binaries absolutely should go in version control. The fact that Git is incapable of efficiently supporting that workflow is a separate topic.
    - seabass-labrax 3y ago
      I think you are right. Tarballs with version strings as filenames served from an HTTPD are effectively a poor man's version control. Git commits are immutable and efficient (storing only the delta) for text, but those same benefits could apply to binary files too if only the tooling was better. You may be interested to know that Debian actually does use Git to store binary data, although only when it is not possible to use the source files to generate the binary data directly.
BearhatBeer 3y ago
I think some day rather than the current paradigm, even including declarative package managers and environments and distros, the future will be per-user and even per-app chrooting or jails, or something similar. Apple already uses something like this today. Many people who are smart about information security have one login for shopping and banking and bill pay, one for their business, and one for cruising the web or gaming or whatever. That way a breach of one doesn't necessarily end up pwning the whole system. I only have two accounts, one "serious" and one "off hours" but I still feel better protected than most people.
- czscout 3y ago
  Isn't this already how Android handles app permissions? Each application runs as its own user for the sake of security. The Application Sandbox is a pretty cool interpretation of the long existing Unix user/group paradigm.
- nextaccountic 3y ago
  > Apple already uses something like this today Some Linux distros too have things like this but unfortunately there is no buy-in across the ecosystem so "sandboxing" is done in a half-baked way. The problem is when applications in general aren't written with sandboxing in mind, and when you have to choose between apps not working properly or having a leaky sandbox, you will opt for the latter. I wish some big corp bit the bullet and ported hundreds of apps to a new, sandboxed environment in Linux, while attempting to upstream the whole effort. This would necessarily involve things like fully migrating to Wayland (X11 security is awful), only granting filesystem access through distro-sanctioned file pickers (so you need some coordination among Gtk, Qt, and other toolkits), and generally having a deny-by-default policy: first you make secure, then you fix what broke.
  - BearhatBeer 3y ago
    Firejail proves you can sandbox most anything, OpenBSD has their pledge and unveil too. I guess there's a gradient there but each program should be written or constrained to only being able to access what it needs ideally. Per-task groups could go a long way toward solving this, they researched it in the '80s even. Just create and destroy groups on the fly to enable processes to access only what they need. Unix is flexible enough to permit experimenting here.
dclowd9901 3y ago
Man, I really wish frontend package management could be this simple, but with a system that ultimately ships bytes over the wire, we have to not only massage the dough with chopsticks (insofar as try to reduce our dependency tree with weak, feckless tools), but also, with the sheer number of transitive dependencies for each package addition, it’s almost impossible to really know what’s going on with your dependencies unless you write plug-in after plug-in.
larusso 3y ago
Awesome post! I worked with a bunch of package managers over the years and one can see that this design got inspired by the hood parts of a few I know. The only design part I don‘t really like is the ‚latest‘ version specifier in the spec file. Which moves the declaration what the latest version is to the hosted location (in the example GitHub via GitHub API) paired with the fact that the checksums are also fetched rather than being part of the spec. This makes no sense for me. The spec needs to be versioned or better the spec is the actual location to declare a release. I kind of understand where the desire comes from to have a floating spec. Makes the publish process easier since one only creates a GitHub release in this case. But I would still argue that explicitly creating a new version of the spec for each version with baked checksums is better. The benefit for me would be that one could create a hash for each spec version for instance and use that internally for instance.
- friendzis 3y ago
  Probably the simplest version of a "package manager" are git submodules. Pointing submodule to `master` is effectively the same thing as pointing a "real" package manager to `latest`. This is trunk based development, however you implement it. One could easily argue that floating versions are considered harmful for releases outside of development team and should always be pinned, but on the other hand it is hard to argue against support for floating versions in development. As much as I dislike floating versions, I am not aware of any other way to force changes downstream.
  - larusso 3y ago
    Brew has the —head flag where one can instruct to build the latest commit from the repo. But the spec/Formular needs to set a head to pull from.
deepsun 3y ago
Sounds like Maven had all this solved many years ago. Yes, it cannot run arbitrary code, like NPM does, it just copies files, but the dependencies and specfiles were there from the beginning.
- ojkelly 3y ago
  Not running arbitrary code is a feature, that yarn brings back to the js ecosystem. It sounds nice in theory, but it impacts the soundness of the whole system.
- layer8 3y ago
  Maven didn’t support transitive dependencies from the beginning.
david2ndaccount 3y ago
Honestly pretty strange to write a package manager for sqlite and to use directories + json files to store the data instead of sqlite.
- gosenx 3y ago
  Great point
- gjvc 3y ago
  local system log "files" and package databases would be an excellent example of using sqlite
- grumblingdev 3y ago
  SQL not so good for tree structures. Recursive CTEs are pretty gnarly. Graphdb would be ideal.
  - jeremyjh 3y ago
    You want to bring in a dependency to avoid writing one 8 line CTE?
    - christophberger 3y ago
      The number of lines needed is not a good measure for gnarliness. (To be fair, I don't know if gnarliness is measurable at all.)
      - jeremyjh 3y ago
        It isn’t gnarly at all, but recursive CTE are outside the experience of most SQL devs. Here “gnarly” is used to describe “unfamiliar”.
  - jbverschoor 3y ago
    SQLite is a pretty darn good solution for any file(set) that needs atomic / consistent writes
mijoharas 3y ago
It's surprisingly to me that no-one has built a asdf style package manager. (I'm not talking about system package managers, their language software packages are always out of date and get installed globally instead of locally to a project). Having a unified interface to a package manager per language that will use the languages registry could be really nice (I guess you'd have some core dependency management functions and agnostic ways to store and download the packages, and plugins per language that would use these common building blocks to actually do stuff?). Another advantage of this could be cross language dependencies which often aren't handled well by package managers.
- ljm 3y ago
  You could argue that Bazel, Nix and Guix solve that problem but they introduce a lot of complexity as a result because it's not a simple abstraction. Hell, you could probably pull it off somehow in Cmake if you placed no value on your sanity. Most standard lib package managers double up as build tools for bundling your code after all. It's a totally different problem space than the one asdf aims to solve, which is just managing multiple independent versions of some programming language binaries in a consistent way, so you're not dancing between nvm, rvm/rbenv/chruby, virtualenv, etc. etc.
- kdeldycke 3y ago
  Something like Meta Package Manager? https://github.com/kdeldycke/meta-package-manager https://github.com/kdeldycke/meta-package-manager
- otabdeveloper4 3y ago
  Nix solves that problem. (While creating some new ones.)
riffraff 3y ago
Isn't this missing the SQLite version? Are SQLite extensions guaranteed to work across different SQLite installs?
- matharmin 3y ago
  SQLite is quite good at staying backwards compatible, and this includes the APIs exposed to extensions, but extensions could definitely have a minimum SQLite version.
brabel 3y ago
The author added a lockfile without understanding why they exist. A lockfile is meant to "freeze" dependency version resolution when package authors can specify dependencies on other packages using version ranges... it also "freezes" choices of transitive packages' versions when different packages depend on the same one, but with different versions. They chose to not handle package dependencies at all, and I believe there's no version ranges either, so I really don't see why they added a lockfile.
- pharmakom 3y ago
  A version number is just a label, and labels are mutable. A lock-file containing hashes will always resolve to the same packages (or fail).
  - qwerty456127 3y ago
    Depending on a single specific version is wrong. A library can be changed to fix a bug without changing the interface in any breaking way. It should be possible for a user (also their package manager, automatically) to replace the library with the new version in this case.
    - zivkan 3y ago
      Depending on how important supply chain security is to your industry/company/team, validating the hash of every package is critical. If an attacker can manage an interception/man in the middle attack on your CI network, the hash check provides protection. If an attacker compromises the server you get packages from, having a hash provides protection as well. Automatically trusting the server's response to be correct, or automatically upgrading to newer versions (even if the package author strictly adheres to SemVer), puts you at risk at attacks similar to what we saw with SolarWinds around 2021.
      - brabel 3y ago
        Your argument supports the idea of getting rid of the lock file and instead committing the hash in the original dependencies file - so that it's never an automatic process to update the hashes/versions.
planede 3y ago
So, how does it handle diamonds in the dependency graph? How does it handle version compatibility? IMO these are the most interesting aspects of a package manager, other stuff are implementation detail. Although there is a mention for semver at the end: > Decide what versioning scheme to use (Probably semver, or something like it/enhancing it with a total order). I wonder if total order is appropriate (assuming the relation <= means compatibility), but it sure simplifies things. Basically throw away the major version, embed that in the name of the package if you have to.
- hgs3 3y ago
  Agreed. Version resolution is the interesting problem. Most package managers use a SAT solver to resolve dependencies. The Dart team has a detailed write up on their SAT-based approach which is worth a read [1]. For contrast, Russ Cox presents an algorithm that doesn't use a SAT solver (intended for Go) [2]. [1] https://github.com/dart-lang/pub/blob/master/doc/solver.md https://github.com/dart-lang/pub/blob/master/doc/solver.md [2] https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs
- nonameiguess 3y ago
  I could be wrong here, but given what sqlite's documentation includes and the fact his spec files don't even include dependencies, I believe sqlite extensions don't depend upon anything except sqlite itself (he says here "most" don't). You'd need to ensure the extensions and sqlite were both compiled against the same libc, but I don't see what this package manager can do about this given the metadata doesn't seem to be available in these Github releases. In fact, going to the repo for the example in his spec file, the reason for the release was to downgrade the build host OS from Ubuntu 22.04 to 20.04 to resolve an issue with some user not being able to run this because of a missing GLIBC symbol. This is an underappreciated problem and why actual distros include everything. If you're going to distribute binary files, you need to ensure they're compatible in many more ways than just the architecture and kernel matching. A complete package manager would include a build system and public registry with builds matching a complete machine triplet (as in, what you'd get by running gcc -dumpmachine). If you consume upstream Github releases as this is doing, they may not have that level of granularity, and in fact, we can see they do not.
peefy 3y ago
Wonderful post, how about using standard OCI to unify products? We have also been working hard recently on a package manager for configuring language KCL, which currently supports Git, OCI, and more.
- palotasb 3y ago
  What do you mean by OCI and KCL?
qwerty456127 3y ago
Why do we need a separate package manager for every programming language and every extensible library/app?
- progx 3y ago
  Need is the wrong word, sometimes it is necessary. Look at npm, long time they not include new features and made less progress. That time yarn was born. pnpm and other followed, cause npm did not have features, security solutions, and other things that people want. In theory we need only one for all, but in reality, that is impossible.
- peterfirefly 3y ago
  We don't, but it is the easiest way to ensure the package manager is adapted to the language and the library ecosystem. It is also easier to add features, remove misfeatures, and fix bugs when the package manager only has to handle a single language. Once we understand package management better -- we don't understand it well yet -- it will likely be feasible to consolidate many package managers into a few omnimanagers.
- angio 3y ago
  In theory nix can solve that, in practice is more ergonomic to leverage a language's native package manager even in nix.
  - qwerty456127 3y ago
    Does using nix still make the sense it's meant to make if you use a language's native package manager?
Eddygandr 3y ago
Bit of a tangent here but what’s a pip/npm/cargo like package manager for C++? For example ‘pip install boost’? I’ve never worked it out for hobby projects and never worked with it commercially
- oblio 3y ago
  There isn't any. There are a few partial ones like conan, etc. And many people in the community are against it, you'll hear stuff ranging from "just use the distro package manager" to "don't use any, make it a single file library". Personally, I'd say all hope is lost.
  - wizzledonker 3y ago
    Just because I’m curious, in what respect is Conan a “partial” package manager? Within the constraints of existing C++ projects and the variance of their build systems, I can’t imagine how to do it differently. In my experience, most of the value of Conan is with creating packages yourself when needed. You can then self-host a Conan remote and have pre-compiled binaries ready development. Having a conan recipes repository with CI that produces binaries for each required build configuration has become a de-facto standard for projects I have worked on
    - oblio 3y ago
      Well, I'll expand. For 95% of Java devs, JavaScript devs, Python devs, if it's Open Source and it isn't on Maven Central, NPM, PyPi, it might as well not exist. The reverse is true, if you have any library or tool worth a damn, it's there.
- wizzledonker 3y ago
  The closest thing we have at the moment is conan[1]. It’s a cross platform package manager that attempts to implement “integrations”, whereby different build systems can consume the packages[2]. This is a big problem with package management in C/C++, there’s no single, standardised build system that most projects use. There isn’t even a standardised compiler! So when hosting your own packages using Conan, often you need to make sure you build your application for three different compilers, for three different platforms. Sometimes (for modern MacOS) also for two different architectures each. If you control the compiler AND build system you can get away with just one package for most cases. This true for Microsoft’s C/C++ package manager, NuGet[3] Historically, the convention has been to use the package manager of the underlying system to install packages, as there are so many different build configurations to worry about when packaging the libraries. The other advantage of using the system package manager is that dependencies (shared libraries) that are common can be shared between many applications, saving space. [1] https://conan.io/ https://conan.io/ [2] https://docs.conan.io/1/creating_packages/toolchains.html https://docs.conan.io/1/creating_packages/toolchains.html [3] https://devblogs.microsoft.com/cppblog/nuget-for-c/ https://devblogs.microsoft.com/cppblog/nuget-for-c/
tpoacher 3y ago
Not a package manager, but I made this somewhat relevant "app" if anyone's interested: https://sr.ht/~tpapastylianou/misc-updater/ https://sr.ht/~tpapastylianou/misc-updater/ It has solved my own pain points elegantly, so happy to share.
po8tin 3y ago
Source control is my package manager. Package managers as we usually think of them are syntax sugar and abstraction for the sake of abstraction Working on a Linux distro that is one unified/generalized/normalized code base (with the help of AI/ML) and a model to sample and establish correct state from memory of the initial code base. One way to think of it is like a game engine with action plans to allocate resources to recreate Firefox, for example. Not compile and run Firefox, but using *alloc() and free(), etc to establish the correct machine state to browse websites after learning what that state is in the abstract from Firefox’s code. My thinking is many of our “truisms” in IT are outdated given modern machine performance and network reliability relative to the 80s and 90s when many of those “truisms” were defined.
- edgyquant 3y ago
  Most package managers work with version control already, they are not solving the same problems. Package managers deal with the building and dependency graph along with delivery of working executables. Version control solves zero of these problems.
- seabass-labrax 3y ago
  If you haven't already researched them, you may want to investigate Plan9 and Erlang, both early systems embodying a distributed and networked philosophy of computing.

163 points by celiktom 3y ago

[dead]

Writing a package manager is not one of the most common programming tasks. After all, there are many out-of-the-box ones available.

3np 3y ago
These copy-pasted one-liners aren't useful. Skip them.
000ooo000 3y ago
Imagine what'd we be using for package management today if package manager >= #2 author said "you know what, one already exists".
- jsunderland323 3y ago
  This is the package manager paradox. The problem is multiple package managers but to address it we build more package managers.
- queuebert 3y ago
  I think we'd all be using 'tar'.

forrestthewoods 3y ago

Wonderful post! I've written a couple of packager manager like things over the years. Although they've all been internal-only which gives some flexibility. > I like the idea of allowing both project and global scope Yikes! Hard no from me. Globals are pure evil. Avoid like the plague. Environment variables too. Kill them all with fire. > what if the user wants to reinstall the packages on another machine or CI server? That’s where the lockfile comes in I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control! That's a rant for another day. Treating the packages folder as source of truth is basically the equivalent I think? > I understand that dropping dependencies altogether may not be something you are ready to accept. Nice. A corollary to this is that if all your packages are internal then you can simply disallow dependency graphs that want different versions of the same package. Simply bump and fix-up all version number dependencies for all packages. The basically means "all packages play nicely on latest version". Which when all the packages live in a monorepo is perfectly plausible. Totally doesn't work for public package managers, but that's a different story.

verdverm 3y ago
It totally works for Go with MVS, which bumps a lot of versions behind the scenes for you.
RajT88 3y ago
> Environment variables too. Kill them all with fire. I feel this way about host file entries. It is shocking how common they remain, and not just in QA/Dev.
nonameiguess 3y ago
> I've typically bypassed the need for a lockfile by simply checking in the dependencies. Dependencies belong in version control! These are binary dependencies. You're checking shared object files into source control? Note that he's also doing this to add functionality to a binary installation of sqlite, which is presumably running somewhere like /usr/bin/sqlite. This isn't a custom application he's developing. The extensions are in his home directory. "Checking them into version control" would entail making his entire filesystem a Git repo. If you're willing to pay whatever IBM charges these days for ClearCase, something like that might actually work. Regular source control like Git, though? I don't think so.
- forrestthewoods 3y ago
  Binaries absolutely should go in version control. The fact that Git is incapable of efficiently supporting that workflow is a separate topic.
  - seabass-labrax 3y ago
    I think you are right. Tarballs with version strings as filenames served from an HTTPD are effectively a poor man's version control. Git commits are immutable and efficient (storing only the delta) for text, but those same benefits could apply to binary files too if only the tooling was better. You may be interested to know that Debian actually does use Git to store binary data, although only when it is not possible to use the source files to generate the binary data directly.

BearhatBeer 3y ago

I think some day rather than the current paradigm, even including declarative package managers and environments and distros, the future will be per-user and even per-app chrooting or jails, or something similar. Apple already uses something like this today. Many people who are smart about information security have one login for shopping and banking and bill pay, one for their business, and one for cruising the web or gaming or whatever. That way a breach of one doesn't necessarily end up pwning the whole system. I only have two accounts, one "serious" and one "off hours" but I still feel better protected than most people.

czscout 3y ago
Isn't this already how Android handles app permissions? Each application runs as its own user for the sake of security. The Application Sandbox is a pretty cool interpretation of the long existing Unix user/group paradigm.
nextaccountic 3y ago
> Apple already uses something like this today Some Linux distros too have things like this but unfortunately there is no buy-in across the ecosystem so "sandboxing" is done in a half-baked way. The problem is when applications in general aren't written with sandboxing in mind, and when you have to choose between apps not working properly or having a leaky sandbox, you will opt for the latter. I wish some big corp bit the bullet and ported hundreds of apps to a new, sandboxed environment in Linux, while attempting to upstream the whole effort. This would necessarily involve things like fully migrating to Wayland (X11 security is awful), only granting filesystem access through distro-sanctioned file pickers (so you need some coordination among Gtk, Qt, and other toolkits), and generally having a deny-by-default policy: first you make secure, then you fix what broke.
- BearhatBeer 3y ago
  Firejail proves you can sandbox most anything, OpenBSD has their pledge and unveil too. I guess there's a gradient there but each program should be written or constrained to only being able to access what it needs ideally. Per-task groups could go a long way toward solving this, they researched it in the '80s even. Just create and destroy groups on the fly to enable processes to access only what they need. Unix is flexible enough to permit experimenting here.

dclowd9901 3y ago

Man, I really wish frontend package management could be this simple, but with a system that ultimately ships bytes over the wire, we have to not only massage the dough with chopsticks (insofar as try to reduce our dependency tree with weak, feckless tools), but also, with the sheer number of transitive dependencies for each package addition, it’s almost impossible to really know what’s going on with your dependencies unless you write plug-in after plug-in.

larusso 3y ago

Awesome post! I worked with a bunch of package managers over the years and one can see that this design got inspired by the hood parts of a few I know. The only design part I don‘t really like is the ‚latest‘ version specifier in the spec file. Which moves the declaration what the latest version is to the hosted location (in the example GitHub via GitHub API) paired with the fact that the checksums are also fetched rather than being part of the spec. This makes no sense for me. The spec needs to be versioned or better the spec is the actual location to declare a release. I kind of understand where the desire comes from to have a floating spec. Makes the publish process easier since one only creates a GitHub release in this case. But I would still argue that explicitly creating a new version of the spec for each version with baked checksums is better. The benefit for me would be that one could create a hash for each spec version for instance and use that internally for instance.

friendzis 3y ago
Probably the simplest version of a "package manager" are git submodules. Pointing submodule to `master` is effectively the same thing as pointing a "real" package manager to `latest`. This is trunk based development, however you implement it. One could easily argue that floating versions are considered harmful for releases outside of development team and should always be pinned, but on the other hand it is hard to argue against support for floating versions in development. As much as I dislike floating versions, I am not aware of any other way to force changes downstream.
- larusso 3y ago
  Brew has the —head flag where one can instruct to build the latest commit from the repo. But the spec/Formular needs to set a head to pull from.

deepsun 3y ago

Sounds like Maven had all this solved many years ago. Yes, it cannot run arbitrary code, like NPM does, it just copies files, but the dependencies and specfiles were there from the beginning.

ojkelly 3y ago
Not running arbitrary code is a feature, that yarn brings back to the js ecosystem. It sounds nice in theory, but it impacts the soundness of the whole system.
layer8 3y ago
Maven didn’t support transitive dependencies from the beginning.

david2ndaccount 3y ago

Honestly pretty strange to write a package manager for sqlite and to use directories + json files to store the data instead of sqlite.

gosenx 3y ago
Great point
gjvc 3y ago
local system log "files" and package databases would be an excellent example of using sqlite
grumblingdev 3y ago
SQL not so good for tree structures. Recursive CTEs are pretty gnarly. Graphdb would be ideal.
- jeremyjh 3y ago
  You want to bring in a dependency to avoid writing one 8 line CTE?
  - christophberger 3y ago
    The number of lines needed is not a good measure for gnarliness. (To be fair, I don't know if gnarliness is measurable at all.)
    - jeremyjh 3y ago
      It isn’t gnarly at all, but recursive CTE are outside the experience of most SQL devs. Here “gnarly” is used to describe “unfamiliar”.
- jbverschoor 3y ago
  SQLite is a pretty darn good solution for any file(set) that needs atomic / consistent writes

mijoharas 3y ago

It's surprisingly to me that no-one has built a asdf style package manager. (I'm not talking about system package managers, their language software packages are always out of date and get installed globally instead of locally to a project). Having a unified interface to a package manager per language that will use the languages registry could be really nice (I guess you'd have some core dependency management functions and agnostic ways to store and download the packages, and plugins per language that would use these common building blocks to actually do stuff?). Another advantage of this could be cross language dependencies which often aren't handled well by package managers.

ljm 3y ago
You could argue that Bazel, Nix and Guix solve that problem but they introduce a lot of complexity as a result because it's not a simple abstraction. Hell, you could probably pull it off somehow in Cmake if you placed no value on your sanity. Most standard lib package managers double up as build tools for bundling your code after all. It's a totally different problem space than the one asdf aims to solve, which is just managing multiple independent versions of some programming language binaries in a consistent way, so you're not dancing between nvm, rvm/rbenv/chruby, virtualenv, etc. etc.
kdeldycke 3y ago
Something like Meta Package Manager? https://github.com/kdeldycke/meta-package-manager https://github.com/kdeldycke/meta-package-manager
otabdeveloper4 3y ago
Nix solves that problem. (While creating some new ones.)

riffraff 3y ago

Isn't this missing the SQLite version? Are SQLite extensions guaranteed to work across different SQLite installs?

matharmin 3y ago
SQLite is quite good at staying backwards compatible, and this includes the APIs exposed to extensions, but extensions could definitely have a minimum SQLite version.

brabel 3y ago

The author added a lockfile without understanding why they exist. A lockfile is meant to "freeze" dependency version resolution when package authors can specify dependencies on other packages using version ranges... it also "freezes" choices of transitive packages' versions when different packages depend on the same one, but with different versions. They chose to not handle package dependencies at all, and I believe there's no version ranges either, so I really don't see why they added a lockfile.

pharmakom 3y ago
A version number is just a label, and labels are mutable. A lock-file containing hashes will always resolve to the same packages (or fail).
- qwerty456127 3y ago
  Depending on a single specific version is wrong. A library can be changed to fix a bug without changing the interface in any breaking way. It should be possible for a user (also their package manager, automatically) to replace the library with the new version in this case.
  - zivkan 3y ago
    Depending on how important supply chain security is to your industry/company/team, validating the hash of every package is critical. If an attacker can manage an interception/man in the middle attack on your CI network, the hash check provides protection. If an attacker compromises the server you get packages from, having a hash provides protection as well. Automatically trusting the server's response to be correct, or automatically upgrading to newer versions (even if the package author strictly adheres to SemVer), puts you at risk at attacks similar to what we saw with SolarWinds around 2021.
    - brabel 3y ago
      Your argument supports the idea of getting rid of the lock file and instead committing the hash in the original dependencies file - so that it's never an automatic process to update the hashes/versions.

planede 3y ago

So, how does it handle diamonds in the dependency graph? How does it handle version compatibility? IMO these are the most interesting aspects of a package manager, other stuff are implementation detail. Although there is a mention for semver at the end: > Decide what versioning scheme to use (Probably semver, or something like it/enhancing it with a total order). I wonder if total order is appropriate (assuming the relation <= means compatibility), but it sure simplifies things. Basically throw away the major version, embed that in the name of the package if you have to.

hgs3 3y ago
Agreed. Version resolution is the interesting problem. Most package managers use a SAT solver to resolve dependencies. The Dart team has a detailed write up on their SAT-based approach which is worth a read [1]. For contrast, Russ Cox presents an algorithm that doesn't use a SAT solver (intended for Go) [2]. [1] https://github.com/dart-lang/pub/blob/master/doc/solver.md https://github.com/dart-lang/pub/blob/master/doc/solver.md [2] https://research.swtch.com/vgo-mvs https://research.swtch.com/vgo-mvs
nonameiguess 3y ago
I could be wrong here, but given what sqlite's documentation includes and the fact his spec files don't even include dependencies, I believe sqlite extensions don't depend upon anything except sqlite itself (he says here "most" don't). You'd need to ensure the extensions and sqlite were both compiled against the same libc, but I don't see what this package manager can do about this given the metadata doesn't seem to be available in these Github releases. In fact, going to the repo for the example in his spec file, the reason for the release was to downgrade the build host OS from Ubuntu 22.04 to 20.04 to resolve an issue with some user not being able to run this because of a missing GLIBC symbol. This is an underappreciated problem and why actual distros include everything. If you're going to distribute binary files, you need to ensure they're compatible in many more ways than just the architecture and kernel matching. A complete package manager would include a build system and public registry with builds matching a complete machine triplet (as in, what you'd get by running gcc -dumpmachine). If you consume upstream Github releases as this is doing, they may not have that level of granularity, and in fact, we can see they do not.

peefy 3y ago

Wonderful post, how about using standard OCI to unify products? We have also been working hard recently on a package manager for configuring language KCL, which currently supports Git, OCI, and more.

palotasb 3y ago
What do you mean by OCI and KCL?

qwerty456127 3y ago

Why do we need a separate package manager for every programming language and every extensible library/app?

progx 3y ago
Need is the wrong word, sometimes it is necessary. Look at npm, long time they not include new features and made less progress. That time yarn was born. pnpm and other followed, cause npm did not have features, security solutions, and other things that people want. In theory we need only one for all, but in reality, that is impossible.
peterfirefly 3y ago
We don't, but it is the easiest way to ensure the package manager is adapted to the language and the library ecosystem. It is also easier to add features, remove misfeatures, and fix bugs when the package manager only has to handle a single language. Once we understand package management better -- we don't understand it well yet -- it will likely be feasible to consolidate many package managers into a few omnimanagers.
angio 3y ago
In theory nix can solve that, in practice is more ergonomic to leverage a language's native package manager even in nix.
- qwerty456127 3y ago
  Does using nix still make the sense it's meant to make if you use a language's native package manager?

Eddygandr 3y ago

Bit of a tangent here but what’s a pip/npm/cargo like package manager for C++? For example ‘pip install boost’? I’ve never worked it out for hobby projects and never worked with it commercially

oblio 3y ago
There isn't any. There are a few partial ones like conan, etc. And many people in the community are against it, you'll hear stuff ranging from "just use the distro package manager" to "don't use any, make it a single file library". Personally, I'd say all hope is lost.
- wizzledonker 3y ago
  Just because I’m curious, in what respect is Conan a “partial” package manager? Within the constraints of existing C++ projects and the variance of their build systems, I can’t imagine how to do it differently. In my experience, most of the value of Conan is with creating packages yourself when needed. You can then self-host a Conan remote and have pre-compiled binaries ready development. Having a conan recipes repository with CI that produces binaries for each required build configuration has become a de-facto standard for projects I have worked on
  - oblio 3y ago
    Well, I'll expand. For 95% of Java devs, JavaScript devs, Python devs, if it's Open Source and it isn't on Maven Central, NPM, PyPi, it might as well not exist. The reverse is true, if you have any library or tool worth a damn, it's there.
wizzledonker 3y ago
The closest thing we have at the moment is conan[1]. It’s a cross platform package manager that attempts to implement “integrations”, whereby different build systems can consume the packages[2]. This is a big problem with package management in C/C++, there’s no single, standardised build system that most projects use. There isn’t even a standardised compiler! So when hosting your own packages using Conan, often you need to make sure you build your application for three different compilers, for three different platforms. Sometimes (for modern MacOS) also for two different architectures each. If you control the compiler AND build system you can get away with just one package for most cases. This true for Microsoft’s C/C++ package manager, NuGet[3] Historically, the convention has been to use the package manager of the underlying system to install packages, as there are so many different build configurations to worry about when packaging the libraries. The other advantage of using the system package manager is that dependencies (shared libraries) that are common can be shared between many applications, saving space. [1] https://conan.io/ https://conan.io/ [2] https://docs.conan.io/1/creating_packages/toolchains.html https://docs.conan.io/1/creating_packages/toolchains.html [3] https://devblogs.microsoft.com/cppblog/nuget-for-c/ https://devblogs.microsoft.com/cppblog/nuget-for-c/

tpoacher 3y ago

Not a package manager, but I made this somewhat relevant "app" if anyone's interested: https://sr.ht/~tpapastylianou/misc-updater/ https://sr.ht/~tpapastylianou/misc-updater/ It has solved my own pain points elegantly, so happy to share.

po8tin 3y ago

Source control is my package manager. Package managers as we usually think of them are syntax sugar and abstraction for the sake of abstraction Working on a Linux distro that is one unified/generalized/normalized code base (with the help of AI/ML) and a model to sample and establish correct state from memory of the initial code base. One way to think of it is like a game engine with action plans to allocate resources to recreate Firefox, for example. Not compile and run Firefox, but using *alloc() and free(), etc to establish the correct machine state to browse websites after learning what that state is in the abstract from Firefox’s code. My thinking is many of our “truisms” in IT are outdated given modern machine performance and network reliability relative to the 80s and 90s when many of those “truisms” were defined.

edgyquant 3y ago
Most package managers work with version control already, they are not solving the same problems. Package managers deal with the building and dependency graph along with delivery of working executables. Version control solves zero of these problems.
seabass-labrax 3y ago
If you haven't already researched them, you may want to investigate Plan9 and Erlang, both early systems embodying a distributed and networked philosophy of computing.