4 ms·

Different package managers make different design decisions that make them incompatible with each other without sacrifice. Guix/Nix focus heavily on reproducibi

Different package managers make different design decisions that make them incompatible with each other without sacrifice. Guix/Nix focus heavily on reproducibility and not relying on any third party binaries. These features would have to be thrown away if it unified pip, npm, etc. because they make no such guarantees. Every package manager works different, and trying to accomadate all of them with a unifying tool will require a lot of time wasted writing interfaces between them all. I don't see a way to do it without lowering the feature set to the least common denominator and settling with that. The real solution is to elevate our system package managers to the point of handling the important use cases that currently only language package managers provide, such as virtualenv/Bundler style management (i.e. installing packages somewhere besides /). Nix/Guix accomodate all such use cases, which is why I promote them.

zokier 12y ago
> These features would have to be thrown away if it unified pip, npm, etc. because they make no such guarantees. Again I disagree. For example in this case, it would just mean that when installing from nix repos you get the reproducibility etc guarantees, installing from upstream repos would get you vanilla version and installing from OS repos would get you an integrated/patched version. But the installation process (from user point of view) could still be unified, because regardless what happens behind the scenes the high-level stuff is pretty much the same; nix -i/apt-get install/pip install could definitely be unified under one umbrella tool. > I don't see a way to do it without lowering the feature set to the least common denominator and settling with that. And the common denominator would probably cover 80-90% of uses. For the rest you'd still have to option to delve deeper and use some implementation-specific tools if necessary.
- davexunit 12y ago
  I just think it's a mistake to believe that the solution to the proliferation of package managers is to write a package manager manager.
  - igravious 12y ago
    I'm not sure that anybody is advocating taking that route as the way forward...
- sparkie 12y ago
  If we've any hope to unify package management, we need to get to the essence of what package management is. It's really quite simple though - it's the ability to say that one piece of software depends upon another, and to have a piece of software which can automatically resolve the dependencies (which form a DAG). To construct our DAG we need a list of nodes (the packages), and a list of edges (the dependencies of a package). If we say that packages are basically just binary blobs of data (say, a .tar.*), then we might construct our database to identify (key) our package payloads. I'll use some pseudo pgsql for illustration purposes. CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, CONSTRAINT pk_package PRIMARY KEY ("package_name") ); CREATE TABLE package_dependency ( dependant character varying NOT NULL, dependency character varying NOT NULL, CONSTRAINT package_dependency_dependant_dependency_key UNIQUE (dependant, dependency) CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant) REFERENCES packages (package_name), CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency) REFERENCES packages (package_name), ); Simple. But we're mising a bit here. We need to update software, so a name is not sufficient to identify a dependency. Lets add that. CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, version integer NOT NULL, CONSTRAINT pk_package PRIMARY KEY (package_name, version) ); CREATE TABLE package_dependency ( dependant character varying NOT NULL, dependency character varying NOT NULL, dependant_version integer NOT NULL, dependency_version integer NOT NULL, CONSTRAINT package_dependency_pkey UNIQUE (dependant, dependency, dependant_version, dependency_version), CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant, dependant_version) REFERENCES packages (package_name, version), CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency, dependency_version) REFERENCES packages (package_name, version) ); Cool, now we have a composite key we've got sufficient information to idenfity a dependency right? Well no, we now have the problem that the same piece of software with the same version could be distributed by different vendors (with different dependency chains/configurations, etc). We had to modify the original solution to get here rather than extend it. Let's modify it again! CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, version integer NOT NULL, vendor character varying NOT NULL, CONSTRAINT pk_package PRIMARY KEY (package_name, version, vendor) ); ... Great, now given a combo of package_name, version and vendor, we can uniquely identify a dependency without worry. All problems solved? What now if Vendor has a customer with different needs, and must distribute two different derivations of the same package version and number? Do we add another field for "configuration", and if so, what type do we make it? Do we just rename the package and lose the relationship that exists between them? It should be blindingly obvious by now that we're just trying to add structure where it isn't really present, and we're making the solution to the problem more and more complicated. Now take into account the possibility that Package Manager A implements dependencies using a tuple of (package_name, version, configuration), and Package Manager B implements dependencies using a tuple of (package_name, version, vendor), then if we want to unify these models under a "one true package manager", then our OTPM needs to model things using ("package_name, version, optional[vendor], optional[configuration]), and so forth. Multiply by N package managers with their own individual quirks and you get a "unified" model which is barely unified at all, the only structure to it is really our initial solution - packages with names. Here's a reduction in complexity: Instead of using a name, which may be ambiguous and therfore requires us to constantly add fields and change the underlying model - let's propose we have some means of creating identifiers with some means of guaranteeing uniqueness. We can basically go back to our initial model: CREATE TABLE packages ( payload bytea NOT NULL, identity uuid NOT NULL, CONSTRAINT pk_packages PRIMARY KEY (unique_id) ); CREATE TABLE package_dependency ( dependant_id uuid NOT NULL, dependency_id uuid NOT NULL, CONSTRAINT package_dependency_dependant_id_dependency_id_key UNIQUE (dependant_id, dependency_id) CONSTRAINT package_dependency_dependant_id_fkey FOREIGN KEY (dependant_id) REFERENCES packages (identity), CONSTRAINT package_dependency_dependency_id_fkey FOREIGN KEY (dependency_id) REFERENCES packages (identity), ); Now, if we want to integrate "Package Manager B" into this new model, we can extend our system. (Note keyword extend, not modify). We can implement a new table which references the base model. CREATE TABLE pmBpackages ( identity uuid NOT NULL, package_name character varying NOT NULL, version integer NOT_NULL, vendor character varying NOT NULL, CONSTRAINT pmBpackages_identity_fkey FOREIGN KEY (identity) REFERENCES packages (identity) ); And as far as pmB is concerned, this is just an implementation detail - we can hide it from any users and just present the legacy view to them. CREATE OR REPLACE VIEW "pm_B_view" AS SELECT package_name, version, vendor, payload FROM "pm_B_packages" NATURAL JOIN packages; So here's a challenge. Begin with the schema for "Package manager A" as implied above, and try to implement "Package manager B" by extension (not modification). The first point of struggle might be to notice that you need to invent "configuration" values, since "Package Manager A" requires them as part of the key, and they're NOT NULL. Hopefully it becomes obvious now why trying to build another model on top of other overcomplicated models is the real fools errand, because none of them so far have understood the essence of the problem. I've glossed over how we might guarantee uniqueness for package identities so far. The solution is to use a cryptographic has of the payload, under the (fair) assumption that a modern hashing algorithm is sufficiently collision resistant. There's nothing language/framework/operating system specific about SHA-1 or whatever the choice of algorithm.

zokier 12y ago

> These features would have to be thrown away if it unified pip, npm, etc. because they make no such guarantees. Again I disagree. For example in this case, it would just mean that when installing from nix repos you get the reproducibility etc guarantees, installing from upstream repos would get you vanilla version and installing from OS repos would get you an integrated/patched version. But the installation process (from user point of view) could still be unified, because regardless what happens behind the scenes the high-level stuff is pretty much the same; nix -i/apt-get install/pip install could definitely be unified under one umbrella tool. > I don't see a way to do it without lowering the feature set to the least common denominator and settling with that. And the common denominator would probably cover 80-90% of uses. For the rest you'd still have to option to delve deeper and use some implementation-specific tools if necessary.

davexunit 12y ago
I just think it's a mistake to believe that the solution to the proliferation of package managers is to write a package manager manager.
- igravious 12y ago
  I'm not sure that anybody is advocating taking that route as the way forward...
sparkie 12y ago
If we've any hope to unify package management, we need to get to the essence of what package management is. It's really quite simple though - it's the ability to say that one piece of software depends upon another, and to have a piece of software which can automatically resolve the dependencies (which form a DAG). To construct our DAG we need a list of nodes (the packages), and a list of edges (the dependencies of a package). If we say that packages are basically just binary blobs of data (say, a .tar.*), then we might construct our database to identify (key) our package payloads. I'll use some pseudo pgsql for illustration purposes. CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, CONSTRAINT pk_package PRIMARY KEY ("package_name") ); CREATE TABLE package_dependency ( dependant character varying NOT NULL, dependency character varying NOT NULL, CONSTRAINT package_dependency_dependant_dependency_key UNIQUE (dependant, dependency) CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant) REFERENCES packages (package_name), CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency) REFERENCES packages (package_name), ); Simple. But we're mising a bit here. We need to update software, so a name is not sufficient to identify a dependency. Lets add that. CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, version integer NOT NULL, CONSTRAINT pk_package PRIMARY KEY (package_name, version) ); CREATE TABLE package_dependency ( dependant character varying NOT NULL, dependency character varying NOT NULL, dependant_version integer NOT NULL, dependency_version integer NOT NULL, CONSTRAINT package_dependency_pkey UNIQUE (dependant, dependency, dependant_version, dependency_version), CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant, dependant_version) REFERENCES packages (package_name, version), CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency, dependency_version) REFERENCES packages (package_name, version) ); Cool, now we have a composite key we've got sufficient information to idenfity a dependency right? Well no, we now have the problem that the same piece of software with the same version could be distributed by different vendors (with different dependency chains/configurations, etc). We had to modify the original solution to get here rather than extend it. Let's modify it again! CREATE TABLE packages ( payload bytea NOT NULL, package_name character varying NOT NULL, version integer NOT NULL, vendor character varying NOT NULL, CONSTRAINT pk_package PRIMARY KEY (package_name, version, vendor) ); ... Great, now given a combo of package_name, version and vendor, we can uniquely identify a dependency without worry. All problems solved? What now if Vendor has a customer with different needs, and must distribute two different derivations of the same package version and number? Do we add another field for "configuration", and if so, what type do we make it? Do we just rename the package and lose the relationship that exists between them? It should be blindingly obvious by now that we're just trying to add structure where it isn't really present, and we're making the solution to the problem more and more complicated. Now take into account the possibility that Package Manager A implements dependencies using a tuple of (package_name, version, configuration), and Package Manager B implements dependencies using a tuple of (package_name, version, vendor), then if we want to unify these models under a "one true package manager", then our OTPM needs to model things using ("package_name, version, optional[vendor], optional[configuration]), and so forth. Multiply by N package managers with their own individual quirks and you get a "unified" model which is barely unified at all, the only structure to it is really our initial solution - packages with names. Here's a reduction in complexity: Instead of using a name, which may be ambiguous and therfore requires us to constantly add fields and change the underlying model - let's propose we have some means of creating identifiers with some means of guaranteeing uniqueness. We can basically go back to our initial model: CREATE TABLE packages ( payload bytea NOT NULL, identity uuid NOT NULL, CONSTRAINT pk_packages PRIMARY KEY (unique_id) ); CREATE TABLE package_dependency ( dependant_id uuid NOT NULL, dependency_id uuid NOT NULL, CONSTRAINT package_dependency_dependant_id_dependency_id_key UNIQUE (dependant_id, dependency_id) CONSTRAINT package_dependency_dependant_id_fkey FOREIGN KEY (dependant_id) REFERENCES packages (identity), CONSTRAINT package_dependency_dependency_id_fkey FOREIGN KEY (dependency_id) REFERENCES packages (identity), ); Now, if we want to integrate "Package Manager B" into this new model, we can extend our system. (Note keyword extend, not modify). We can implement a new table which references the base model. CREATE TABLE pmBpackages ( identity uuid NOT NULL, package_name character varying NOT NULL, version integer NOT_NULL, vendor character varying NOT NULL, CONSTRAINT pmBpackages_identity_fkey FOREIGN KEY (identity) REFERENCES packages (identity) ); And as far as pmB is concerned, this is just an implementation detail - we can hide it from any users and just present the legacy view to them. CREATE OR REPLACE VIEW "pm_B_view" AS SELECT package_name, version, vendor, payload FROM "pm_B_packages" NATURAL JOIN packages; So here's a challenge. Begin with the schema for "Package manager A" as implied above, and try to implement "Package manager B" by extension (not modification). The first point of struggle might be to notice that you need to invent "configuration" values, since "Package Manager A" requires them as part of the key, and they're NOT NULL. Hopefully it becomes obvious now why trying to build another model on top of other overcomplicated models is the real fools errand, because none of them so far have understood the essence of the problem. I've glossed over how we might guarantee uniqueness for package identities so far. The solution is to use a cryptographic has of the payload, under the (fair) assumption that a modern hashing algorithm is sufficiently collision resistant. There's nothing language/framework/operating system specific about SHA-1 or whatever the choice of algorithm.