5 ms·

Packaging Con 2021: a conference for package management devs and communities

108 points by droelf 5y ago

scrollaway 5y ago
Maybe we will finally get some protocol standardization in package distribution. If you think about it it's kind of crazy that we are reinventing the wheel with every language, OS, game store, extension marketplace etc. A lot of Linux enthusiasts get lost in the idea that there should be one and only one package manager. But actually it's more that there should be a standard protocol for package discovery, distribution and updating; and package managers can just be an interface to those.
- Macha 5y ago
  > But actually it's more that there should be a standard protocol for package discovery, distribution and updating; and package managers can just be an interface to those. PackageKit exists as a standard protocol on top of myriad package managers. Most major distro package managers (apt, yum/dnf, apk, pacman) have a package kit wrapper, and UI tools such as Gnome Software or KDE Discover exist that use this abstraction. The missing piece here is that it still requires packages to be built in all the different formats that these abstracted package managers require.
- noja 5y ago
  Exactly. And that one correct true best winning choice can surely only be rpm.
  - magpi3 5y ago
    I always wonder how much the split between .deb and .rpm had to do with the names. They serve the exact same function to the point where there is translation software from one to the other and vice versa, but the names themselves promote the distros they were designed for. Is that the biggest reason for the split? Of course the debian project managed the development of .debs and Red Hat managed the development of rpms, so control of that development path was probably another reason. But this is one split in the linux community that really feels unnecessary.
    - stryan 5y ago
      .deb predates RPM's by a few years, so it's probably more then just the name. While they accomplish the same thing, RPM's are actually pretty different from .deb since they're binary files. There's more metadata encoded in the package themselves, as well as a built in signature. .debs are, at their core, still just fancy tarballs organized in a specific manner. Honestly though, RPM's are kind of a mess and pretty difficult to implement without using one of the reference programs, along with containing a lot of extraneous info that you need to fill out. There's a great blog post out there describing someone's attempt to create RPM's from scratch and their gradual descent into insanity. Though, the extra metadata has it's features (better search, dependency resolution, etc).
      - wichert 5y ago
        It has probably been 20 years since I did anything with deb and rpm, but I think this still holds true: The deb and rpm formats had a very different philosophy to handling dependencies: the deb format relies completely on package names, while the rpm format also allows you depend on the existing of files (or other things?). That implies that the doing/deb ecosystem only works if you manage the entire OS, while rpm (at least theoretically) allows you to use rpms in a system managed in some other way. Of course that capability in the package format itself did not mean people used that. The file formats are also slightly different: dpkg/deb is completely build using tar and ar, while rpm is a custom binary based on cpio. That also made the deb format flexible enough to not have needed any changes in 30 years, which is pretty amazing. The source formats have a lot more differences, but I don’t know enough about those. From what I remember srpm used to be more powerful, but Debian’s source format has caught up by now. A nice side effect of only Debian using dpkg/deb, at least historically, was that when you got a deb file it was guaranteed to work on your Debian system, but for an rpm you could never predict if it would work on your RedHat/SuSe/Mandriva/TurboLinux/etc. system.
- ziggus 5y ago
  There's already a standard that's been in place for ages and ages: ./configure make make install
  - jcelerier 5y ago
    it's not a useful standard if it does not work out-of-the-box on a native windows install
  - lhorie 5y ago
    That's just for building though, isn't it? Package managers these days deal w/ network, registries, version management, transitive dependency resolution, etc. Not to mention Windows.
  - chrisseaton 5y ago
    I'm not sure that's what people mean by package management. For example if a configure script has a dependency on a library... then what? It doesn't help me find and install the right version of that dependency, does it? In my experience it just says 'dependency not installed'.
    - klyrs 5y ago
      I think it's meant as a joke, but for many of us, there's some truth to it. If you're missing a dependency, you'll need to download, configure, make, install it.
      - chrisseaton 5y ago
        > If you're missing a dependency, you'll need to download, configure, make, install it. Right but people want their package managers do this for them. That’s the whole point.
        klyrs 5y ago
        That's what indicates to me that it was intended as a joke, yes.
- HeyImAlex 5y ago
  I think you’re overestimating how generic package management is. They’re very tightly coupled to their respective languages and build systems.
  - scrollaway 5y ago
    I worked on the problem for two years. I'm not overestimating anything; it's just that the wrong targets are being set. Like I said in my post, it's a "single package manager to rule them all" is the wrong goal. What you want is a common protocol, and multiple implementations of it. "I want an app store" = spin up your favourite PackManProtocol server, configure it to your needs, done. The basics of package management are all very similar and can be covered by core use cases. The solution would need to be extensible enough to be usable beyond those. But having restrictions can also be good for an ecosystem (preventing weird-feature creep).
- yissp 5y ago
  http://xkcd.com/927/ http://xkcd.com/927/ sorry, but I had to
  - javajosh 5y ago
    Upvoted because it needs repeating. And also https://xkcd.com/1053/ https://xkcd.com/1053/
DonHopkins 5y ago
I imagine the conference swag consists of branded tote bags, boxes, baggies, bins, pouches, bottles, backpacks, and laundry bags.
- mbesto 5y ago
  All made using YARN!
- yewenjie 5y ago
  Also eggs and wheels, cause Python.
cybrexalpha 5y ago
Using Go in anger for a few years has made me realise just how much a good package management solution adds to an ecosystem, and just how difficult a bad one makes life.
- sieabah 5y ago
  I've been flip flopping between Rust and Go and landed on primarily Rust due to not liking how imports are done by URL in Go. Probably works for Google, but it doesn't make sense for most use cases where you want to handle major version changes.
- prpl 5y ago
  That’s fine when you are working within one language, but many of these people work on packaging across language boundaries (executables and libraries)
- BiteCode_dev 5y ago
  If you don't embed a c lib, or don't need to setup a database.
  - cpach 5y ago
    I can imagine FFI being tricky. But what’s tricky with regards to databases in this context?
    - BiteCode_dev 5y ago
      If your app depends on postgres, the current habits to state in the readme you have to install it and setup it yourself. That's not packaging. Goog packaging deals with that for you.
      - cpach 5y ago
        I’m not sure I follow… If you write an application in Java/Javascript/Python/C/C#/C++/Racket/Common Lisp, isn’t it the same story there? I.e. that you’re expected to provide the DB yourself. If one desires a convenient solution there’s always Firebase, DynamoDB or even hosted PostgreSQL solutions. For local development one can spin up a Docker image for the DBMS.
freemint 5y ago
Interesting that also the Julia Package manager has a presentation.
tmpfs 5y ago
Client implementations typically need to be deeply integrated with the host PL/OS but I think we can create a shared package registry with a protocol that sits on top of IPFS. A single tool for publishing to the registry would push the package to IPFS, get the returned CID (hash of the package) and store the hash with some package meta data in a blockchain. Storing the hash in a blockchain would provide confidence the package has not been tampered with assuming no 51% attack and that client implementations always verify the hash against the package bytes. By making the hash immutable and linked to the content address of the package we have more confidence in the integrity of our packages. Then we add an HTTPS bridge so that client implementations can easily migrate to consuming packages via this new distributed registry. I am exploring this now so if anyone is interested in this kind of distributed language-agnostic package registry drop me a line at muji [at] tmpfs.org.
- tadfisher 5y ago
  Nix is already most of the way there with content-addressed packages. An IPFS project is also in the works using this feature. A blockchain is unnecessary because reproducible packages can be checked with `nix build --rebuild` and stores are signed by default. A bridge is unnecessary because Nix wraps language-specific package managers when building and does not allow network access in its sandbox.
  - tmpfs 5y ago
    Thanks, i will take a closer look at nix!
antman 5y ago
It would be grest if they offered group discounts for Python packaging solutions developers.
- decodebytes 5y ago
  it seems to be pay what you can afford.
BiteCode_dev 5y ago
Great, I have a message about py, -m, --user, shiv and nuitka that I've been repeating for years but really need to reach a broader audience. That's going to be a great talk to give, and 20 minutes is the perfect format for it. Python packaging would be way less painful if people had been given those key informations. Not painless mind you, but from missing a leg to balls itching.

108 points by droelf 5y ago

Maybe we will finally get some protocol standardization in package distribution. If you think about it it's kind of crazy that we are reinventing the wheel with every language, OS, game store, extension marketplace etc. A lot of Linux enthusiasts get lost in the idea that there should be one and only one package manager. But actually it's more that there should be a standard protocol for package discovery, distribution and updating; and package managers can just be an interface to those.

Macha 5y ago
> But actually it's more that there should be a standard protocol for package discovery, distribution and updating; and package managers can just be an interface to those. PackageKit exists as a standard protocol on top of myriad package managers. Most major distro package managers (apt, yum/dnf, apk, pacman) have a package kit wrapper, and UI tools such as Gnome Software or KDE Discover exist that use this abstraction. The missing piece here is that it still requires packages to be built in all the different formats that these abstracted package managers require.
noja 5y ago
Exactly. And that one correct true best winning choice can surely only be rpm.
- magpi3 5y ago
  I always wonder how much the split between .deb and .rpm had to do with the names. They serve the exact same function to the point where there is translation software from one to the other and vice versa, but the names themselves promote the distros they were designed for. Is that the biggest reason for the split? Of course the debian project managed the development of .debs and Red Hat managed the development of rpms, so control of that development path was probably another reason. But this is one split in the linux community that really feels unnecessary.
  - stryan 5y ago
    .deb predates RPM's by a few years, so it's probably more then just the name. While they accomplish the same thing, RPM's are actually pretty different from .deb since they're binary files. There's more metadata encoded in the package themselves, as well as a built in signature. .debs are, at their core, still just fancy tarballs organized in a specific manner. Honestly though, RPM's are kind of a mess and pretty difficult to implement without using one of the reference programs, along with containing a lot of extraneous info that you need to fill out. There's a great blog post out there describing someone's attempt to create RPM's from scratch and their gradual descent into insanity. Though, the extra metadata has it's features (better search, dependency resolution, etc).
    - wichert 5y ago
      It has probably been 20 years since I did anything with deb and rpm, but I think this still holds true: The deb and rpm formats had a very different philosophy to handling dependencies: the deb format relies completely on package names, while the rpm format also allows you depend on the existing of files (or other things?). That implies that the doing/deb ecosystem only works if you manage the entire OS, while rpm (at least theoretically) allows you to use rpms in a system managed in some other way. Of course that capability in the package format itself did not mean people used that. The file formats are also slightly different: dpkg/deb is completely build using tar and ar, while rpm is a custom binary based on cpio. That also made the deb format flexible enough to not have needed any changes in 30 years, which is pretty amazing. The source formats have a lot more differences, but I don’t know enough about those. From what I remember srpm used to be more powerful, but Debian’s source format has caught up by now. A nice side effect of only Debian using dpkg/deb, at least historically, was that when you got a deb file it was guaranteed to work on your Debian system, but for an rpm you could never predict if it would work on your RedHat/SuSe/Mandriva/TurboLinux/etc. system.
ziggus 5y ago
There's already a standard that's been in place for ages and ages: ./configure make make install
- jcelerier 5y ago
  it's not a useful standard if it does not work out-of-the-box on a native windows install
- lhorie 5y ago
  That's just for building though, isn't it? Package managers these days deal w/ network, registries, version management, transitive dependency resolution, etc. Not to mention Windows.
- chrisseaton 5y ago
  I'm not sure that's what people mean by package management. For example if a configure script has a dependency on a library... then what? It doesn't help me find and install the right version of that dependency, does it? In my experience it just says 'dependency not installed'.
  - klyrs 5y ago
    I think it's meant as a joke, but for many of us, there's some truth to it. If you're missing a dependency, you'll need to download, configure, make, install it.
    - chrisseaton 5y ago
      > If you're missing a dependency, you'll need to download, configure, make, install it. Right but people want their package managers do this for them. That’s the whole point.
      - klyrs 5y ago
        That's what indicates to me that it was intended as a joke, yes.
HeyImAlex 5y ago
I think you’re overestimating how generic package management is. They’re very tightly coupled to their respective languages and build systems.
- scrollaway 5y ago
  I worked on the problem for two years. I'm not overestimating anything; it's just that the wrong targets are being set. Like I said in my post, it's a "single package manager to rule them all" is the wrong goal. What you want is a common protocol, and multiple implementations of it. "I want an app store" = spin up your favourite PackManProtocol server, configure it to your needs, done. The basics of package management are all very similar and can be covered by core use cases. The solution would need to be extensible enough to be usable beyond those. But having restrictions can also be good for an ecosystem (preventing weird-feature creep).
yissp 5y ago
http://xkcd.com/927/ http://xkcd.com/927/ sorry, but I had to
- javajosh 5y ago
  Upvoted because it needs repeating. And also https://xkcd.com/1053/ https://xkcd.com/1053/

DonHopkins 5y ago

I imagine the conference swag consists of branded tote bags, boxes, baggies, bins, pouches, bottles, backpacks, and laundry bags.

mbesto 5y ago
All made using YARN!
yewenjie 5y ago
Also eggs and wheels, cause Python.

cybrexalpha 5y ago

Using Go in anger for a few years has made me realise just how much a good package management solution adds to an ecosystem, and just how difficult a bad one makes life.

sieabah 5y ago
I've been flip flopping between Rust and Go and landed on primarily Rust due to not liking how imports are done by URL in Go. Probably works for Google, but it doesn't make sense for most use cases where you want to handle major version changes.
prpl 5y ago
That’s fine when you are working within one language, but many of these people work on packaging across language boundaries (executables and libraries)
BiteCode_dev 5y ago
If you don't embed a c lib, or don't need to setup a database.
- cpach 5y ago
  I can imagine FFI being tricky. But what’s tricky with regards to databases in this context?
  - BiteCode_dev 5y ago
    If your app depends on postgres, the current habits to state in the readme you have to install it and setup it yourself. That's not packaging. Goog packaging deals with that for you.
    - cpach 5y ago
      I’m not sure I follow… If you write an application in Java/Javascript/Python/C/C#/C++/Racket/Common Lisp, isn’t it the same story there? I.e. that you’re expected to provide the DB yourself. If one desires a convenient solution there’s always Firebase, DynamoDB or even hosted PostgreSQL solutions. For local development one can spin up a Docker image for the DBMS.

freemint 5y ago

Interesting that also the Julia Package manager has a presentation.

tmpfs 5y ago

Client implementations typically need to be deeply integrated with the host PL/OS but I think we can create a shared package registry with a protocol that sits on top of IPFS. A single tool for publishing to the registry would push the package to IPFS, get the returned CID (hash of the package) and store the hash with some package meta data in a blockchain. Storing the hash in a blockchain would provide confidence the package has not been tampered with assuming no 51% attack and that client implementations always verify the hash against the package bytes. By making the hash immutable and linked to the content address of the package we have more confidence in the integrity of our packages. Then we add an HTTPS bridge so that client implementations can easily migrate to consuming packages via this new distributed registry. I am exploring this now so if anyone is interested in this kind of distributed language-agnostic package registry drop me a line at muji [at] tmpfs.org.

tadfisher 5y ago
Nix is already most of the way there with content-addressed packages. An IPFS project is also in the works using this feature. A blockchain is unnecessary because reproducible packages can be checked with `nix build --rebuild` and stores are signed by default. A bridge is unnecessary because Nix wraps language-specific package managers when building and does not allow network access in its sandbox.
- tmpfs 5y ago
  Thanks, i will take a closer look at nix!

antman 5y ago

It would be grest if they offered group discounts for Python packaging solutions developers.

decodebytes 5y ago
it seems to be pay what you can afford.

BiteCode_dev 5y ago

Great, I have a message about py, -m, --user, shiv and nuitka that I've been repeating for years but really need to reach a broader audience. That's going to be a great talk to give, and 20 minutes is the perfect format for it. Python packaging would be way less painful if people had been given those key informations. Not painless mind you, but from missing a leg to balls itching.