8 ms·

Apt 2.0

91 points by jcastro 7y ago

techntoke 7y ago
After working with Debian/Ubuntu packaging, and switching to Arch and Pacman, it is so much easier to create/maintain packages on Arch and Alpine distros than the alternatives.
- taeric 7y ago
  My age has given me too much cynicism. Green field is always easier. Usually does a ton less, though. In some cases, shedding requirements is good. In most, it is just a race to feature parity. :(
  - linsomniac 7y ago
    Perhaps, but as a counter example I'd compare it to RPM. RPM is a solid, mature, heavily used package format. And the .spec file I've always found to be so much easier to work with than Debian packaging. I've made many debs, and my current work uses debs for deploying all our software, but I still find it so much harder to make them. I have a workflow for my current packages, but if I want to take a new piece of software and turn it into a .deb, it is always pretty painful. I usually have to ask a debian packager for help. They are always super nice and helpful, but I just wish I could package things myself. Seems like if you package .debs all the time, it becomes easier. But as an infrequent packager, I find RPMs .spec to be much easier to manage than Debian's system.
    - kelnos 7y ago
      I agree; I do .deb packaging just infrequently enough that every time I go to do it, I have to re-learn it nearly from scratch. And it's not simple at all.
      - yjftsjthsd-h 7y ago
        I'm fond of FPM for this kind of thing. It provides a usable interface over a bunch of package formats and explicitly aims to make the whole thing painless, and in my experience it succeeds.
    - taeric 7y ago
      I suspect it is easier, per package. Is it easier for all of them, though? An odd spot where I suspect it is not a simple summation of effort.
    - CameronNemo 7y ago
      I think spec files are asinine. Throwing it all in one specially formatted file and using weird macros to define the build steps. Deb is terrible too, but mostly just because they split everything up too much. The actual file formats are nice to work with. E.g. the rules file is just a make script and the install files are lists of files to install and their destinations. Ports systems like alpine/arch/void are nice, but they also handle less (except for void maybe, that ports system has built in support for a lot of different build systems, so packaging a cmake or meson project is a breeze).
    - Conan_Kudo 7y ago
      I prefer RPM packaging over all major styles of dsc (Debian Source Control) packaging as well, but... It's actually possible to use RPM spec files to build Debian packages: https://github.com/debbuild/debbuild https://github.com/debbuild/debbuild And I've been going through and slowly porting much of the RPM ecosystem common macros to run on debbuild so that spec files can be reused to comply with Debian packaging policies as much as possible: https://github.com/debbuild/debbuild-macros https://github.com/debbuild/debbuild-macros (Disclosure: I'm the current maintainer of debbuild)
- LessDmesg 7y ago
  After using Ubuntu then Debian, and briefly trying Arch, it is so much easier to manage packages on Gentoo than the alternatives. USE flags, package.mask and the slots system are indispensable. And you get freedom from systemd.
nrclark 7y ago
Cool! Does anybody know if there are plans to fix the .deb size limit? It's a bummer that .debs can't be more than 10 GB. That number seems big, but I've hit it before when packaging custom toolchains for internal use at my company.
- teruakohatu 7y ago
  Can't you just create a dependancy and split it up?
  - nrclark 7y ago
    Definitely possible in theory. Kind of a pain in practice though. I was working on trying to package Xilinx Vivado for internal use on my company's build servers. It's around a 20-25GB installation, mostly of smallish files. I could have definitely manually split the package up. It's the kind of thing that's hard to do algorithmically though. After a day or so of trying to solve the packaging problem, I eventually gave up on the idea altogether. Large software installs are still fairly uncommon on Linux, but I see more and more of them in the wild. Especially when I look at some games and stuff - I have a bunch of software that's larger than 10 GB. It's kind of nuts that the .deb package format is still so constrained, especially considering its importance to so many distros.
Aloha 7y ago
This item: "The apt(8) command no longer accepts regular expressions or wildcards as package arguments, use patterns (see New Features)." Seems like a pretty big breaking change.
- systemvoltage 7y ago
  Also feels like a regression. What's wrong with keeping that option? Regex is universal way of matching strings.
  - tssva 7y ago
    Apt patterns allows matching on criteria besides strings such as searching for all packages with broken dependencies. It also allows combining different search criteria in various manners which would be difficult or impossible using just a regex search. If you want to just match packages using a regex against package names that is still possible using the name apt pattern. I would suggest reading the apt-pattern man page referenced before passing judgement.
    - systemvoltage 7y ago
      > difficult or impossible using just a regex search I am sorry but this isn't true. I can prove it. Regex is used to parse the program itself that runs APT, this is called Lexical analysis[1]. GNU Bison and Flex [2] are lexical analyzers for example that compile a lot of code we write today. So whatever pattern matching code was written in C as part of the APT 2.0 package, that code itself is read and compiled by regex. So, it provably cannot be impossible to do a match using this new feature you're referring to that cannot be matched using plain old regex. Difficulty is subjective depending on who the target demographic is. All these things are actually moot - it doesn't cost you anything to keep Regex option to search for some package names, perhaps a bit of additional code to maintain. Also, other means of pattern matching can be added in addition to regex search feature. Why not both? [1] https://en.wikipedia.org/wiki/Lexical_analysis https://en.wikipedia.org/wiki/Lexical_analysis [2] https://aquamentus.com/flex_bison.html https://aquamentus.com/flex_bison.html
sneak 7y ago
Does it still redownload 30MB and spin for 10s when you (or an installer script) do “apt update” a few seconds after the last invocation of same instead of comparing a root hash or some other sane method?
- throwaway8941 7y ago
  Check your repository and/or apt settings. apt has had support for pdiff for what, two decades now? https://debian-administration.org/article/439/Avoiding_slow_package_updates_with_package_diffs https://debian-administration.org/article/439/Avoiding_slow_... I am no fan of it though, dnf is just so much better in all respects.
  - sneak 7y ago
    I’m using archive.ubuntu.com and whatever the default apt settings in Ubuntu LTS are. I feel like a ton of debian-style packaging is simply “this is the way we’ve always done it”. debmirror, for example, is hot garbage.
    - pas 7y ago
      Debian was never about ergonomics :( I remember checking their packaging howto every few years to maybe really/properly/truly understand it, but just the sheer uselessness of the text signaled each time that it's just not worth it. Just google whatever you want copy from stackoverflow, and be done with it, don't try to understand it. (Eg. if I wanted a virtual package that provides some package so I can fake that so it won't get pulled in as useless dependency for other packages.)
      - sneak 7y ago
        I would absolutely love to maintain a metapackage and some tools on a personal apt mirror that I can add to machines and update periodically. The burden/overhead of learning the ridiculously tradition-based and overcomplicated system in use has kept me from doing it for years.
Avamander 7y ago
I wonder when will apt will start supporting specifying packages per repository. It's not safe how any package could come from any repository by-default, unless annoying effort is spent manually pinning packages.
- JoshTriplett 7y ago
  If you install a package, that package can do arbitrarily bad things to your system; a repository you don't trust is never "safe".
  - asdfasgasdgasdg 7y ago
    I can think of at least one vector. Suppose a particular library announces one generic repository as their approved distribution channel. Suppose the repo does not do any vetting on what gets uploaded. In this circumstance, you'd only want to pull that particular package from the repo, not any arbitrary package. So I think the trust model bears scoping. It's not all or nothing.
    - geofft 7y ago
      Is there any apt repo software that permits multiple parties to upload to a repo but also enforces that people can't upload new versions of other people's packages? As it happens, Debian itself kinda does this (Debian "maintainers" can only upload to specific package names), but I don't think anyone else runs that software. The usual third-party repo tools like reprepro and aptly don't support it, as far as I know. And sites like Launchpad or OBS just set up a separate apt repo per account (or even multiple apt repos per account), because doing that is very easy. In other words - yes, the trust model you propose is coherent, but I don't think anyone actually does that, because there's a more straightforward option already.
      - andrewshadura 7y ago
        Actually, both reprepro and aptly support this.
CoffeeDregs 7y ago
Argh. Was just upgrading a Debian machine and was wishing (again) for parallel downloads. apt-fast is a severely limited hack (I couldn't figure out what it actually supports). I assumed any significant apt update would include performance/DL improvement. No?
- julian-klode 7y ago
  Non-parallel downloads from the same server are by design - turning that restriction off would be an easy thing. Server resources are limited, and people should not be cheating their way around bandwidth limits the server has. Put multiple mirrors in a mirrors.list, and then use `mirror+file://path to mirror list` instead of the http source in sources.list so that apt downloads from all these mirrors in parallel. Be aware that high parallelisation of downloads may reduce your throughput for small number of things to fetch. There also should be no latency need - APT keeps the number of requests it has sent to the server at 10, so there should be no latency overhead. Yes I know, this does not work for Google because their latency is crazy high, but their speed is super high too.
m0zg 7y ago
Missed opportunity to switch to zstd IMO. Much faster than gzip, especially during decompression, and essentially the same compression ratio. Them apt upgrades take too long.
- anttisalmela 7y ago
  apt has supported zstd since 1.6 already.
  - m0zg 7y ago
    TIL. Ubuntu needs to get it act together then, it sounds like. I think they're using lzma, which is basically the slowest format available.
    - andrewshadura 7y ago
      Ubuntu were the first who started using zstd, only a bit later it was ported to Debian.
    - julian-klode 7y ago
      Last evaluation has shown that the size increase of zstd compared to xz warrants an introduction of delta debs first, so that people don't have to download more. But then with deltas, mirror sizes grows even more, which is a hard sell. Arch Linux gets away with using zstd -21 but that's not practical for general purpose Linux distributions, as it has significant memory requirements compared to xz. Also the speedup in practice is negligent - while zstd is much faster than xz, most of the time installing packages is actually spent in fsync(). So you only see huge speedups when run in eatmydata.
      - julian-klode 7y ago
        Like, zstd -21 --ultra requires a whopping 365MB of RAM to compress a 16 MiB file, zstd -19 requires 12 MB, and xz requires 10 MB. Decompressing -21 requires 20MB vs 10MB for -19 and xz.
cosmiccatnap 7y ago
I know they are kind of just wrapping aptitude but it's frustrating I can't just wildcard for things. It's a package manager not a development framework, why would regex and wildcard not be enough for anyone?
- julian-klode 7y ago
  We could reinstate wildcards (but not regexes, they are unsafe as their magic characters overlap with valid package names, so g++ can mean the package g++, the package g+, or any package matching the regex g+). I feel like once people got used to patterns, that will be sufficient though, and the error reporting gets nicer.

91 points by jcastro 7y ago

After working with Debian/Ubuntu packaging, and switching to Arch and Pacman, it is so much easier to create/maintain packages on Arch and Alpine distros than the alternatives.

taeric 7y ago
My age has given me too much cynicism. Green field is always easier. Usually does a ton less, though. In some cases, shedding requirements is good. In most, it is just a race to feature parity. :(
- linsomniac 7y ago
  Perhaps, but as a counter example I'd compare it to RPM. RPM is a solid, mature, heavily used package format. And the .spec file I've always found to be so much easier to work with than Debian packaging. I've made many debs, and my current work uses debs for deploying all our software, but I still find it so much harder to make them. I have a workflow for my current packages, but if I want to take a new piece of software and turn it into a .deb, it is always pretty painful. I usually have to ask a debian packager for help. They are always super nice and helpful, but I just wish I could package things myself. Seems like if you package .debs all the time, it becomes easier. But as an infrequent packager, I find RPMs .spec to be much easier to manage than Debian's system.
  - kelnos 7y ago
    I agree; I do .deb packaging just infrequently enough that every time I go to do it, I have to re-learn it nearly from scratch. And it's not simple at all.
    - yjftsjthsd-h 7y ago
      I'm fond of FPM for this kind of thing. It provides a usable interface over a bunch of package formats and explicitly aims to make the whole thing painless, and in my experience it succeeds.
  - taeric 7y ago
    I suspect it is easier, per package. Is it easier for all of them, though? An odd spot where I suspect it is not a simple summation of effort.
  - CameronNemo 7y ago
    I think spec files are asinine. Throwing it all in one specially formatted file and using weird macros to define the build steps. Deb is terrible too, but mostly just because they split everything up too much. The actual file formats are nice to work with. E.g. the rules file is just a make script and the install files are lists of files to install and their destinations. Ports systems like alpine/arch/void are nice, but they also handle less (except for void maybe, that ports system has built in support for a lot of different build systems, so packaging a cmake or meson project is a breeze).
  - Conan_Kudo 7y ago
    I prefer RPM packaging over all major styles of dsc (Debian Source Control) packaging as well, but... It's actually possible to use RPM spec files to build Debian packages: https://github.com/debbuild/debbuild https://github.com/debbuild/debbuild And I've been going through and slowly porting much of the RPM ecosystem common macros to run on debbuild so that spec files can be reused to comply with Debian packaging policies as much as possible: https://github.com/debbuild/debbuild-macros https://github.com/debbuild/debbuild-macros (Disclosure: I'm the current maintainer of debbuild)
LessDmesg 7y ago
After using Ubuntu then Debian, and briefly trying Arch, it is so much easier to manage packages on Gentoo than the alternatives. USE flags, package.mask and the slots system are indispensable. And you get freedom from systemd.

nrclark 7y ago

Cool! Does anybody know if there are plans to fix the .deb size limit? It's a bummer that .debs can't be more than 10 GB. That number seems big, but I've hit it before when packaging custom toolchains for internal use at my company.

teruakohatu 7y ago
Can't you just create a dependancy and split it up?
- nrclark 7y ago
  Definitely possible in theory. Kind of a pain in practice though. I was working on trying to package Xilinx Vivado for internal use on my company's build servers. It's around a 20-25GB installation, mostly of smallish files. I could have definitely manually split the package up. It's the kind of thing that's hard to do algorithmically though. After a day or so of trying to solve the packaging problem, I eventually gave up on the idea altogether. Large software installs are still fairly uncommon on Linux, but I see more and more of them in the wild. Especially when I look at some games and stuff - I have a bunch of software that's larger than 10 GB. It's kind of nuts that the .deb package format is still so constrained, especially considering its importance to so many distros.

Aloha 7y ago

This item: "The apt(8) command no longer accepts regular expressions or wildcards as package arguments, use patterns (see New Features)." Seems like a pretty big breaking change.

systemvoltage 7y ago
Also feels like a regression. What's wrong with keeping that option? Regex is universal way of matching strings.
- tssva 7y ago
  Apt patterns allows matching on criteria besides strings such as searching for all packages with broken dependencies. It also allows combining different search criteria in various manners which would be difficult or impossible using just a regex search. If you want to just match packages using a regex against package names that is still possible using the name apt pattern. I would suggest reading the apt-pattern man page referenced before passing judgement.
  - systemvoltage 7y ago
    > difficult or impossible using just a regex search I am sorry but this isn't true. I can prove it. Regex is used to parse the program itself that runs APT, this is called Lexical analysis[1]. GNU Bison and Flex [2] are lexical analyzers for example that compile a lot of code we write today. So whatever pattern matching code was written in C as part of the APT 2.0 package, that code itself is read and compiled by regex. So, it provably cannot be impossible to do a match using this new feature you're referring to that cannot be matched using plain old regex. Difficulty is subjective depending on who the target demographic is. All these things are actually moot - it doesn't cost you anything to keep Regex option to search for some package names, perhaps a bit of additional code to maintain. Also, other means of pattern matching can be added in addition to regex search feature. Why not both? [1] https://en.wikipedia.org/wiki/Lexical_analysis https://en.wikipedia.org/wiki/Lexical_analysis [2] https://aquamentus.com/flex_bison.html https://aquamentus.com/flex_bison.html

sneak 7y ago

Does it still redownload 30MB and spin for 10s when you (or an installer script) do “apt update” a few seconds after the last invocation of same instead of comparing a root hash or some other sane method?

throwaway8941 7y ago
Check your repository and/or apt settings. apt has had support for pdiff for what, two decades now? https://debian-administration.org/article/439/Avoiding_slow_package_updates_with_package_diffs https://debian-administration.org/article/439/Avoiding_slow_... I am no fan of it though, dnf is just so much better in all respects.
- sneak 7y ago
  I’m using archive.ubuntu.com and whatever the default apt settings in Ubuntu LTS are. I feel like a ton of debian-style packaging is simply “this is the way we’ve always done it”. debmirror, for example, is hot garbage.
  - pas 7y ago
    Debian was never about ergonomics :( I remember checking their packaging howto every few years to maybe really/properly/truly understand it, but just the sheer uselessness of the text signaled each time that it's just not worth it. Just google whatever you want copy from stackoverflow, and be done with it, don't try to understand it. (Eg. if I wanted a virtual package that provides some package so I can fake that so it won't get pulled in as useless dependency for other packages.)
    - sneak 7y ago
      I would absolutely love to maintain a metapackage and some tools on a personal apt mirror that I can add to machines and update periodically. The burden/overhead of learning the ridiculously tradition-based and overcomplicated system in use has kept me from doing it for years.

Avamander 7y ago

I wonder when will apt will start supporting specifying packages per repository. It's not safe how any package could come from any repository by-default, unless annoying effort is spent manually pinning packages.

JoshTriplett 7y ago
If you install a package, that package can do arbitrarily bad things to your system; a repository you don't trust is never "safe".
- asdfasgasdgasdg 7y ago
  I can think of at least one vector. Suppose a particular library announces one generic repository as their approved distribution channel. Suppose the repo does not do any vetting on what gets uploaded. In this circumstance, you'd only want to pull that particular package from the repo, not any arbitrary package. So I think the trust model bears scoping. It's not all or nothing.
  - geofft 7y ago
    Is there any apt repo software that permits multiple parties to upload to a repo but also enforces that people can't upload new versions of other people's packages? As it happens, Debian itself kinda does this (Debian "maintainers" can only upload to specific package names), but I don't think anyone else runs that software. The usual third-party repo tools like reprepro and aptly don't support it, as far as I know. And sites like Launchpad or OBS just set up a separate apt repo per account (or even multiple apt repos per account), because doing that is very easy. In other words - yes, the trust model you propose is coherent, but I don't think anyone actually does that, because there's a more straightforward option already.
    - andrewshadura 7y ago
      Actually, both reprepro and aptly support this.

CoffeeDregs 7y ago

Argh. Was just upgrading a Debian machine and was wishing (again) for parallel downloads. apt-fast is a severely limited hack (I couldn't figure out what it actually supports). I assumed any significant apt update would include performance/DL improvement. No?

julian-klode 7y ago
Non-parallel downloads from the same server are by design - turning that restriction off would be an easy thing. Server resources are limited, and people should not be cheating their way around bandwidth limits the server has. Put multiple mirrors in a mirrors.list, and then use `mirror+file://path to mirror list` instead of the http source in sources.list so that apt downloads from all these mirrors in parallel. Be aware that high parallelisation of downloads may reduce your throughput for small number of things to fetch. There also should be no latency need - APT keeps the number of requests it has sent to the server at 10, so there should be no latency overhead. Yes I know, this does not work for Google because their latency is crazy high, but their speed is super high too.

m0zg 7y ago

Missed opportunity to switch to zstd IMO. Much faster than gzip, especially during decompression, and essentially the same compression ratio. Them apt upgrades take too long.

anttisalmela 7y ago
apt has supported zstd since 1.6 already.
- m0zg 7y ago
  TIL. Ubuntu needs to get it act together then, it sounds like. I think they're using lzma, which is basically the slowest format available.
  - andrewshadura 7y ago
    Ubuntu were the first who started using zstd, only a bit later it was ported to Debian.
  - julian-klode 7y ago
    Last evaluation has shown that the size increase of zstd compared to xz warrants an introduction of delta debs first, so that people don't have to download more. But then with deltas, mirror sizes grows even more, which is a hard sell. Arch Linux gets away with using zstd -21 but that's not practical for general purpose Linux distributions, as it has significant memory requirements compared to xz. Also the speedup in practice is negligent - while zstd is much faster than xz, most of the time installing packages is actually spent in fsync(). So you only see huge speedups when run in eatmydata.
    - julian-klode 7y ago
      Like, zstd -21 --ultra requires a whopping 365MB of RAM to compress a 16 MiB file, zstd -19 requires 12 MB, and xz requires 10 MB. Decompressing -21 requires 20MB vs 10MB for -19 and xz.

cosmiccatnap 7y ago

I know they are kind of just wrapping aptitude but it's frustrating I can't just wildcard for things. It's a package manager not a development framework, why would regex and wildcard not be enough for anyone?

julian-klode 7y ago
We could reinstate wildcards (but not regexes, they are unsafe as their magic characters overlap with valid package names, so g++ can mean the package g++, the package g+, or any package matching the regex g+). I feel like once people got used to patterns, that will be sufficient though, and the error reporting gets nicer.