8 ms·

Debian discusses vendoring again (2021)

115 points by goranmoomin 3y ago

brycewray 3y ago
(2021)
- avgcorrection 3y ago
  This is Debian so default oldness should be expected.
teddyh 3y ago
Discussion at the time: <https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863>
- dang 3y ago
  Thanks! Macroexpanded: Debian discusses vendoring again - https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863 - Jan 2021 (159 comments)
worksonmine 3y ago
I use Debian because it is boring and stable. My core system is small and never changes by design. For anything that needs bells and whistles I either spin up a VM, use another package manager or build from source. Don't try to change Debian, not every package needs to be on every system.
- jampekka 3y ago
  Even unstable is nowadays boring and stable, and it's hard to get into trouble even with experimental! Decade or two ago it was more exciting when apt upgrade on unstable (or testing after a recent stable release) quite regularly got your computer into an unbootable state, the DE fubared or packages into some interlocking dependency mess that could be more or less impossible to solve. Haven't had that excitement many many years now.
  - PedroBatista 3y ago
    Yeah, but I don't know if that's true throughout all the ecosystem. One of the main selling points ( if not THE selling point of Debian ) is the stability and lack of "surprises". If that fails and Debian starts to be perceived as "once was the gold standard, now they are slacking", trust erodes. Even if you're right about 80% of it.
    - jampekka 3y ago
      I didn't mean this as a criticism, but a flippant praise of how well Debian does things. Unstable is for most packages relatively bleeding edge. Of course not e.g. on Arch level, where the excitement is there all the time and the bleeding could be even literal in some use cases. (In Arch this is mitigated somewhat though by the vastly superior packaging system that makes problems easier to fix manually than the awful .deb.)
lmm 3y ago
The Debian model of an artisanal handcrafted build for every library is just not sustainable. Integrating deeply with the language's packaging ecosystem might be one answer, but they did that with Perl to (IMO) not great results, and I don't think there's much appetite for doing that with Javascript (from either side).
- edhelas 3y ago
  Yet its there today and still one of the more reliable packaging system and distro you can find.
  - lmm 3y ago
    I don't know anyone who uses Debian and actually sticks with their packaged apps. The most important programs they're using on their computers will be installed either from third-party repositories, or some other tool (whether that's snap/flatpak/nix/etc., or a language-specific package manager like NPM), or built from source themselves, because Debian's "native" packaged versions of everything are chronically out of date.
    - worksonmine 3y ago
      I daily drive Debian stable and the few packages that aren't from the repositories are by choice. My most important programs are Neovim, Alacritty (for vim keybindings) and Firefox. Before Bookworm I had to build Alacritty from source but that's it and these days they're an apt install away. I build dwm and dmenu as well but they exist in the repos if I didn't like patching them. If I go outside the repositories that's a choice, and I think hard if I really need it on my system, or maybe a VM for isolation makes more sense if I don't trust the developer. If one needs bleeding edge everything maybe Debian isn't the right choice. Why does everything always have to cater to the masses? Let Debian be boring for those of us who love it for what it is.
      - frankjr 3y ago
        > Neovim You're missing almost 1.5 years of bug fixes then (Bookworm+Unstable 0.7.2, Jun 26, 2022; Neovim stable 0.9.4, Oct 9, 2023). LSP support in particular has gotten much better in the meantime. > If one needs bleeding edge everything maybe Debian isn't the right choice. Stable versions are not bleeding edge.
nerdponx 3y ago
This is one of the places that the often-maligned Homebrew package manager does a good job. App dependencies in languages like Python and Lua are fetched manually, with exact filenames and hashes, and installed into an isolated location just for that app. If they turn out to be used by several apps, then they get packaged normally. I wish other package managers did it this way. It would make packaging complicated applications much easier and result in many fewer dependency conflicts, and allows devs to produce more stable software. It's basically static linking but for Python packages. Sometimes you want shared libraries, but for one-off dependencies it's possibly more trouble than it's worth.
- lloydatkinson 3y ago
  Nix/NixOS too
- gcr 3y ago
  I didn't know this about homebrew. Is this only true for installing homebrew python packages? To be honest, I would `brew install python3` or whatever, then manage my own per-project virtual environments with direnv, so I'd bypass all of homebrew's efforts to encapsulate and package python packages...
  - nerdponx 3y ago
    It's at the discretion of the package maintainer, but it's pretty common. Just look at the `Formula.rb` for any standalone app in Python or Lua. Example: https://github.com/Homebrew/homebrew-core/blob/66c42fbd814b3a0c291fb872a569abc441747685/Formula/l/locust.rb#L26-L129 https://github.com/Homebrew/homebrew-core/blob/66c42fbd814b3... I think using per-project venv is still the sensible thing to do, not just in Python but using the equivalent in Lua (hererocks), Ruby (Bundler), etc. This is for system-wide installations of standalone CLI tools and other "apps". However if you do find something on PyPI that is a standalone app, but isn't in Homebrew, you can create an isolated env for it automatically using Pipx https://pypa.github.io/pipx/ https://pypa.github.io/pipx/.
- cogman10 3y ago
  You are basically describing appimage, flatpak, and to a lesser extent, snap and docker. AppImage is the closest analog as it will reuse files across images.
dgrabla 3y ago
My solution: Debian + Flatpak for the few applications you need at the latest version
deleted 3y ago
[deleted]
RedShift1 3y ago
I think this is partially caused by the frankly anemic standard library some languages come with. Take node for example, it tooks years (almost a decade?) for "fetch" to be available and you still can't connect to a database without external libraries, so you need to install those which then bring in their own dependencies because the standard library is so limited...
- masklinn 3y ago
  “Go” is cited as one of the problematic ecosystems and it’s generally considered a “batteries included” language. The problem is not anemic standard libraries, it’s ecosystems where vendoring is common, because will unvendor packages so you use whatever debian ships. Which on one hand is understandable, if they ship a security fix in one of their packages they want every dependent on the system to get that fix, on the other hand there’s no guarantees whatsoever the package works once unvendored.
  - mananaysiempre 3y ago
    > there’s no guarantees whatsoever the package works once unvendored. And it’s precisely the job of the maintainer to ensure that it does, and more generally that all the various things that they ship together not conflict with each other. Assembling a coherent system despite each individual developer’s understandably narrow view—in explicit opposition to that view, if need be,—is and always was a distro maintainer’s job description. Unfortunately, the bug reporting story has become worse over time. GNU packages included a configuration-time override for the bug reporting email to accomodate distributions and other modifications, but in most software today you’ll more often see a hardcoded link to the upstream homepage. And of course people will often just ask Google for the bug tracker address instead. This leads to understandable annoyance on the part of upstream developers. (It still doesn’t make acting against their intent improper in any way, though.)
- loloquwowndueo 3y ago
  Do database drivers belong in a languages standard library? I think not, to be honest.
deleted 3y ago
[deleted]
andrewaylett 3y ago
One difficulty with NPM packaging is that it's traditionally quite hard to work out whether your dependency spec is correct or not. You can either keep bumping the version, or leave it be and hope no-one checks becuase it's hard to test. Shameless self-promotion: https://www.npmjs.com/package/downgrade-build https://www.npmjs.com/package/downgrade-build This duplicates your project in a temporary directory and locks all the direct dependencies to their lowest semver-compatible version. It's not infallible, but it helps. The only problem is that some projects rely on their direct dependencies bumping versions in order to force update transitive dependencies. Which is horrible, and also relatively easy to avoid needing to do by using a tool like Renovate.
- lloydatkinson 3y ago
  Pretty cool! Does it work with PNPM? I’ve mostly fully switched to that because NPM has for many years stood out as one of the worst package managers across ecosystems.
  - andrewaylett 3y ago
    I'm afraid I've not tried it with PNPM, but it's unlikely to work without at least a small tweak as it executes an "npm run <script>" internally. You get to pick the script though.
- philipwhiuk 3y ago
  But the package.json isn't the important bit. The lock file is.
  - andrewaylett 3y ago
    If you're shipping a library, it's the consumers' lock files that are important. You probably want your lockfile to be as up-to-date as it can reasonably be, for testing, but you also (at least in my opinion) should keep your dependency bounds as wide as is correct so you're not forcing dependency upgrades on your consumers. What Renovate recommend (and I concur) is specifying exact versions of development dependencies in your package.json, and broad version specifiers for peer and production dependencies. Then you use Renovate to test and bump the lock file versions, keeping them up-to-date. Downgrade build then helps you ensure that the lower bound on your peer and production dependencies isn't set too low.
rurban 3y ago
Not much happened since: https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa (last modified 2020-12-17) still hundreds of npm packages missing. gsa itself still at 0.0.0 (npm view gsa)
skywhopper 3y ago
This problem in various forms has been around for decades now. In 2007 it was clear to me that Debian style packaging was insufficient for handling anything beyond trivial deployments of any tools or services from the Python, Ruby, and Java ecosystems. Things have only gotten faster-churning and more complex since then. The old model of packages for shared libraries that are updated across the board just isn’t workable. Both from the packagers point of view, because of the busywork, but also from the users’ point of view, since the low package coverage means many things just haven’t been packaged anyway. In my opinion it’s probably fine for Debian to stick to the old model still for the core OS features, but operators need to know when to use something different for their own needs.
- tremon 3y ago
  The old model of packages for shared libraries that are updated across the board just isn’t workable Why is it not workable, other than "developers no longer care about API stability"? Should we uncritically accept the state of the developing world as it is today, or should we strive for something better?
gettodachoppa 3y ago
>"I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it. " What a naive soul, thinking anyone in the JS world cares even remotely about API stability. xyz-1 will become obsolete within 2.5 weeks and the dev will tell the users to deal with it. You can't bring accountability into that culture.
loughnane 3y ago
I’ve only been using Debian b as a daily driver and a home server for the past 4 months, but I love it. I had been on Ubuntu for years but ultimately turned away because of them pushing snap packages. Also I had been curious about Debian and wanted to give it a shot to see if it was too bare bones. Most of the work I do is scientific python with a dash of web dev. The standard packages “just work” for most of my needs, and when I need to venture out for something like Signal or Slack, flatpak has me taken care of. For projects I either use venv or docker, which feels cleaner to me anyway.
- kwk1 3y ago
  > Debian b Just FYI, although the Ubuntu version naming scheme lends itself to being abbreviated to a single letter, this isn't true for Debian: the last three releases were Bookworm, Bullseye, and Buster.
  - checkyoursudo 3y ago
    And yet, saying you use Debian b could plausibly mean any of those anyway. Several of my servers are stuck on Debian b, and I am not 100% sure which b it is.
    - k12sosse 3y ago
      It was a typo on mobile. B know what I mean?
      - loughnane 3y ago
        totally was a typo. i'm on bookworm
giancarlostoro 3y ago
The article is from 2021 btw, the title should be updated.
sumanthvepa 3y ago
This is a huge problem for Linux distributions. I've been thinking of trying to open source a project (it's a web application simulator) that we have internally, that is built using Javascript/NodeJS and Python/Flask. While the mechanics of slapping a GPLv3 license on the code we wanted to publish were fairly straightforward, severe problems arose when we wanted to package it for distribution with an OS like Debian. The application is a web app that uses both Node and Python, and the complexities packaging it were overwhelming. For now we planning on releasing it as a container image. I'm not fan of doing this, but it seemed like a solution we conceivably implement with the time and resources we had available to us.
- zbentley 3y ago
  If you ever revisit that decision, check out FPM. It can shave off a few of the rough edges related to packaging: https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
  - johnny22 3y ago
    maybe im reading it wrong, but it sounds like they wanted it to be able to be included in the distribution and not just make a package for it. If that's the case, fpm wouln't help. They'd still have to split out all their deps and get them packaged and aso hope the deps don't collide with existing deps version wise.
- tupolef 3y ago
  Please just provide a list of dependencies with min/max versions, one for Build and one for Usage. Then provide a detailed example of the config files, a systemd unit file, and maybe some config examples for the 3 main webservers. People will then deal with it. The target for such a project is not a noob user and it's more ressource friendly and it help people to understand how to debug the app and how it works. I'm testing around 10 new tools every week and I hate when the project page only gives me a docker/flatpack/snap solution, I always find a way to install it raw without any indications, and it's painful.
vbezhenar 3y ago
I'd prefer vendoring approach to all packages. I don't care about disk space or traffic, but I care about pinning precise dependency versions. I'd even say, I'd prefer to virtualize every application in a docker manner. That's what I mostly do with server applications using docker or k8s and it works for me. There's no reason it wouldn't work with desktop applications.
- worksonmine 3y ago
  I agree for projects I maintain, where I can bump a package and the dependencies myself. But on a server with 10 services, depending on 10 versions of a badly maintained project that requires an urgent security patch? Not so much, I rather just update a shared lib. It's a matter of scale and resources. The great thing is that there are options, Flatpaks do what you want, no need to demand that Debian does the same.
- tremon 3y ago
  If you don't care about disk space or traffic, does that mean that other people who do care about it are automatically wrong? What about people who care about software security and want to know which vulnerable versions of which software exist on their servers or desktops? here's no reason it wouldn't work with desktop applications Have you tried it with desktop applications?
moomin 3y ago
I had a think about how you could best deal with this and then realised I was probably just designing a worse Nix. I think, unfortunately, the Debian model just can’t scale to the sheer volume of what it’s being asked to do. And, as pointed out by one of the original posters, it’s not clear what benefit creating a common standardised version of lodash buys anyone.
- yxhuvud 3y ago
  Distro specific packagers are broken because the surface area and workload is too high as it is essentially n packages * m distros to maintain. Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way. The latter seems easier to solve as long as there is someone that actually wants to solve it. Currently the language packager maintainers doesn't seem to be interested in a common solution (understandably, as it is a very complex problem).
  - smilliken 3y ago
    > Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way. > The latter seems easier to solve as long as there is someone that actually wants to solve it. This is solved by Nix. Packages of any type can depend on eachother.
pmontra 3y ago
They should just drop gsa. It's a vulnerability scanner. People interested in that and in its web fronted are supposed to have the skill to follow the instructions on GitHub and install those two pieces of software. One has a dockerfile, the other one must be built from source. It can probably run in a socket container with the correct dependencies. Problem solved. I use Debian on my laptop but in no way I'm using their vendored languages or packages except what's needed to run the OS. Everything else moves too fast. It's either language managers or docker containers. What I want from a distro is a solid foundation to run my desktop, editor and browser, then I care about everything else.
- hedora 3y ago
  The article suggests moving it to contrib, and having it download its stuff post-install. That seems completely reasonable. If it uses common npm libraries that don’t break api’s every few months, then those could be moved to stable. I’d rather libraries packaged by debian releases continue to work for more than 30 days than to have them package unsupportable dev packages that bitrots my dev environment every “apt get update”. It’s kind of amusing to me that there are so many comments saying this is unsustainable, given the fact that multiple BSDs maintain a quality bar similar to Debian’s.
- NegativeK 3y ago
  There are a number of benefits that come from letting the distro maintain a piece of software, such as unattended, or even just a unified UI for updates.
  - pmontra 3y ago
    I know, but they are usually months or years behind. If a customer tells me that they are working with Language x.y.z I need to install that version, no matter what my distro gives me. And if a project is N versions ahead of the version in my distro, I usually install from either their own PPA or docker image. If I had one euro for every time I read something along the lines of "the version of our software available in Debian is terribly out of date, do not install it, it's full of bugs that we fixed on the last years plus new features. Follow our installation procedure instead."
phkahler 3y ago
>> If we continue to push back on bundling, we need to improve our tools and processes and policies to make it feasible to maintain unbundled packages. Otherwise, we need to build tools and processes and policies around bundled dependencies. Or neither. The dependency hell created by that approach could be considered a bug with the software created that way - even a blocker. Like hey, your project isn't going to get included until it's not a giant spaghetti pile of dependencies.
- jancsika 3y ago
  But they've already caved for Firefox and Chromium.
- rcxdude 3y ago
  Given users are generally much more attached to applications than distributions, that's unlikely to work out well for the distros.
  - phkahler 3y ago
    It was just another possible way to look at it. I prefer distro-provided packages myself, and won't jump through many hoops beyond that simplicity.
    - eternityforest 3y ago
      I'm not really a fan of dynamic linking and shared packages. It's an idea that developed on hand-maintained systems used to process data and run CLI apps. Snaps and APKs seem to work much better for large apps on installs meant to be replaceable commodites that can always be rebuilt I switched to Ubuntu recently, and I'm very glad I did. The app selection is much larger, and they just work without any fussing.
raverbashing 3y ago
Keeping local versions of every dependency and trying to get around the mess is a 90's/2000's model of dependencies This does not scale Linux distros are too big. They need to determine what to ship and be part of the system and what to have the bigger vendors bundle
ChrisArchitect 3y ago
(2021) Previous discussion: https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863
josephcsible 3y ago
One of my favorite things about Debian's approach is that it provides immunity to the left-pad problem.

115 points by goranmoomin 3y ago

brycewray 3y ago

(2021)

avgcorrection 3y ago
This is Debian so default oldness should be expected.

teddyh 3y ago

Discussion at the time: <https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863>

dang 3y ago
Thanks! Macroexpanded: Debian discusses vendoring again - https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863 - Jan 2021 (159 comments)

worksonmine 3y ago

I use Debian because it is boring and stable. My core system is small and never changes by design. For anything that needs bells and whistles I either spin up a VM, use another package manager or build from source. Don't try to change Debian, not every package needs to be on every system.

jampekka 3y ago
Even unstable is nowadays boring and stable, and it's hard to get into trouble even with experimental! Decade or two ago it was more exciting when apt upgrade on unstable (or testing after a recent stable release) quite regularly got your computer into an unbootable state, the DE fubared or packages into some interlocking dependency mess that could be more or less impossible to solve. Haven't had that excitement many many years now.
- PedroBatista 3y ago
  Yeah, but I don't know if that's true throughout all the ecosystem. One of the main selling points ( if not THE selling point of Debian ) is the stability and lack of "surprises". If that fails and Debian starts to be perceived as "once was the gold standard, now they are slacking", trust erodes. Even if you're right about 80% of it.
  - jampekka 3y ago
    I didn't mean this as a criticism, but a flippant praise of how well Debian does things. Unstable is for most packages relatively bleeding edge. Of course not e.g. on Arch level, where the excitement is there all the time and the bleeding could be even literal in some use cases. (In Arch this is mitigated somewhat though by the vastly superior packaging system that makes problems easier to fix manually than the awful .deb.)

lmm 3y ago

The Debian model of an artisanal handcrafted build for every library is just not sustainable. Integrating deeply with the language's packaging ecosystem might be one answer, but they did that with Perl to (IMO) not great results, and I don't think there's much appetite for doing that with Javascript (from either side).

edhelas 3y ago
Yet its there today and still one of the more reliable packaging system and distro you can find.
- lmm 3y ago
  I don't know anyone who uses Debian and actually sticks with their packaged apps. The most important programs they're using on their computers will be installed either from third-party repositories, or some other tool (whether that's snap/flatpak/nix/etc., or a language-specific package manager like NPM), or built from source themselves, because Debian's "native" packaged versions of everything are chronically out of date.
  - worksonmine 3y ago
    I daily drive Debian stable and the few packages that aren't from the repositories are by choice. My most important programs are Neovim, Alacritty (for vim keybindings) and Firefox. Before Bookworm I had to build Alacritty from source but that's it and these days they're an apt install away. I build dwm and dmenu as well but they exist in the repos if I didn't like patching them. If I go outside the repositories that's a choice, and I think hard if I really need it on my system, or maybe a VM for isolation makes more sense if I don't trust the developer. If one needs bleeding edge everything maybe Debian isn't the right choice. Why does everything always have to cater to the masses? Let Debian be boring for those of us who love it for what it is.
    - frankjr 3y ago
      > Neovim You're missing almost 1.5 years of bug fixes then (Bookworm+Unstable 0.7.2, Jun 26, 2022; Neovim stable 0.9.4, Oct 9, 2023). LSP support in particular has gotten much better in the meantime. > If one needs bleeding edge everything maybe Debian isn't the right choice. Stable versions are not bleeding edge.

nerdponx 3y ago

This is one of the places that the often-maligned Homebrew package manager does a good job. App dependencies in languages like Python and Lua are fetched manually, with exact filenames and hashes, and installed into an isolated location just for that app. If they turn out to be used by several apps, then they get packaged normally. I wish other package managers did it this way. It would make packaging complicated applications much easier and result in many fewer dependency conflicts, and allows devs to produce more stable software. It's basically static linking but for Python packages. Sometimes you want shared libraries, but for one-off dependencies it's possibly more trouble than it's worth.

lloydatkinson 3y ago
Nix/NixOS too
gcr 3y ago
I didn't know this about homebrew. Is this only true for installing homebrew python packages? To be honest, I would `brew install python3` or whatever, then manage my own per-project virtual environments with direnv, so I'd bypass all of homebrew's efforts to encapsulate and package python packages...
- nerdponx 3y ago
  It's at the discretion of the package maintainer, but it's pretty common. Just look at the `Formula.rb` for any standalone app in Python or Lua. Example: https://github.com/Homebrew/homebrew-core/blob/66c42fbd814b3a0c291fb872a569abc441747685/Formula/l/locust.rb#L26-L129 https://github.com/Homebrew/homebrew-core/blob/66c42fbd814b3... I think using per-project venv is still the sensible thing to do, not just in Python but using the equivalent in Lua (hererocks), Ruby (Bundler), etc. This is for system-wide installations of standalone CLI tools and other "apps". However if you do find something on PyPI that is a standalone app, but isn't in Homebrew, you can create an isolated env for it automatically using Pipx https://pypa.github.io/pipx/ https://pypa.github.io/pipx/.
cogman10 3y ago
You are basically describing appimage, flatpak, and to a lesser extent, snap and docker. AppImage is the closest analog as it will reuse files across images.

dgrabla 3y ago

My solution: Debian + Flatpak for the few applications you need at the latest version

deleted 3y ago

[deleted]

RedShift1 3y ago

I think this is partially caused by the frankly anemic standard library some languages come with. Take node for example, it tooks years (almost a decade?) for "fetch" to be available and you still can't connect to a database without external libraries, so you need to install those which then bring in their own dependencies because the standard library is so limited...

masklinn 3y ago
“Go” is cited as one of the problematic ecosystems and it’s generally considered a “batteries included” language. The problem is not anemic standard libraries, it’s ecosystems where vendoring is common, because will unvendor packages so you use whatever debian ships. Which on one hand is understandable, if they ship a security fix in one of their packages they want every dependent on the system to get that fix, on the other hand there’s no guarantees whatsoever the package works once unvendored.
- mananaysiempre 3y ago
  > there’s no guarantees whatsoever the package works once unvendored. And it’s precisely the job of the maintainer to ensure that it does, and more generally that all the various things that they ship together not conflict with each other. Assembling a coherent system despite each individual developer’s understandably narrow view—in explicit opposition to that view, if need be,—is and always was a distro maintainer’s job description. Unfortunately, the bug reporting story has become worse over time. GNU packages included a configuration-time override for the bug reporting email to accomodate distributions and other modifications, but in most software today you’ll more often see a hardcoded link to the upstream homepage. And of course people will often just ask Google for the bug tracker address instead. This leads to understandable annoyance on the part of upstream developers. (It still doesn’t make acting against their intent improper in any way, though.)
loloquwowndueo 3y ago
Do database drivers belong in a languages standard library? I think not, to be honest.

deleted 3y ago

[deleted]

andrewaylett 3y ago

One difficulty with NPM packaging is that it's traditionally quite hard to work out whether your dependency spec is correct or not. You can either keep bumping the version, or leave it be and hope no-one checks becuase it's hard to test. Shameless self-promotion: https://www.npmjs.com/package/downgrade-build https://www.npmjs.com/package/downgrade-build This duplicates your project in a temporary directory and locks all the direct dependencies to their lowest semver-compatible version. It's not infallible, but it helps. The only problem is that some projects rely on their direct dependencies bumping versions in order to force update transitive dependencies. Which is horrible, and also relatively easy to avoid needing to do by using a tool like Renovate.

lloydatkinson 3y ago
Pretty cool! Does it work with PNPM? I’ve mostly fully switched to that because NPM has for many years stood out as one of the worst package managers across ecosystems.
- andrewaylett 3y ago
  I'm afraid I've not tried it with PNPM, but it's unlikely to work without at least a small tweak as it executes an "npm run <script>" internally. You get to pick the script though.
philipwhiuk 3y ago
But the package.json isn't the important bit. The lock file is.
- andrewaylett 3y ago
  If you're shipping a library, it's the consumers' lock files that are important. You probably want your lockfile to be as up-to-date as it can reasonably be, for testing, but you also (at least in my opinion) should keep your dependency bounds as wide as is correct so you're not forcing dependency upgrades on your consumers. What Renovate recommend (and I concur) is specifying exact versions of development dependencies in your package.json, and broad version specifiers for peer and production dependencies. Then you use Renovate to test and bump the lock file versions, keeping them up-to-date. Downgrade build then helps you ensure that the lower bound on your peer and production dependencies isn't set too low.

rurban 3y ago

Not much happened since: https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa (last modified 2020-12-17) still hundreds of npm packages missing. gsa itself still at 0.0.0 (npm view gsa)

skywhopper 3y ago

This problem in various forms has been around for decades now. In 2007 it was clear to me that Debian style packaging was insufficient for handling anything beyond trivial deployments of any tools or services from the Python, Ruby, and Java ecosystems. Things have only gotten faster-churning and more complex since then. The old model of packages for shared libraries that are updated across the board just isn’t workable. Both from the packagers point of view, because of the busywork, but also from the users’ point of view, since the low package coverage means many things just haven’t been packaged anyway. In my opinion it’s probably fine for Debian to stick to the old model still for the core OS features, but operators need to know when to use something different for their own needs.

tremon 3y ago
The old model of packages for shared libraries that are updated across the board just isn’t workable Why is it not workable, other than "developers no longer care about API stability"? Should we uncritically accept the state of the developing world as it is today, or should we strive for something better?

gettodachoppa 3y ago

>"I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it. " What a naive soul, thinking anyone in the JS world cares even remotely about API stability. xyz-1 will become obsolete within 2.5 weeks and the dev will tell the users to deal with it. You can't bring accountability into that culture.

loughnane 3y ago

I’ve only been using Debian b as a daily driver and a home server for the past 4 months, but I love it. I had been on Ubuntu for years but ultimately turned away because of them pushing snap packages. Also I had been curious about Debian and wanted to give it a shot to see if it was too bare bones. Most of the work I do is scientific python with a dash of web dev. The standard packages “just work” for most of my needs, and when I need to venture out for something like Signal or Slack, flatpak has me taken care of. For projects I either use venv or docker, which feels cleaner to me anyway.

kwk1 3y ago
> Debian b Just FYI, although the Ubuntu version naming scheme lends itself to being abbreviated to a single letter, this isn't true for Debian: the last three releases were Bookworm, Bullseye, and Buster.
- checkyoursudo 3y ago
  And yet, saying you use Debian b could plausibly mean any of those anyway. Several of my servers are stuck on Debian b, and I am not 100% sure which b it is.
  - k12sosse 3y ago
    It was a typo on mobile. B know what I mean?
    - loughnane 3y ago
      totally was a typo. i'm on bookworm

giancarlostoro 3y ago

The article is from 2021 btw, the title should be updated.

sumanthvepa 3y ago

This is a huge problem for Linux distributions. I've been thinking of trying to open source a project (it's a web application simulator) that we have internally, that is built using Javascript/NodeJS and Python/Flask. While the mechanics of slapping a GPLv3 license on the code we wanted to publish were fairly straightforward, severe problems arose when we wanted to package it for distribution with an OS like Debian. The application is a web app that uses both Node and Python, and the complexities packaging it were overwhelming. For now we planning on releasing it as a container image. I'm not fan of doing this, but it seemed like a solution we conceivably implement with the time and resources we had available to us.

zbentley 3y ago
If you ever revisit that decision, check out FPM. It can shave off a few of the rough edges related to packaging: https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
- johnny22 3y ago
  maybe im reading it wrong, but it sounds like they wanted it to be able to be included in the distribution and not just make a package for it. If that's the case, fpm wouln't help. They'd still have to split out all their deps and get them packaged and aso hope the deps don't collide with existing deps version wise.
tupolef 3y ago
Please just provide a list of dependencies with min/max versions, one for Build and one for Usage. Then provide a detailed example of the config files, a systemd unit file, and maybe some config examples for the 3 main webservers. People will then deal with it. The target for such a project is not a noob user and it's more ressource friendly and it help people to understand how to debug the app and how it works. I'm testing around 10 new tools every week and I hate when the project page only gives me a docker/flatpack/snap solution, I always find a way to install it raw without any indications, and it's painful.

vbezhenar 3y ago

I'd prefer vendoring approach to all packages. I don't care about disk space or traffic, but I care about pinning precise dependency versions. I'd even say, I'd prefer to virtualize every application in a docker manner. That's what I mostly do with server applications using docker or k8s and it works for me. There's no reason it wouldn't work with desktop applications.

worksonmine 3y ago
I agree for projects I maintain, where I can bump a package and the dependencies myself. But on a server with 10 services, depending on 10 versions of a badly maintained project that requires an urgent security patch? Not so much, I rather just update a shared lib. It's a matter of scale and resources. The great thing is that there are options, Flatpaks do what you want, no need to demand that Debian does the same.
tremon 3y ago
If you don't care about disk space or traffic, does that mean that other people who do care about it are automatically wrong? What about people who care about software security and want to know which vulnerable versions of which software exist on their servers or desktops? here's no reason it wouldn't work with desktop applications Have you tried it with desktop applications?

moomin 3y ago

I had a think about how you could best deal with this and then realised I was probably just designing a worse Nix. I think, unfortunately, the Debian model just can’t scale to the sheer volume of what it’s being asked to do. And, as pointed out by one of the original posters, it’s not clear what benefit creating a common standardised version of lodash buys anyone.

yxhuvud 3y ago
Distro specific packagers are broken because the surface area and workload is too high as it is essentially n packages * m distros to maintain. Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way. The latter seems easier to solve as long as there is someone that actually wants to solve it. Currently the language packager maintainers doesn't seem to be interested in a common solution (understandably, as it is a very complex problem).
- smilliken 3y ago
  > Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way. > The latter seems easier to solve as long as there is someone that actually wants to solve it. This is solved by Nix. Packages of any type can depend on eachother.

pmontra 3y ago

They should just drop gsa. It's a vulnerability scanner. People interested in that and in its web fronted are supposed to have the skill to follow the instructions on GitHub and install those two pieces of software. One has a dockerfile, the other one must be built from source. It can probably run in a socket container with the correct dependencies. Problem solved. I use Debian on my laptop but in no way I'm using their vendored languages or packages except what's needed to run the OS. Everything else moves too fast. It's either language managers or docker containers. What I want from a distro is a solid foundation to run my desktop, editor and browser, then I care about everything else.

hedora 3y ago
The article suggests moving it to contrib, and having it download its stuff post-install. That seems completely reasonable. If it uses common npm libraries that don’t break api’s every few months, then those could be moved to stable. I’d rather libraries packaged by debian releases continue to work for more than 30 days than to have them package unsupportable dev packages that bitrots my dev environment every “apt get update”. It’s kind of amusing to me that there are so many comments saying this is unsustainable, given the fact that multiple BSDs maintain a quality bar similar to Debian’s.
NegativeK 3y ago
There are a number of benefits that come from letting the distro maintain a piece of software, such as unattended, or even just a unified UI for updates.
- pmontra 3y ago
  I know, but they are usually months or years behind. If a customer tells me that they are working with Language x.y.z I need to install that version, no matter what my distro gives me. And if a project is N versions ahead of the version in my distro, I usually install from either their own PPA or docker image. If I had one euro for every time I read something along the lines of "the version of our software available in Debian is terribly out of date, do not install it, it's full of bugs that we fixed on the last years plus new features. Follow our installation procedure instead."

phkahler 3y ago

>> If we continue to push back on bundling, we need to improve our tools and processes and policies to make it feasible to maintain unbundled packages. Otherwise, we need to build tools and processes and policies around bundled dependencies. Or neither. The dependency hell created by that approach could be considered a bug with the software created that way - even a blocker. Like hey, your project isn't going to get included until it's not a giant spaghetti pile of dependencies.

jancsika 3y ago
But they've already caved for Firefox and Chromium.
rcxdude 3y ago
Given users are generally much more attached to applications than distributions, that's unlikely to work out well for the distros.
- phkahler 3y ago
  It was just another possible way to look at it. I prefer distro-provided packages myself, and won't jump through many hoops beyond that simplicity.
  - eternityforest 3y ago
    I'm not really a fan of dynamic linking and shared packages. It's an idea that developed on hand-maintained systems used to process data and run CLI apps. Snaps and APKs seem to work much better for large apps on installs meant to be replaceable commodites that can always be rebuilt I switched to Ubuntu recently, and I'm very glad I did. The app selection is much larger, and they just work without any fussing.

raverbashing 3y ago

Keeping local versions of every dependency and trying to get around the mess is a 90's/2000's model of dependencies This does not scale Linux distros are too big. They need to determine what to ship and be part of the system and what to have the bigger vendors bundle

ChrisArchitect 3y ago

(2021) Previous discussion: https://news.ycombinator.com/item?id=25758863 https://news.ycombinator.com/item?id=25758863

josephcsible 3y ago

One of my favorite things about Debian's approach is that it provides immunity to the left-pad problem.