5 ms·

This might be the most mundane topic Ive found myself naturally extremely excited about. If your a developer working full time in only one or two languages you

This might be the most mundane topic Ive found myself naturally extremely excited about. If your a developer working full time in only one or two languages you may never experience just how good/bad you have it. When you do, its really eye opening. Every time I transition to a new language professionally it can be like opening a bag of Bertie Bott's Every Flavor Beans when you look into the packaging story. * Go binary release story is great but the gopath method for dependencies is annoying * Elixir has lockfiles and built-in package docs but the release story deviates too much * Javascript now that everything has settled into npm is a delight but the lack of stdlib, painful local aliasing and extremely heavy node_modules folder can be offputting * Python just sucks (lets hope poetry can bring the promised land of deterministic builds)

seabass-labrax 3y ago
The Python world won't improve as long as programmers add dependencies on libraries written in other languages and (here's the important part) attempt to compile those packages themselves within a Python build process. Poetry is a nice chapter in the Python package definition story, but it is only a tentative step to fixing the wider problem.
bluGill 3y ago
I firmly believe that languages should not manage packages. While it makes the simple cases easy for beginners, the trade off is mixing languages becomes harder. There is no perfect language and often mixing should be the right answer and we don't want any more friction there.
- seabass-labrax 3y ago
  I quite agree! And would it even be harder for beginners? When trying a new a program written in a language I don't use frequently, I usually spend a good half-hour working out what command to run to install the dependencies and going through the logs working out what implicit requirement wasn't in the README. A holistic package, even one written for a different software distribution than the one I use (Debian at present), would immediately get me 99% of the way there. PS. If you find yourself in the same situation I do, Repology is your friend: https://repology.org/ https://repology.org/
  - borodi 3y ago
    I do believe this is an issue of not having explicit dependencies. Julia takes the approach of, we build and ship everything for every OS, which means Pkg (the package manager) knows about binary dependencies as well. Making things more reproducible in language
    - pxc 3y ago
      Linux distros often do things to force packages to declare all their dependencies: Nix and Guix use unique prefixes and sandboxed builds, openSUSE builds all their packages in blank slate VMs that only install what is declared, standard Fedora tools for running builds in minimal chroot environments, etc. I'm not aware of any language ecosystem package managers taking similar measures to ensure that dependency declarations in their packages are complete.
- maple3142 3y ago
  The problem with system packaging is there are so many systems. For example if you only package something for Debian, how should a Fedora, Arch, Gentoo or even Mac and Windows user use that?
  - seabass-labrax 3y ago
    Debian has by far the most rigorous packaging standards of that list, so subsequent packaging for the other distributions should not be very difficult. Bundling the dependencies into an OCI container or AppImage to run them on non-Debian systems is also trivial once you have a Debian package, but of course that comes with disadvantages. Neither Mac nor Windows have a proper first-party package manager (although winget makes some inroads, thanks of course to Keivan Beigi), so comparison becomes rather a moot point for those platforms. [1]: https://keivan.io/the-day-appget-died/ https://keivan.io/the-day-appget-died/
    - nightpool 3y ago
      "so comparison becomes rather a moot point for those platforms." No, because the OP was comparing them to language package managers, which are widely supported on Windows
      - seabass-labrax 3y ago
        Parent said that the "problem with system packaging is there are so many systems" - emphasis mine.
  - bluGill 3y ago
    The problem with systems packages is they solve a slightly different problem from language packages, and so while they are closer to what is needed, they are not right either. That is one reason I didn't specify the solution.
- code_biologist 3y ago
  Who then? Distros? One of my favorite software engineering comics is "User Friendly" from 1999: https://distrowatch.com/images/other/userfriendly-comic.png https://distrowatch.com/images/other/userfriendly-comic.png
  - oneshtein 3y ago
    Yep. Package management is the core of a (Linux) software distribution.
    - adhamsalama 3y ago
      Linux package management is already fragmented, the last thing we should do is add programming languages package management to it.
    - ungamedplayer 3y ago
      I thought the current process was just to herpderp it and put everything into a container.
      - bluGill 3y ago
        I hope that fad dies. It assumes all the world is x86-64 Linux. Maybe you get a few who acknowledge the raspberry pi. However there is a whole world of new other processors, *BSD, and others that the fad makes difficult to use.
  - bluGill 3y ago
    Nobody that currently exists - at least to my knowledge. This is a hard, mostly thankless problem. Distros solved a similar problem, but they have different motivations and so didn't fully solve everything. Languages like rust have a partial solution, but they don't play well with other languages and the full complexity that can result.
- RcouF1uZ4gsC 3y ago
  This really holds back languages and libraries. Trying to release libraries and language upgrades when you have to worry about different versions. One of the reasons Rust has been able to move so rapidly is that it is not dependent on system packagers.
- deleted 3y ago
  [deleted]
- globular-toast 3y ago
  Agreed. It's unbelievable to see all these languages inventing packaging over and over again. It's just an archive with some metadata and a hash/signature and a transport mechanism.
DylanSp 3y ago
FYI - Go no longer uses GOPATH for managing dependencies since the official module system was released in Go 1.11. There's still plenty of tutorials and the like out there that mention GOPATH, but it shouldn't be needed any more for basic scenarios.
- seabass-labrax 3y ago
  All that old information is terribly confusing, but it's not entirely useless. GOPATH is still critical, as it's how you can run fully-offline builds of Go programs and override dependencies without changing the root project itself.
  - DylanSp 3y ago
    Good to know, thanks! I haven't tried doing either of those.
  - XorNot 3y ago
    This isn't accurate: `go mod vendor` will dump the dependencies into the vendor repository these days, so you can build totally offline. There's also a mechanism for renaming dependencies (so you can replace them basically).
    - seabass-labrax 3y ago
      Not quite: both vendoring and the 'replace' directive in go.mod files (which I presume is the mechanism for renaming dependencies you are referring to) both require modifying the root package. With GOPATH, I can package a single library and use that package to fulfil the dependency on the library of any other Go application offline. Conversely, with the vendoring mechanism, the Go tooling will need to download (or copy from cache) the library when you originally create the vendor directory or when you add a new dependency to the root project.
codetrotter 3y ago
To add to your list: * Rust is awesome, and nothing can beat it. Cargo (the package manager that ships with Rust) is the best thing since sliced bread! Every other language can suck a bag of burritos.
- bnchrch 3y ago
  Ok ya true. Cargo is slick.
- shicholas 3y ago
  Finally a hot take on this site I agree with!
- nextaccountic 3y ago
  I actually agree, yet it still has shortcomings. For eg https://github.com/rust-lang/cargo/issues/545 https://github.com/rust-lang/cargo/issues/545 is still open (it's from 2014!!)
- jeffparsons 3y ago
  I'm sure we can all agree that Cargo is the best thing since sliced arrays, but I've finally reached the point where we have enough different Rust projects at work that rebuilding the world from scratch all the time and having ~40 GiB `target/` directories scattered around the place is getting old. I'm not meaning to criticize Rust or Cargo here; they're both doing their jobs just fine. But I do find myself craving a different compromise for much of what I/we use Rust for today. And I'm really hoping that Wasm Components (and WASI) will be that different compromise — e.g.: - Don't rebuild an entire HTTP stack from scratch for every tool that happens to use HTTP. - Therefore most projects won't have enormous `target/` dirs. - Reuse components built in CI for Linux directly on a Mac dev machine (cross platform). - Mix and match new processor architectures available in AWS without building two versions of everything. - Also reuse some components in the web browser (yes, we actually have real, boring use cases for this). - Wait, I'll be able to do all of this and still be writing Rust? Shut up and take my money.
  - rtpg 3y ago
p1necone 3y ago
One package management feature that not even the really good languages seem to have is synchronizing dependency versions across multiple packages, usually in the context of a monorepo. Like if my codebase has webapp A, library A and library B rather than separately defining that they all use third party library foo v3.1.4, it would be really nice to have a single source of truth.
KolenCh 3y ago
The answer to the Python problem you mention is conda/mamba.
- alfalfasprout 3y ago
  It's one part of the problem. Reliably managing those environments at scale can be tricky. Plus dealing with hybrid environments that include pip and conda packages.
whartung 3y ago
As an old school server side Java dev, I boggle at the hoops folks jump through to get stuff done when it comes to packaging. We had two large dependencies: the JDK, and the app server (our container, like Tomcat). Maven, that we’ve had forever, for library dependency. The resulting War files were effectively self contained. JDKs were trivially installed, explode it somewhere (anywhere) and set JAVA_HOME. App servers were the same. Self contained directory trees. Install as many as you like, just change the port they used. Or just add another War to the one you have already. Pros and cons. War files were essentially static linked, carrying all of their libraries. Self contained blob you could drag and drop into a directory and watch the server shutdown your old version and fire up the new one. Sure, we had our bit of DLL Hell, rarely when building the app. I’m certainly not going to suggest we never had class loader issues. And enterprise Java has a notoriety all its own, but packaging was pretty low on the list. But we didn’t need Docker, or dedicated VMs or anything like that. Our OS dependencies were all handled by the JDK. Java didn’t have any shared library concerns. I honestly never questioned it, either it was statically linked, or just entirely self contained outside of the C library. Everything else was in Java. OpenSSL, for example, was never an issue. I’m on board though, I loathe packaging. I’m not a big fan of arcane parameters passed on over the council fire and tea. Just never been my drive.
- mike_hearn 3y ago
  > Java didn’t have any shared library concerns It definitely does have such concerns, but the community has just accepted that the JDK will never help with this and so every JAR that uses a native library hacks around the lack of proper support in its own unique and special way. Usually some project-specific ad-hoc code that extracts native libraries into some directory in the user's home directory. Is the "cache" versioned properly? Maybe. Can you control where it is? Maybe. Code signing? Probably not. Can you ship only native code for the machines you actually care about? Maybe. Does it work when you run as a dedicated server UNIX users that doesn't have a home directory? Nope. Hydraulic Conveyor (https://hydraulic.dev/ https://hydraulic.dev/ - disclosure, my company) fixes a lot of these problems automatically. But it's not like there are no problems to fix, and of course writing a library that uses native code is still a big pain. It's a major blind spot of the JDK unfortunately, and the community has never risen to the challenge of fixing it.
eviks 3y ago
> extremely heavy node_modules folder what about global folder with most of the packages zipped with yarn?

seabass-labrax 3y ago

The Python world won't improve as long as programmers add dependencies on libraries written in other languages and (here's the important part) attempt to compile those packages themselves within a Python build process. Poetry is a nice chapter in the Python package definition story, but it is only a tentative step to fixing the wider problem.

bluGill 3y ago

I firmly believe that languages should not manage packages. While it makes the simple cases easy for beginners, the trade off is mixing languages becomes harder. There is no perfect language and often mixing should be the right answer and we don't want any more friction there.

seabass-labrax 3y ago
I quite agree! And would it even be harder for beginners? When trying a new a program written in a language I don't use frequently, I usually spend a good half-hour working out what command to run to install the dependencies and going through the logs working out what implicit requirement wasn't in the README. A holistic package, even one written for a different software distribution than the one I use (Debian at present), would immediately get me 99% of the way there. PS. If you find yourself in the same situation I do, Repology is your friend: https://repology.org/ https://repology.org/
- borodi 3y ago
  I do believe this is an issue of not having explicit dependencies. Julia takes the approach of, we build and ship everything for every OS, which means Pkg (the package manager) knows about binary dependencies as well. Making things more reproducible in language
  - pxc 3y ago
    Linux distros often do things to force packages to declare all their dependencies: Nix and Guix use unique prefixes and sandboxed builds, openSUSE builds all their packages in blank slate VMs that only install what is declared, standard Fedora tools for running builds in minimal chroot environments, etc. I'm not aware of any language ecosystem package managers taking similar measures to ensure that dependency declarations in their packages are complete.
maple3142 3y ago
The problem with system packaging is there are so many systems. For example if you only package something for Debian, how should a Fedora, Arch, Gentoo or even Mac and Windows user use that?
- seabass-labrax 3y ago
  Debian has by far the most rigorous packaging standards of that list, so subsequent packaging for the other distributions should not be very difficult. Bundling the dependencies into an OCI container or AppImage to run them on non-Debian systems is also trivial once you have a Debian package, but of course that comes with disadvantages. Neither Mac nor Windows have a proper first-party package manager (although winget makes some inroads, thanks of course to Keivan Beigi), so comparison becomes rather a moot point for those platforms. [1]: https://keivan.io/the-day-appget-died/ https://keivan.io/the-day-appget-died/
  - nightpool 3y ago
    "so comparison becomes rather a moot point for those platforms." No, because the OP was comparing them to language package managers, which are widely supported on Windows
    - seabass-labrax 3y ago
      Parent said that the "problem with system packaging is there are so many systems" - emphasis mine.
- bluGill 3y ago
  The problem with systems packages is they solve a slightly different problem from language packages, and so while they are closer to what is needed, they are not right either. That is one reason I didn't specify the solution.
code_biologist 3y ago
Who then? Distros? One of my favorite software engineering comics is "User Friendly" from 1999: https://distrowatch.com/images/other/userfriendly-comic.png https://distrowatch.com/images/other/userfriendly-comic.png
- oneshtein 3y ago
  Yep. Package management is the core of a (Linux) software distribution.
  - adhamsalama 3y ago
    Linux package management is already fragmented, the last thing we should do is add programming languages package management to it.
  - ungamedplayer 3y ago
    I thought the current process was just to herpderp it and put everything into a container.
    - bluGill 3y ago
      I hope that fad dies. It assumes all the world is x86-64 Linux. Maybe you get a few who acknowledge the raspberry pi. However there is a whole world of new other processors, *BSD, and others that the fad makes difficult to use.
- bluGill 3y ago
  Nobody that currently exists - at least to my knowledge. This is a hard, mostly thankless problem. Distros solved a similar problem, but they have different motivations and so didn't fully solve everything. Languages like rust have a partial solution, but they don't play well with other languages and the full complexity that can result.
RcouF1uZ4gsC 3y ago
This really holds back languages and libraries. Trying to release libraries and language upgrades when you have to worry about different versions. One of the reasons Rust has been able to move so rapidly is that it is not dependent on system packagers.
deleted 3y ago
[deleted]
globular-toast 3y ago
Agreed. It's unbelievable to see all these languages inventing packaging over and over again. It's just an archive with some metadata and a hash/signature and a transport mechanism.

DylanSp 3y ago

FYI - Go no longer uses GOPATH for managing dependencies since the official module system was released in Go 1.11. There's still plenty of tutorials and the like out there that mention GOPATH, but it shouldn't be needed any more for basic scenarios.

seabass-labrax 3y ago
All that old information is terribly confusing, but it's not entirely useless. GOPATH is still critical, as it's how you can run fully-offline builds of Go programs and override dependencies without changing the root project itself.
- DylanSp 3y ago
  Good to know, thanks! I haven't tried doing either of those.
- XorNot 3y ago
  This isn't accurate: `go mod vendor` will dump the dependencies into the vendor repository these days, so you can build totally offline. There's also a mechanism for renaming dependencies (so you can replace them basically).
  - seabass-labrax 3y ago
    Not quite: both vendoring and the 'replace' directive in go.mod files (which I presume is the mechanism for renaming dependencies you are referring to) both require modifying the root package. With GOPATH, I can package a single library and use that package to fulfil the dependency on the library of any other Go application offline. Conversely, with the vendoring mechanism, the Go tooling will need to download (or copy from cache) the library when you originally create the vendor directory or when you add a new dependency to the root project.

codetrotter 3y ago

To add to your list: * Rust is awesome, and nothing can beat it. Cargo (the package manager that ships with Rust) is the best thing since sliced bread! Every other language can suck a bag of burritos.

bnchrch 3y ago
Ok ya true. Cargo is slick.
shicholas 3y ago
Finally a hot take on this site I agree with!
nextaccountic 3y ago
I actually agree, yet it still has shortcomings. For eg https://github.com/rust-lang/cargo/issues/545 https://github.com/rust-lang/cargo/issues/545 is still open (it's from 2014!!)
jeffparsons 3y ago
I'm sure we can all agree that Cargo is the best thing since sliced arrays, but I've finally reached the point where we have enough different Rust projects at work that rebuilding the world from scratch all the time and having ~40 GiB `target/` directories scattered around the place is getting old. I'm not meaning to criticize Rust or Cargo here; they're both doing their jobs just fine. But I do find myself craving a different compromise for much of what I/we use Rust for today. And I'm really hoping that Wasm Components (and WASI) will be that different compromise — e.g.: - Don't rebuild an entire HTTP stack from scratch for every tool that happens to use HTTP. - Therefore most projects won't have enormous `target/` dirs. - Reuse components built in CI for Linux directly on a Mac dev machine (cross platform). - Mix and match new processor architectures available in AWS without building two versions of everything. - Also reuse some components in the web browser (yes, we actually have real, boring use cases for this). - Wait, I'll be able to do all of this and still be writing Rust? Shut up and take my money.
- rtpg 3y ago

p1necone 3y ago

One package management feature that not even the really good languages seem to have is synchronizing dependency versions across multiple packages, usually in the context of a monorepo. Like if my codebase has webapp A, library A and library B rather than separately defining that they all use third party library foo v3.1.4, it would be really nice to have a single source of truth.

KolenCh 3y ago

The answer to the Python problem you mention is conda/mamba.

alfalfasprout 3y ago
It's one part of the problem. Reliably managing those environments at scale can be tricky. Plus dealing with hybrid environments that include pip and conda packages.

whartung 3y ago

As an old school server side Java dev, I boggle at the hoops folks jump through to get stuff done when it comes to packaging. We had two large dependencies: the JDK, and the app server (our container, like Tomcat). Maven, that we’ve had forever, for library dependency. The resulting War files were effectively self contained. JDKs were trivially installed, explode it somewhere (anywhere) and set JAVA_HOME. App servers were the same. Self contained directory trees. Install as many as you like, just change the port they used. Or just add another War to the one you have already. Pros and cons. War files were essentially static linked, carrying all of their libraries. Self contained blob you could drag and drop into a directory and watch the server shutdown your old version and fire up the new one. Sure, we had our bit of DLL Hell, rarely when building the app. I’m certainly not going to suggest we never had class loader issues. And enterprise Java has a notoriety all its own, but packaging was pretty low on the list. But we didn’t need Docker, or dedicated VMs or anything like that. Our OS dependencies were all handled by the JDK. Java didn’t have any shared library concerns. I honestly never questioned it, either it was statically linked, or just entirely self contained outside of the C library. Everything else was in Java. OpenSSL, for example, was never an issue. I’m on board though, I loathe packaging. I’m not a big fan of arcane parameters passed on over the council fire and tea. Just never been my drive.

mike_hearn 3y ago
> Java didn’t have any shared library concerns It definitely does have such concerns, but the community has just accepted that the JDK will never help with this and so every JAR that uses a native library hacks around the lack of proper support in its own unique and special way. Usually some project-specific ad-hoc code that extracts native libraries into some directory in the user's home directory. Is the "cache" versioned properly? Maybe. Can you control where it is? Maybe. Code signing? Probably not. Can you ship only native code for the machines you actually care about? Maybe. Does it work when you run as a dedicated server UNIX users that doesn't have a home directory? Nope. Hydraulic Conveyor (https://hydraulic.dev/ https://hydraulic.dev/ - disclosure, my company) fixes a lot of these problems automatically. But it's not like there are no problems to fix, and of course writing a library that uses native code is still a big pain. It's a major blind spot of the JDK unfortunately, and the community has never risen to the challenge of fixing it.

eviks 3y ago

> extremely heavy node_modules folder what about global folder with most of the packages zipped with yarn?