24 ms·

This comment section itself clearly shows how crazy dependency and environment management is in Python. In this thread alone, we've received instructions to...

This comment section itself clearly shows how crazy dependency and environment management is in Python. In this thread alone, we've received instructions to... - poetry - "Just pin the dependencies and use Docker" - pip freeze - Vendoring in dependency code - pipreqs - virtualenv This is simply a mess and it's handled much better in other languages. I manage a small agency team and there are some weeks where I feel like we need a full-time devops person to just help resolve environment issues with Python projects around the team.

throwawaymaths 4y ago
Now try installing tensorflow. Treat yourself to ice cream if you get it to install without having to reinstall Linux and without borking the active project you're on.
- raffraffraff 4y ago
  And unless things have gotten a lot better in the 2 years since I last did `pip install numpy` on ARM, prepare for a very long wait because you'll be building it from source.
  - Gilboboy 4y ago
    There's a precompiled package for ARM these days.
    - codethief 4y ago
      Does it come with GPU support?
  - 0x008 4y ago
    Sometimes if you don’t install wheel, it will also automatically fall back to source build. Very sweet.
- derac 4y ago
  The major issues you'll see involve library version mismatches. It's a very good idea to use venv with these tools since there are often mismatches between projects.
  - throwawaymaths 4y ago
    Tensorflow sometimes is pinned to Nvidia drivers, and protobuf. And I think it has to be system level unless you reaaaaally want to fiddle with the internals.
- smeagull 4y ago
  I'd be too fat from all that icecream.
  - throwawaymaths 4y ago
    Ok. I know two companies that exist because of this problem.
feet 4y ago
Even using conda to manage reqs is an absolute nightmare. Did a subreq get updated? Did the author of the library pin that subreq? No? Have fun hunting down which library needs to be downgraded manually
- MonkeyMalarky 4y ago
  I used a couple of tricks to solve this. First, make cond env export a build step and environment.yml an artifact so you've got a nice summary of what got installed. Second, nightly builds so you aren't surprised by random package upgrade errors the next time you commit code to your project.
  - feet 4y ago
    My bad experiences are usually around setting up projects created by others, sadly
MonkeyMalarky 4y ago
Don't forget Anaconda because you're on windows and have no idea how to compile random packages that are really C++ code with Python bindings!
- jjoonathan 4y ago
  Solving environment | Solving environment / Solving environment - Solving environment \ Solving environment | ... One day it takes 10 seconds, next month it takes 10 minutes, and the month after that it takes 30 minutes and then fails entirely.
  - MonkeyMalarky 4y ago
    That's half of why from 2017 to 2021 I had a yearly "uninstall Anaconda and start fresh" routine. The other half is because I'd eventually corrupt my environments and have no choice but to start over.
    - jjoonathan 4y ago
      Me too. In 2021 it got so egregious that I finally jumped ship. Pip or bust. Since you specifically mentioned 2021 instead of 2022, I half suspect you had the same experience.
  - darkarmani 4y ago
    Are you using conda-forge? Solving from over 6TBs of packages can take quite a while. Conda-forge builds everything. This isn't a criticism, but because of that the number of packages is massive.
    - jjoonathan 4y ago
      "You can't use the big channel with all the packages because it has all the packages" isn't an exoneration, it's an indictment. To answer your question: yes, we were using conda-forge, and then when it stopped building we moved to a mix of conda and a critical subchannel, and then a major GIS library broke on conda and stayed that way for months so we threw in the towel and just used pip + a few touch-up scripts. Now that everyone else has followed suit, pip is the place where things "just work" so now we just use pip, no touchups required.
ksdnjweusdnkl21 4y ago
I agree that it is handled better in many other languages. However, Go has some weird thing with imports going on. When I tried to learn it I just could not import a function from another file. Some env variable making the program not find the path. Many stackoverflow/reddit threads condecendenly pointed to some setup guide in official docs which did not fix or explain the situation. After an few hours or so of not making much progress in AOC day 1 I just gave up and never continued learning Go.
Bullfight2Cond 4y ago
one more that supports the latest standards: https://pdm.fming.dev/ https://pdm.fming.dev/
korijn 4y ago
Is it a mess? Yes. But, is the problem to be solved perhaps much simpler "in other languages"? Do you interface with C++ libraries, system-managed dependencies, and perhaps your GPU in these other languages? Or are all your dependencies purely coded in these other languages, making everything simpler? Of course the answer to these questions could be anything but to me it feels like attacks on Python's package management are usually cheap shots on a much much more complicated problem domain than the "complainers" are typically aware of.
- pdimitar 4y ago
  Or the "complainers" work with Rust and Elixir and giggle at Python's last-century dependency-management woes, while they run a command or two and can upgrade and/or pin their dependencies and put that in version control and have builds identical [to those on their dev machines] in their CI/CD environment. ¯\_(ツ)_/¯ Your comment hints that you are feeling personally attacked when Python is criticized. Friendly unsolicited advice: don't do that, it's not healthy for you. Python is a relic. Its popularity and integration with super-strong C/C++ libraries has been carrying it for at least the last 5 years, if not 10. There's no mystery: it's a network effect. Quality is irrelevant when something is popular. And yes I used Python. Hated it every time. I guess I have to thank Python for learning bash scripting well. I still ended up wasting less time.
  - Zopieux 4y ago
    It's a bit ironic to pick someone's up on "taking things personally upon criticism", then proceeding to display a deep, manichean, unfounded hatered for a language that, despite its numerous flaws, remains a popular and useful tool.
    - pdimitar 4y ago
      Hanging people upon dawn was popular as well; people even got their kids for the event and it was happening regularly. Popularity says nothing about quality or even viability. Use Python if it's useful for you, obviously. To me though the writing is on the wall -- it's on its loooong and painful (due to people being in denial) way out. EDIT: I don't "hate"; it was a figure of speech. Our work has no place for such emotions. I simply get "sick of" (read: become weary of) something being preached as good when it clearly is not, at least in purely technical terms. And the "hate" is not at all unfounded. Choosing to ignore what doesn't conform to your view is not an argument.
okasaki 4y ago
Sometimes I feel people are using Python very differently than me. I just use pip freeze and virtualenv (these are Python basics, not some exotic tools) and I feel it works great. Granted, you don't get a nice executable, but it's still miles ahead of C++ (people literally put their code into header files so you don't have to link to a library), and even modern languages like rust (stuff is always broken, or I have some incompatible version, even when it builds it doesn't work) By the way if you're a Python user, Nim is worth checking out. It's compiled, fast and very low fuss kind of language that looks a lot like Python.
- pdimitar 4y ago
  > and even modern languages like rust (stuff is always broken, or I have some incompatible version, even when it builds it doesn't work) Been working on and off with Rust for the last 3 years, never happened to me once -- with the exception of the Tokio async runtime that has breaking changes between versions. Everything else always worked on the first try (checked with tests, too). Comparing Python with C++ doesn't make do argument any favours, and neither does stretching a single Rust accident to mean the ecosystem is bad.
  - mcronce 4y ago
    This is consistent with my experience. Semantic versioning is very very widely used in the Rust ecosystem, so you're not looking at breaking changes unless you select a different major version (or different minor version, for 0.x crates) - which you have to do manually, cargo will only automatically update dependencies to versions which semver specifies should be compatible. For crates that don't follow semver (which I'm fairly certain I've encountered zero times) you can pin a specific exact version.
- tempest_ 4y ago
  Comparing anything to C++ is a very low bar. I have rarely encountered issues in rust. Most rust crates stick to semver so you know when there will be a breaking change. My rust experience with Cargo can only be described as problem free(though I only develop for x86 linux). As for pip freeze and virtualenv things start to fall apart especially quickly when you require various C/C++ dependencies (which in various parts of the ecosystem is a lot) as well as different python versions (if you are supporting any kind of legacy software). This is also assuming other people working on the same project have the same python yadda yadda the list goes on, its not great.
punnerud 4y ago
Not a mess, but options. That’s what’s encourage innovation and open up for new ideas. This Fortran vs Python example is worth reading: https://cerfacs.fr/coop/fortran-vs-python https://cerfacs.fr/coop/fortran-vs-python
chpmrc 4y ago
- Poetry is a 3rd party package manager, I'm sure it's great but it's not widely used (yet) - Pip freeze just pins all dependencies at once to requirements.txt - I don't know what "vendoring in dependency code" means - I've never used pipreqs in my life (and 80% of my work has been in Python) - Virtualenvs are just a convenient way to keep project runtimes separated And for 90% of Python projects in existence the following is sufficient (assuming Python3 is installed): - python -m venv .venv - source .venv/bin/activate - pip install -r requirements.txt That's it. And all of that requires a single dependency: Python. Could it be better? Sure. But to call that a "mess" is an exaggeration.
- linsomniac 4y ago
  "Vendoring" means including the library in your package. So instead of listing it in requirements.txt, you copy the code of the library.
  - giantrobot 4y ago
    You can get that by bundling up your venv. When you install a package is a venv it installs it into that venv rather than the system. As far as the venv is concerned it is the system Python. Unfortunately passing around a venv can be problematic, say between Mac and Linux or between different architectures when binaries are involved.
- jeremydw 4y ago
  Sounds like someone has never been asked to clone and run software targeting Python 3.x when their system-installed Python is 3.y and the two are incompatible.
  - Spivak 4y ago
    Then you use pyenv and pyenv virtualenv. pyenv install 3.9.47 pyenv virtualenv 3.9.47 my app git clone …/myapp cd …/myapp pyenv local myapp pip install -r requirements.txt Is it annoying, maybe, but I normally don’t trust system deps for anything.
takeda 4y ago
Keep in mind that Python is 31 year old (it's even older than Java) it was created around the same time as world wide web. So it started when no one even knew they would need dependency management and evolved over time from people posting packages on their home pages, to a central website to what we now call PyPI. Similarly the tooling and way of packaging the code evolved. What you described are multiple tools that also target different areas: > - poetry from what you listed this seems like the only tool that actually takes care of dependency management > - "Just pin the dependencies and use Docker" this is standard fallback for all languages when people are lazy and don't want to figure out how to handle the dependencies > - pip freeze all this does it just lists currently installed packages in a form that can be automatically read by pip > - Vendoring in dependency code this again is just a way that applies to all languages, and it is still necessary even if there's a robust dependency management as there are some cases where bundling everything together is preferred > - pipreqs this is just a tool that scans your code and tells you what dependencies you are using. You are really lost if you need a tool to tell you what packages is your application is using, but I suppose it can be useful for one offs if you inherit some python code that wasn't using any dependence management. > - virtualenv this is just a method to have dependencies installed locally in project directory instead per system. This was created especially for development (although it can be used for deployment as well) as people started working on multiple services with different dependencies. It's now included in python so it's more like a feature of the language.
- sph 4y ago
  Being 31 years old doesn't preclude having a decent, official and reproducible way of installing packages in 2022. That's just a bad excuse to justify subpar package managers and terrible governance around this problem. Package management is pretty much a solved problem, no matter how old is your language. It smells to an outsider like me like a lot of bike-shedding and not enough pragmatism is going on in Python land over this issue. Has a new BDFL stepped up after Guido left?
  - FridgeSeal 4y ago
    No but there’s a walrus operator! Not sure anyone was asking for that, unlike a fix for packaging issues.
ActorNightly 4y ago
Its not a mess, people just make it a mess because of the lack of understanding around it, and getting lazy with using a combination of pip install, apt install, and whatever else. Also, the problem is compounded by people using Mac to develop, which have a different way of handling system wide python installs from brew, and then trying to port that to Linux.
- KptMarchewa 4y ago
  No one sane actually advocates using system python to develop anything, whether on mac or linux
- 0x008 4y ago
  One could argue that if you need a lot of understanding it is a mess.
SloopJon 4y ago
This has indeed been eye opening. We got bit by a dependency problem in which TensorFlow started pulling in an incompatible version of protobuf. After reading these comments, I don't think that pip freeze is quite what we want, but poetry sounds promising. We have a relatively small set of core dependencies, and a bunch of transitive dependencies that just need to work, and which we sometimes need to update for security fixes.
- codethief 4y ago
  Why do you think that `pip freeze` wouldn't be what you want? (I once had the exact same issue with TF and protobuf and specifying the exact protobuf version I wanted solved it.)
Gordonjcp 4y ago
It's not crazy at all. You use requirements.txt to keep a track of the dependencies needed for development, and you put the dependencies needed to build and install a package into setup.py where the packaging script can get it. These are two different things, because they do two different jobs. It's really very simple.
smeagull 4y ago
The real issue is that you can't have multiple versions installed at the same time. if you could import numpy==3.2 it'd solve many problems.
- thinker5555 4y ago
  Granted, I'm more of a hobbiest than a dev, but I think this is part of the problem that virtualenvs are supposed to help solve. One project (virtualenv) can have numpy==3.2, and another can have numpy==3.1. Maybe I'm naive, but it seems like having a one project with multiple versions of numpy being imported/called at various times would be asking for trouble.
  - Spivak 4y ago
    The thing you can’t do is solve for this situation. A depends on B, C B depends on D==1.24 C depends on D==2.02 There should in an super ideal world be no problem with this. You would just have a “linker” that inserts itself in the module loader that presents different module objects when deps ask for “D” but it hasn’t happened yet.
- goodpoint 4y ago
  That's a feature, not a bug.
  - smeagull 4y ago
    It's really not. I can't install tensorflow more than 10 times before I'm done. Virtualenvs are not fit for purpose.
bsder 4y ago
> This is simply a mess and it's handled much better in other languages. I don't agree. The problem is simply that Python encompasses a MUCH larger space with "package management" than most languages. It also has been around long enough to generate fairly deep dependency chains. As a counterexample, try using rust-analyzer or rust-skia on a Beaglebone Black. Good luck. Whereas, my Python stuff runs flawlessly. What many newer languages do is precompile the happy paths(x86, x86-64, and arm64) and then hang you out to dry if that's not what you are on.
brightball 4y ago
When I tried learning Python, this mess is what turned me off so badly. Python is the first language I ever came across where I felt like Docker was necessary just to keep the mess in a sandbox. Coming to that from hearing stories that there was supposed to be one way to do everything disenchanted me quickly.
- shaklee3 4y ago
  python has had virtual environments forever now. what was it missing?
  - brightball 4y ago
    Every article I found suggested different versions of Python, like Anaconda. They all suggested different virtual environments too. Rarely was an explanation given. At the time, the mix of Python 2 vs Python 3 was a mess. The code itself was okay, but everything around it was a train wreck compared to every other language I’d been using (Go, Java, Ruby, Elixir, even Perl). I attempted to get into it based on good things I’d heard online, but in the end it just wasn’t my cup of tea.
earthboundkid 4y ago
The core problem with Python dependencies is denial. There are tons of people who make excuses for a system that is so bad Randall Munroe declared his dependencies a “superfund site” years ago. In a good ecosystem, that comic would have prompted change. Instead, it just prompted a lot of “works for me comments”. Monads are just monoids in the category of endofunctors, and C just requires you to free memory at the right time. Python dependencies just work as long as you do a thing I’ve never seen my colleagues do at any job I’ve worked at.

throwawaymaths 4y ago

Now try installing tensorflow. Treat yourself to ice cream if you get it to install without having to reinstall Linux and without borking the active project you're on.

raffraffraff 4y ago
And unless things have gotten a lot better in the 2 years since I last did `pip install numpy` on ARM, prepare for a very long wait because you'll be building it from source.
- Gilboboy 4y ago
  There's a precompiled package for ARM these days.
  - codethief 4y ago
    Does it come with GPU support?
- 0x008 4y ago
  Sometimes if you don’t install wheel, it will also automatically fall back to source build. Very sweet.
derac 4y ago
The major issues you'll see involve library version mismatches. It's a very good idea to use venv with these tools since there are often mismatches between projects.
- throwawaymaths 4y ago
  Tensorflow sometimes is pinned to Nvidia drivers, and protobuf. And I think it has to be system level unless you reaaaaally want to fiddle with the internals.
smeagull 4y ago
I'd be too fat from all that icecream.
- throwawaymaths 4y ago
  Ok. I know two companies that exist because of this problem.

feet 4y ago

Even using conda to manage reqs is an absolute nightmare. Did a subreq get updated? Did the author of the library pin that subreq? No? Have fun hunting down which library needs to be downgraded manually

MonkeyMalarky 4y ago
I used a couple of tricks to solve this. First, make cond env export a build step and environment.yml an artifact so you've got a nice summary of what got installed. Second, nightly builds so you aren't surprised by random package upgrade errors the next time you commit code to your project.
- feet 4y ago
  My bad experiences are usually around setting up projects created by others, sadly

MonkeyMalarky 4y ago

Don't forget Anaconda because you're on windows and have no idea how to compile random packages that are really C++ code with Python bindings!

jjoonathan 4y ago
Solving environment | Solving environment / Solving environment - Solving environment \ Solving environment | ... One day it takes 10 seconds, next month it takes 10 minutes, and the month after that it takes 30 minutes and then fails entirely.
- MonkeyMalarky 4y ago
  That's half of why from 2017 to 2021 I had a yearly "uninstall Anaconda and start fresh" routine. The other half is because I'd eventually corrupt my environments and have no choice but to start over.
  - jjoonathan 4y ago
    Me too. In 2021 it got so egregious that I finally jumped ship. Pip or bust. Since you specifically mentioned 2021 instead of 2022, I half suspect you had the same experience.
- darkarmani 4y ago
  Are you using conda-forge? Solving from over 6TBs of packages can take quite a while. Conda-forge builds everything. This isn't a criticism, but because of that the number of packages is massive.
  - jjoonathan 4y ago
    "You can't use the big channel with all the packages because it has all the packages" isn't an exoneration, it's an indictment. To answer your question: yes, we were using conda-forge, and then when it stopped building we moved to a mix of conda and a critical subchannel, and then a major GIS library broke on conda and stayed that way for months so we threw in the towel and just used pip + a few touch-up scripts. Now that everyone else has followed suit, pip is the place where things "just work" so now we just use pip, no touchups required.

ksdnjweusdnkl21 4y ago

I agree that it is handled better in many other languages. However, Go has some weird thing with imports going on. When I tried to learn it I just could not import a function from another file. Some env variable making the program not find the path. Many stackoverflow/reddit threads condecendenly pointed to some setup guide in official docs which did not fix or explain the situation. After an few hours or so of not making much progress in AOC day 1 I just gave up and never continued learning Go.

Bullfight2Cond 4y ago

one more that supports the latest standards: https://pdm.fming.dev/ https://pdm.fming.dev/

korijn 4y ago

Is it a mess? Yes. But, is the problem to be solved perhaps much simpler "in other languages"? Do you interface with C++ libraries, system-managed dependencies, and perhaps your GPU in these other languages? Or are all your dependencies purely coded in these other languages, making everything simpler? Of course the answer to these questions could be anything but to me it feels like attacks on Python's package management are usually cheap shots on a much much more complicated problem domain than the "complainers" are typically aware of.

pdimitar 4y ago
Or the "complainers" work with Rust and Elixir and giggle at Python's last-century dependency-management woes, while they run a command or two and can upgrade and/or pin their dependencies and put that in version control and have builds identical [to those on their dev machines] in their CI/CD environment. ¯\_(ツ)_/¯ Your comment hints that you are feeling personally attacked when Python is criticized. Friendly unsolicited advice: don't do that, it's not healthy for you. Python is a relic. Its popularity and integration with super-strong C/C++ libraries has been carrying it for at least the last 5 years, if not 10. There's no mystery: it's a network effect. Quality is irrelevant when something is popular. And yes I used Python. Hated it every time. I guess I have to thank Python for learning bash scripting well. I still ended up wasting less time.
- Zopieux 4y ago
  It's a bit ironic to pick someone's up on "taking things personally upon criticism", then proceeding to display a deep, manichean, unfounded hatered for a language that, despite its numerous flaws, remains a popular and useful tool.
  - pdimitar 4y ago
    Hanging people upon dawn was popular as well; people even got their kids for the event and it was happening regularly. Popularity says nothing about quality or even viability. Use Python if it's useful for you, obviously. To me though the writing is on the wall -- it's on its loooong and painful (due to people being in denial) way out. EDIT: I don't "hate"; it was a figure of speech. Our work has no place for such emotions. I simply get "sick of" (read: become weary of) something being preached as good when it clearly is not, at least in purely technical terms. And the "hate" is not at all unfounded. Choosing to ignore what doesn't conform to your view is not an argument.

okasaki 4y ago

Sometimes I feel people are using Python very differently than me. I just use pip freeze and virtualenv (these are Python basics, not some exotic tools) and I feel it works great. Granted, you don't get a nice executable, but it's still miles ahead of C++ (people literally put their code into header files so you don't have to link to a library), and even modern languages like rust (stuff is always broken, or I have some incompatible version, even when it builds it doesn't work) By the way if you're a Python user, Nim is worth checking out. It's compiled, fast and very low fuss kind of language that looks a lot like Python.

pdimitar 4y ago
> and even modern languages like rust (stuff is always broken, or I have some incompatible version, even when it builds it doesn't work) Been working on and off with Rust for the last 3 years, never happened to me once -- with the exception of the Tokio async runtime that has breaking changes between versions. Everything else always worked on the first try (checked with tests, too). Comparing Python with C++ doesn't make do argument any favours, and neither does stretching a single Rust accident to mean the ecosystem is bad.
- mcronce 4y ago
  This is consistent with my experience. Semantic versioning is very very widely used in the Rust ecosystem, so you're not looking at breaking changes unless you select a different major version (or different minor version, for 0.x crates) - which you have to do manually, cargo will only automatically update dependencies to versions which semver specifies should be compatible. For crates that don't follow semver (which I'm fairly certain I've encountered zero times) you can pin a specific exact version.
tempest_ 4y ago
Comparing anything to C++ is a very low bar. I have rarely encountered issues in rust. Most rust crates stick to semver so you know when there will be a breaking change. My rust experience with Cargo can only be described as problem free(though I only develop for x86 linux). As for pip freeze and virtualenv things start to fall apart especially quickly when you require various C/C++ dependencies (which in various parts of the ecosystem is a lot) as well as different python versions (if you are supporting any kind of legacy software). This is also assuming other people working on the same project have the same python yadda yadda the list goes on, its not great.

punnerud 4y ago

Not a mess, but options. That’s what’s encourage innovation and open up for new ideas. This Fortran vs Python example is worth reading: https://cerfacs.fr/coop/fortran-vs-python https://cerfacs.fr/coop/fortran-vs-python

chpmrc 4y ago

- Poetry is a 3rd party package manager, I'm sure it's great but it's not widely used (yet) - Pip freeze just pins all dependencies at once to requirements.txt - I don't know what "vendoring in dependency code" means - I've never used pipreqs in my life (and 80% of my work has been in Python) - Virtualenvs are just a convenient way to keep project runtimes separated And for 90% of Python projects in existence the following is sufficient (assuming Python3 is installed): - python -m venv .venv - source .venv/bin/activate - pip install -r requirements.txt That's it. And all of that requires a single dependency: Python. Could it be better? Sure. But to call that a "mess" is an exaggeration.

linsomniac 4y ago
"Vendoring" means including the library in your package. So instead of listing it in requirements.txt, you copy the code of the library.
- giantrobot 4y ago
  You can get that by bundling up your venv. When you install a package is a venv it installs it into that venv rather than the system. As far as the venv is concerned it is the system Python. Unfortunately passing around a venv can be problematic, say between Mac and Linux or between different architectures when binaries are involved.
jeremydw 4y ago
Sounds like someone has never been asked to clone and run software targeting Python 3.x when their system-installed Python is 3.y and the two are incompatible.
- Spivak 4y ago
  Then you use pyenv and pyenv virtualenv. pyenv install 3.9.47 pyenv virtualenv 3.9.47 my app git clone …/myapp cd …/myapp pyenv local myapp pip install -r requirements.txt Is it annoying, maybe, but I normally don’t trust system deps for anything.

takeda 4y ago

Keep in mind that Python is 31 year old (it's even older than Java) it was created around the same time as world wide web. So it started when no one even knew they would need dependency management and evolved over time from people posting packages on their home pages, to a central website to what we now call PyPI. Similarly the tooling and way of packaging the code evolved. What you described are multiple tools that also target different areas: > - poetry from what you listed this seems like the only tool that actually takes care of dependency management > - "Just pin the dependencies and use Docker" this is standard fallback for all languages when people are lazy and don't want to figure out how to handle the dependencies > - pip freeze all this does it just lists currently installed packages in a form that can be automatically read by pip > - Vendoring in dependency code this again is just a way that applies to all languages, and it is still necessary even if there's a robust dependency management as there are some cases where bundling everything together is preferred > - pipreqs this is just a tool that scans your code and tells you what dependencies you are using. You are really lost if you need a tool to tell you what packages is your application is using, but I suppose it can be useful for one offs if you inherit some python code that wasn't using any dependence management. > - virtualenv this is just a method to have dependencies installed locally in project directory instead per system. This was created especially for development (although it can be used for deployment as well) as people started working on multiple services with different dependencies. It's now included in python so it's more like a feature of the language.

sph 4y ago
Being 31 years old doesn't preclude having a decent, official and reproducible way of installing packages in 2022. That's just a bad excuse to justify subpar package managers and terrible governance around this problem. Package management is pretty much a solved problem, no matter how old is your language. It smells to an outsider like me like a lot of bike-shedding and not enough pragmatism is going on in Python land over this issue. Has a new BDFL stepped up after Guido left?
- FridgeSeal 4y ago
  No but there’s a walrus operator! Not sure anyone was asking for that, unlike a fix for packaging issues.

ActorNightly 4y ago

Its not a mess, people just make it a mess because of the lack of understanding around it, and getting lazy with using a combination of pip install, apt install, and whatever else. Also, the problem is compounded by people using Mac to develop, which have a different way of handling system wide python installs from brew, and then trying to port that to Linux.

KptMarchewa 4y ago
No one sane actually advocates using system python to develop anything, whether on mac or linux
0x008 4y ago
One could argue that if you need a lot of understanding it is a mess.

SloopJon 4y ago

This has indeed been eye opening. We got bit by a dependency problem in which TensorFlow started pulling in an incompatible version of protobuf. After reading these comments, I don't think that pip freeze is quite what we want, but poetry sounds promising. We have a relatively small set of core dependencies, and a bunch of transitive dependencies that just need to work, and which we sometimes need to update for security fixes.

codethief 4y ago
Why do you think that `pip freeze` wouldn't be what you want? (I once had the exact same issue with TF and protobuf and specifying the exact protobuf version I wanted solved it.)

Gordonjcp 4y ago

It's not crazy at all. You use requirements.txt to keep a track of the dependencies needed for development, and you put the dependencies needed to build and install a package into setup.py where the packaging script can get it. These are two different things, because they do two different jobs. It's really very simple.

smeagull 4y ago

The real issue is that you can't have multiple versions installed at the same time. if you could import numpy==3.2 it'd solve many problems.

thinker5555 4y ago
Granted, I'm more of a hobbiest than a dev, but I think this is part of the problem that virtualenvs are supposed to help solve. One project (virtualenv) can have numpy==3.2, and another can have numpy==3.1. Maybe I'm naive, but it seems like having a one project with multiple versions of numpy being imported/called at various times would be asking for trouble.
- Spivak 4y ago
  The thing you can’t do is solve for this situation. A depends on B, C B depends on D==1.24 C depends on D==2.02 There should in an super ideal world be no problem with this. You would just have a “linker” that inserts itself in the module loader that presents different module objects when deps ask for “D” but it hasn’t happened yet.
goodpoint 4y ago
That's a feature, not a bug.
- smeagull 4y ago
  It's really not. I can't install tensorflow more than 10 times before I'm done. Virtualenvs are not fit for purpose.

bsder 4y ago

> This is simply a mess and it's handled much better in other languages. I don't agree. The problem is simply that Python encompasses a MUCH larger space with "package management" than most languages. It also has been around long enough to generate fairly deep dependency chains. As a counterexample, try using rust-analyzer or rust-skia on a Beaglebone Black. Good luck. Whereas, my Python stuff runs flawlessly. What many newer languages do is precompile the happy paths(x86, x86-64, and arm64) and then hang you out to dry if that's not what you are on.

brightball 4y ago

When I tried learning Python, this mess is what turned me off so badly. Python is the first language I ever came across where I felt like Docker was necessary just to keep the mess in a sandbox. Coming to that from hearing stories that there was supposed to be one way to do everything disenchanted me quickly.

shaklee3 4y ago
python has had virtual environments forever now. what was it missing?
- brightball 4y ago
  Every article I found suggested different versions of Python, like Anaconda. They all suggested different virtual environments too. Rarely was an explanation given. At the time, the mix of Python 2 vs Python 3 was a mess. The code itself was okay, but everything around it was a train wreck compared to every other language I’d been using (Go, Java, Ruby, Elixir, even Perl). I attempted to get into it based on good things I’d heard online, but in the end it just wasn’t my cup of tea.

earthboundkid 4y ago

The core problem with Python dependencies is denial. There are tons of people who make excuses for a system that is so bad Randall Munroe declared his dependencies a “superfund site” years ago. In a good ecosystem, that comic would have prompted change. Instead, it just prompted a lot of “works for me comments”. Monads are just monoids in the category of endofunctors, and C just requires you to free memory at the right time. Python dependencies just work as long as you do a thing I’ve never seen my colleagues do at any job I’ve worked at.