6 ms·

This really just reads like someone who ignored literally every good practice for using python and then pikachu_shocked.jpg when he set his world on fire Just

This really just reads like someone who ignored literally every good practice for using python and then pikachu_shocked.jpg when he set his world on fire Just having a virtual environment and requirements.txt alone would solve 90% of this article. Also with python 3.12 you literally CANT install python packages at the system level. Giant full page warning saying “use a venv you idiot” I expected something along these lines and was still disappointed by TFA

zahlman 2y ago
In my experience, one of the biggest factors driving people to have the author's experience is... backwards compatibility. The old ways of doing things have existed for much longer than the new ways, and become well established. Everyone just accepts the idea of copying Pip into every new virtual environment, even though it's a) totally unnecessary (even before the `--python` option was introduced two years ago, you could sort of get by with options like `--target` and `--python-version` and `--prefix` and `--platform` and `--abi`) and b) utterly insane (slow, wasteful, makes it more confusing when your PATH gets messed up, leads to misconceptions...). And that's before considering the things people try to do that aren't officially blessed use cases - like putting `sudo` and `--break-system-packages` on the same command line without a second thought, or putting code in `setup.py` that actually tries to copy the Python files to specific locations for the user, or trying to run Pip explicitly via its non-existent "API" by calling undocumented stuff instead of just specifying dependencies (including "extras" lists) properly. (The Pip documentation explicitly recommends invoking Pip via `subprocess` instead; but you're still probably doing something wrong if this isn't part of your own custom alternative to Poetry etc., and it won't help you if Pip isn't available in the environment - which it doesn't have to be, except to support that kind of insane use case). Another part is that people just don't want to learn. Yes, you get a 'Giant full page warning saying “use a venv you idiot”'. Yes, the distro gets to customize that warning and tell you exactly what to do. Users will still give over a million hits to the corresponding Stack Overflow question (https://stackoverflow.com/questions/75608323 https://stackoverflow.com/questions/75608323), which will collect dozens of terrible answers, many of them suggesting complete circumvention of the package-management lock. It was over a year before anything significant was done about the top answer there (disclosure: I contributed quite a bit after that point; I have a personal policy of not writing new answers for Stack Overflow, but having a proper answer at the top of this question was far too important for me to ignore), which only happened because the matter was brought to the attention of the Python Discourse forum community (https://discuss.python.org/t/_/56900 https://discuss.python.org/t/_/56900).
liontwist 2y ago
npm ecosystem refugees trying out a new platform
jeremyjh 2y ago
The author's complaint is that python is supposed to be a good language for people new to programming to pick up. But the default tooling manages dependencies in a way that is unsound, and this has been known for more than a decade. Yet the defaults are still terrible, and regardless of how many articles there are on "best practices" a lot of people get burned.
- deleted 2y ago
  [deleted]
- drunkenmagician 2y ago
  Yes! This exactly.
- tomnipotent 2y ago
  I don't see any language in the blog post about "people new to programming to pick up". In fifteen years of using Python, the only people I see getting burned are, conveniently, the folks writing blogs on the subject. No one I've worked with or hired seems to be running into these issues. It's not to say that people don't run into issues, but the problems seem exaggerated every time this subject comes up.
- zahlman 2y ago
  People who are new to programming have a long way to go before even the concept of "managing dependencies" could possibly be made coherent for them. And the "unsoundness" described (i.e. not having lockfile-driven workflows by default) really just doesn't matter a huge percentage of the time. I've been writing Python for 20 years and what I write nowadays will still just work on multiple Python versions across a wide range of versions for my dependencies - if it even has any dependencies at all. But nowadays people seem to put the cart before the horse, and try to teach about programming language ecosystems before they've properly taught about programming. People new to programming need to worry about programming first. If there are any concepts they need to learn before syntax and debugging, it's how to use a command line (because it'll be harder to drive tools otherwise; IDEs introduce greater complexity) and how to use version control (so they can make mistakes fearlessly). Educators, my plea: if you teach required basic skills to programmers before you actually teach programming, then those skills are infinitely more important than modern "dependency management". And for heavens' sake, you can absolutely think of a few months' worth of satisfying lesson plans that don't require wrapping one's head around full-scale data-science APIs, or heaven forbid machine-learning libraries. If you need any more evidence of the proper priorities, just look at Stack Overflow. It gets flooded with zero-effort questions dumping some arcane error message from the bowels of Tensorflow, forwarded from some Numpy 2d arrays used as matrices having the wrong shape - and it'll get posted by someone who has no concept of debugging, no idea of any of the underlying ML theory, and very possibly no idea what matrix multiplication is or why it's useful. What good is it to teach "dependency management" to a student who's miles away from understanding the actual dependencies being managed? For that matter, sometimes they'll take a screenshot of the terminal instead of copying and pasting an error message (never mind proper formatting). Sometimes they even use a cell phone to take a picture of the computer monitor. You're just not going to teach "dependency management" successfully to someone who isn't properly comfortable with using a computer.
Starlevel004 2y ago
> and requirements.txt The proliferation of requirements.txt files is a massive reason for why Python dependency management sucks.
- zahlman 2y ago
  You'll be interested in relevant upcoming packaging standards PEPs: https://peps.python.org/pep-0751/ https://peps.python.org/pep-0751/ for lock files (still in discussion - have your say at https://discuss.python.org/t/_/69721 https://discuss.python.org/t/_/69721), and (recently accepted, but I can't immediately point at existing tool support) https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/ for listing groups of dependencies in pyproject.toml.
sunshowers 2y ago
The design of requirements.txt is a bit outdated -- it commits the first sin of developer tooling, which is to mix manually edited and automatically generated (via pip freeze) files. Newer systems use separate lockfiles for that reason, and uv brings this state of the art to Python.
- otteromkram 2y ago
  That's why I have a requirements folder with separate files (eg - dev.txt, prod.txt) for various installation needs. If you want to include test dependencies in development, just add it into the file like you're installing a regular requirements file: -r test.txt And, to double-down, if you read the pip documentation (the second sin of software development?), you can use things other than pip freeze. Like, python -m pip list --not-required That option flag is pretty nice because it excludes packages that aren't dependencies (aka - the primary packages that you need). If you do that you don't to worry about dependency management as much.
- janice1999 2y ago
  I think 'pip freeze' was introduced later and requirements.txt was not designed with such a use in mind. pipenv and other tools have lockfile equivalents for a while.
- askonomm 2y ago
  And that makes this, what, the 100th package management solution for Python? A big reason why Python package management sucks is because there are as many package managers as there are frameworks in JavaScript. Nobody ever knows what is the standard, and by the time that information propagates, it is no longer the standard.
  - sunshowers 2y ago
    I believe and hope that uv settles this discussion.
  - mmcnl 2y ago
    Is it really such a big deal that there are multiple "good enough" solutions available? I understand it is suboptimal, especially for beginners, but dependency management in Python is a solved problem for a while now. And also there are not that many solutions. Previously Poetry was the best, now uv is looking to take its place. Not 100 different solutions imo.
TZubiri 2y ago
"You don't know what caused the breakage, and you don't know how to go back to a working environment. " The author would be well served by using the first person and not including us in his uncertainty.
bmitc 2y ago
It still doesn't make Python's dependency management not to be horrible. Every other modern language has a single tool for these built-in. Python doesn't, with half of the tools coming with the core and half of the tools coming from third-parties and many issues, conflicts, and incompatibilities. Even Poetry, which makes things much, much easier, makes decisions that are incompatible or makes managing dependencies more difficult in some cases.

zahlman 2y ago

In my experience, one of the biggest factors driving people to have the author's experience is... backwards compatibility. The old ways of doing things have existed for much longer than the new ways, and become well established. Everyone just accepts the idea of copying Pip into every new virtual environment, even though it's a) totally unnecessary (even before the `--python` option was introduced two years ago, you could sort of get by with options like `--target` and `--python-version` and `--prefix` and `--platform` and `--abi`) and b) utterly insane (slow, wasteful, makes it more confusing when your PATH gets messed up, leads to misconceptions...). And that's before considering the things people try to do that aren't officially blessed use cases - like putting `sudo` and `--break-system-packages` on the same command line without a second thought, or putting code in `setup.py` that actually tries to copy the Python files to specific locations for the user, or trying to run Pip explicitly via its non-existent "API" by calling undocumented stuff instead of just specifying dependencies (including "extras" lists) properly. (The Pip documentation explicitly recommends invoking Pip via `subprocess` instead; but you're still probably doing something wrong if this isn't part of your own custom alternative to Poetry etc., and it won't help you if Pip isn't available in the environment - which it doesn't have to be, except to support that kind of insane use case). Another part is that people just don't want to learn. Yes, you get a 'Giant full page warning saying “use a venv you idiot”'. Yes, the distro gets to customize that warning and tell you exactly what to do. Users will still give over a million hits to the corresponding Stack Overflow question (https://stackoverflow.com/questions/75608323 https://stackoverflow.com/questions/75608323), which will collect dozens of terrible answers, many of them suggesting complete circumvention of the package-management lock. It was over a year before anything significant was done about the top answer there (disclosure: I contributed quite a bit after that point; I have a personal policy of not writing new answers for Stack Overflow, but having a proper answer at the top of this question was far too important for me to ignore), which only happened because the matter was brought to the attention of the Python Discourse forum community (https://discuss.python.org/t/_/56900 https://discuss.python.org/t/_/56900).

liontwist 2y ago

npm ecosystem refugees trying out a new platform

jeremyjh 2y ago

The author's complaint is that python is supposed to be a good language for people new to programming to pick up. But the default tooling manages dependencies in a way that is unsound, and this has been known for more than a decade. Yet the defaults are still terrible, and regardless of how many articles there are on "best practices" a lot of people get burned.

deleted 2y ago
[deleted]
drunkenmagician 2y ago
Yes! This exactly.
tomnipotent 2y ago
I don't see any language in the blog post about "people new to programming to pick up". In fifteen years of using Python, the only people I see getting burned are, conveniently, the folks writing blogs on the subject. No one I've worked with or hired seems to be running into these issues. It's not to say that people don't run into issues, but the problems seem exaggerated every time this subject comes up.
zahlman 2y ago
People who are new to programming have a long way to go before even the concept of "managing dependencies" could possibly be made coherent for them. And the "unsoundness" described (i.e. not having lockfile-driven workflows by default) really just doesn't matter a huge percentage of the time. I've been writing Python for 20 years and what I write nowadays will still just work on multiple Python versions across a wide range of versions for my dependencies - if it even has any dependencies at all. But nowadays people seem to put the cart before the horse, and try to teach about programming language ecosystems before they've properly taught about programming. People new to programming need to worry about programming first. If there are any concepts they need to learn before syntax and debugging, it's how to use a command line (because it'll be harder to drive tools otherwise; IDEs introduce greater complexity) and how to use version control (so they can make mistakes fearlessly). Educators, my plea: if you teach required basic skills to programmers before you actually teach programming, then those skills are infinitely more important than modern "dependency management". And for heavens' sake, you can absolutely think of a few months' worth of satisfying lesson plans that don't require wrapping one's head around full-scale data-science APIs, or heaven forbid machine-learning libraries. If you need any more evidence of the proper priorities, just look at Stack Overflow. It gets flooded with zero-effort questions dumping some arcane error message from the bowels of Tensorflow, forwarded from some Numpy 2d arrays used as matrices having the wrong shape - and it'll get posted by someone who has no concept of debugging, no idea of any of the underlying ML theory, and very possibly no idea what matrix multiplication is or why it's useful. What good is it to teach "dependency management" to a student who's miles away from understanding the actual dependencies being managed? For that matter, sometimes they'll take a screenshot of the terminal instead of copying and pasting an error message (never mind proper formatting). Sometimes they even use a cell phone to take a picture of the computer monitor. You're just not going to teach "dependency management" successfully to someone who isn't properly comfortable with using a computer.

Starlevel004 2y ago

> and requirements.txt The proliferation of requirements.txt files is a massive reason for why Python dependency management sucks.

zahlman 2y ago
You'll be interested in relevant upcoming packaging standards PEPs: https://peps.python.org/pep-0751/ https://peps.python.org/pep-0751/ for lock files (still in discussion - have your say at https://discuss.python.org/t/_/69721 https://discuss.python.org/t/_/69721), and (recently accepted, but I can't immediately point at existing tool support) https://peps.python.org/pep-0735/ https://peps.python.org/pep-0735/ for listing groups of dependencies in pyproject.toml.

sunshowers 2y ago

The design of requirements.txt is a bit outdated -- it commits the first sin of developer tooling, which is to mix manually edited and automatically generated (via pip freeze) files. Newer systems use separate lockfiles for that reason, and uv brings this state of the art to Python.

otteromkram 2y ago
That's why I have a requirements folder with separate files (eg - dev.txt, prod.txt) for various installation needs. If you want to include test dependencies in development, just add it into the file like you're installing a regular requirements file: -r test.txt And, to double-down, if you read the pip documentation (the second sin of software development?), you can use things other than pip freeze. Like, python -m pip list --not-required That option flag is pretty nice because it excludes packages that aren't dependencies (aka - the primary packages that you need). If you do that you don't to worry about dependency management as much.
janice1999 2y ago
I think 'pip freeze' was introduced later and requirements.txt was not designed with such a use in mind. pipenv and other tools have lockfile equivalents for a while.
askonomm 2y ago
And that makes this, what, the 100th package management solution for Python? A big reason why Python package management sucks is because there are as many package managers as there are frameworks in JavaScript. Nobody ever knows what is the standard, and by the time that information propagates, it is no longer the standard.
- sunshowers 2y ago
  I believe and hope that uv settles this discussion.
- mmcnl 2y ago
  Is it really such a big deal that there are multiple "good enough" solutions available? I understand it is suboptimal, especially for beginners, but dependency management in Python is a solved problem for a while now. And also there are not that many solutions. Previously Poetry was the best, now uv is looking to take its place. Not 100 different solutions imo.

TZubiri 2y ago

"You don't know what caused the breakage, and you don't know how to go back to a working environment. " The author would be well served by using the first person and not including us in his uncertainty.

bmitc 2y ago

It still doesn't make Python's dependency management not to be horrible. Every other modern language has a single tool for these built-in. Python doesn't, with half of the tools coming with the core and half of the tools coming from third-parties and many issues, conflicts, and incompatibilities. Even Poetry, which makes things much, much easier, makes decisions that are incompatible or makes managing dependencies more difficult in some cases.