18 ms·

I have to warn again users that think they have found the silver bullet that pyenv comes with a big caveat: it compiles python on your machine. The number of p

I have to warn again users that think they have found the silver bullet that pyenv comes with a big caveat: it compiles python on your machine. The number of possible modes of failure in this situation is huge. See also: "Why not tell people to "simply" use pyenv, poetry or anaconda" https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup. Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.

bmitc 3y ago
> it compiles python on your machine What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
- bheadmaster 3y ago
  > What is the alternative? Using per-distro official installation channels. E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
- Leynos 3y ago
  Rye downloads pre-built binaries. I'll leave further discussion of the reliability and provenance of said binaries to someone else.
  - BiteCode_dev 3y ago
    Binaries are quite reliable and from a serious project, but rye is young and uses shim. Still, it's part of those new generation of tooling that bringing hope for the next few years.
- kzrdude 3y ago
  These builds are an alternative: https://github.com/indygreg/python-build-standalone https://github.com/indygreg/python-build-standalone Those are what Rye and hatch use. Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
  - formerly_proven 3y ago
    I think PEP711 (https://peps.python.org/pep-0711/ https://peps.python.org/pep-0711/) is (eventually) a better alternative, because it builds on top of the proven manylinux approach to binary compatibility.
    - kzrdude 3y ago
      Those are proof of concept builds now and will eventually hopefully replace indygreg's builds. Not only does the format need to exist but the service of building and publishing them is needed too.
    - codethief 3y ago
      Very cool, this is the first time I'm hearing of PEP 711! I hope this (or some other PEP like it) will get accepted eventually!
    - BiteCode_dev 3y ago
      I hope this will be merged with indygreg builds at some point, but the community move slowly on those issues so it may take years.
- BiteCode_dev 3y ago
  For now most people would do well to stick to python.org installers for mac and windows. For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky. If you use something more exotic, you chose hardship, and you will have to be up to the task. Anything else will come with bigger caveats than the people promoting them will really admit to. The story is of course a bit richer because linux packaging is fun, so I'll complete with: https://www.bitecode.dev/p/installing-python-the-bare-minimum https://www.bitecode.dev/p/installing-python-the-bare-minimu... Thete are currently no silver bullet to bootstrap python on linux. I tried all of them with hundred of colleagues and trainees. I do have hope for astral to come up with one but don't run on rye thinking your life is saved.
  - rbanffy 3y ago
    Even on Macs, you can use MacPorts. They provide a lot of versions pre-compiled (and everything is BSD-solid).
    - BiteCode_dev 3y ago
      I know you mean well, but such advice is why so many beginners have painful Python experience. Macports and homebrew Pythons are dependencies of other packages. They can be used by you, but they are not meant for you. This means at some point, they will contain a surprise, and not a good one.
      - rbanffy 3y ago
        > but such advice is why so many beginners have painful Python experience Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
    - hot_gril 3y ago
      Personally I only use MacPorts or Homebrew to install random smaller tools. It's easier to install Python, NodeJS, Postgres, etc binaries from the main website.
- rbanffy 3y ago
  > Every solution that I know of on Linux requires you to build Python on the machine Unless you need a Python that's not supported by your Linux distribution, you can just use what's available. On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
  - SOLAR_FIELDS 3y ago
    If you’re lucky enough to be on a Linux system that uses apt some thankless soul maintains a repo called deadsnakes with all these binaries. Fabulous if you’re using any somewhat old version of Python in CI for instance. Yum based systems are SOL as far as I can tell. Build and host your own binary for that. Apk doesn’t have this either IIRC
    - rbanffy 3y ago
      If you are deploying to such an ancient OS, it's perhaps easier to have the whole OS packaged as a container or a VM and use that. It's not only different Python versions that might bite you.
      - Hackbraten 3y ago
        What makes you think that GP’s comment is talking about older OSes? My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
        SOLAR_FIELDS 3y ago
        This is not about system packages or OSes, this is about your application needing Python 3.9.7 specifically, and locking to that, and 3.9.7 not being available in repositories anymore (3.9.7 is just an example). So normally you would either need to self host 3.9.7 somewhere or compile it from source on every new machine (which is terrible for CI but fine for local dev, a one off in local dev to build a Python version is nothing but paying 4 minutes every time on ci to compile your version of Python from source is a very angry amount of time for a lot of people).
- tgz 3y ago
  try pyoxidizer
rcarmo 3y ago
Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now). The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker). Additionally, you can build an _optimized_ Python. I have this set for my builds: env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2 [1]: https://taoofmac.com/space/blog/2015/10/03/1245 https://taoofmac.com/space/blog/2015/10/03/1245
- BiteCode_dev 3y ago
  That happens way more often than people want to admit, and I think it's not genuine to present the tool that way. E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard machine, I'll get: pyenv install 3.9 -v /tmp/python-build.20240325124651.73089 ~ Downloading Python-3.9.19.tar.xz... -> https://www.python.org/ftp/python/3.9.19/Python-3.9.19.tar.xz ... LD_LIBRARY_PATH=/tmp/python-build.20240325124651.73089/Python-3.9.19 CC='gcc -pthread' LDSHARED='gcc -pthread -shared -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build running build running build_ext ... etc There are obviously binaries for this Python, since I can apt install it and I didn't specify the minor version. But by default it downloads the source, and compiles it. On top of that it will use a shim which comes with its own world of possible pain. Again, I don't want to bash on pyenv. But I do want to lower people's expectations. It's a tool for experts, not beginners.
  - orf 3y ago
    I’ve quite literally never seen it download a pre-compiled Python - I didn’t actually know it had that functionality, and have been using it for probably a decade.
    - 3y ago
wodenokoto 3y ago
I generally like bite codes newsletter but this one was a miss for me. I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.
mikemcquaid 3y ago
If you use Homebrew: you can use ‘brew pyenv-sync’ to use Homebrew’s pythons with pyenv. Similar commands are available for rbenv/nodenv (which always feel like it is missing an ‘e’ to me)
lb4r 3y ago
> See also: "Why not tell people to "simply" use pyenv, poetry or anaconda" > https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
- chrisfinazzo 3y ago
  I eventually landed on pipx as fighting with Pyenv and Anaconda - via Miniconda - was an exercise in frustration. There's some mucking about in `$HOME/.local`, but this is mostly self-contained and not a huge chore to keep running. Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do. Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
- bootsmann 3y ago
  Agreed pipx solves a lot of packaging issues with no downside to speak of. Not just with poetry but also for tools like virtualenv, ruff and black and non-dev command line tools.
mwexler 3y ago
So, what is "the tool for beginners fighting with python packaging problems"? That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks. Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
- IshKebab 3y ago
  There isn't one. Python distribution and packaging is just fundamentally horribly broken. I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
  - notatallshaw 3y ago
    > Python distribution and packaging is just fundamentally horribly broken It's clearly not because most people successfully use it fine. The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware. The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work. Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-getting-better-a-datapoint/ https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
    - Capricorn2481 3y ago
      I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
cqqxo4zV46cp 3y ago
I think that you are painting a bit of an unreasonably bleak view of pyenv. I think that one can easily get value out of it without being a Python expert. I’m not sure how I can refute your “yes, but you’ll eventually run into trouble. You just haven’t clocked enough hours yet”. I can’t prove a negative. But I’ll say that I’ve been writing Python in my day job for a decade. But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
belter 3y ago
I call it the curse of Python. When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
- throwup238 3y ago
  And then the Devil made CUDA… ”The CUDA version on your system does not match the CUDA version the realm of mortal men was compiled with”
b33j0r 3y ago
If you don’t care which specific version of python you are using, do not use pyenv. You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you: “Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!” Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose. However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway. To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.) For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt. The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
- elevation 3y ago
  This kind of error surprises me: > I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2 I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs. But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
  - b33j0r 3y ago
    (Edit2 tl:dr: it’s that other package maintainers don’t always keep up with new semver constants in their metadata, particularly on anything cutting-edge) It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always. This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu. Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career. Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.
guappa 3y ago
Python is quite self contained, it's not a chore to build it.
whalesalad 3y ago
I have not suffered any issues due to pyenv compiling. In our prod boxes we compile Python ourselves anyway. It's a really trivial build process tbh.
mrbonner 3y ago
Oh man, you just confirm that I’m not crazy. For every single time I use pyenv it just downloads source and build. I got tangled into so many issues such as TLS headers in my AL2 boxes. I thought the whole thing is about getting the official binary and install and not compile from source even for a 2 year old release.
rewgs 3y ago
What's the problem? The repo clearly states that you need to install such and such build prerequisites. Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
d0mine 3y ago
Data point: pyenv works just fine for me. It helps me with installing/managing many python versions (multiple years). I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.
Vaslo 3y ago
Can I ask - why the dislike for virtualenvwrapper? If you are saying that it can have occasional problems, I agree with you. But it makes the process so much simpler. The occasional problems I’ve had (had an issue once or twice deleting a venv) pales in comparison to the advantage I get in remembering a couple fast commands. Is there something better or am I relegated to “source what/directory/was/it/again/bin/activate?
Capricorn2481 3y ago
> People that love pyenv are so adamant at telling everybody they should use it. How it solved all their problems, and got rich and lost 5 pounds. > It doesn’t support Windows. That’s game over right there, for half of the community. What do you mean by this? I use pyenv on windows all the time.
hot_gril 3y ago
Why does it compile Python on your machine?

bmitc 3y ago

> it compiles python on your machine What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.

bheadmaster 3y ago
> What is the alternative? Using per-distro official installation channels. E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
Leynos 3y ago
Rye downloads pre-built binaries. I'll leave further discussion of the reliability and provenance of said binaries to someone else.
- BiteCode_dev 3y ago
  Binaries are quite reliable and from a serious project, but rye is young and uses shim. Still, it's part of those new generation of tooling that bringing hope for the next few years.
kzrdude 3y ago
These builds are an alternative: https://github.com/indygreg/python-build-standalone https://github.com/indygreg/python-build-standalone Those are what Rye and hatch use. Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
- formerly_proven 3y ago
  I think PEP711 (https://peps.python.org/pep-0711/ https://peps.python.org/pep-0711/) is (eventually) a better alternative, because it builds on top of the proven manylinux approach to binary compatibility.
  - kzrdude 3y ago
    Those are proof of concept builds now and will eventually hopefully replace indygreg's builds. Not only does the format need to exist but the service of building and publishing them is needed too.
  - codethief 3y ago
    Very cool, this is the first time I'm hearing of PEP 711! I hope this (or some other PEP like it) will get accepted eventually!
  - BiteCode_dev 3y ago
    I hope this will be merged with indygreg builds at some point, but the community move slowly on those issues so it may take years.
BiteCode_dev 3y ago
For now most people would do well to stick to python.org installers for mac and windows. For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky. If you use something more exotic, you chose hardship, and you will have to be up to the task. Anything else will come with bigger caveats than the people promoting them will really admit to. The story is of course a bit richer because linux packaging is fun, so I'll complete with: https://www.bitecode.dev/p/installing-python-the-bare-minimum https://www.bitecode.dev/p/installing-python-the-bare-minimu... Thete are currently no silver bullet to bootstrap python on linux. I tried all of them with hundred of colleagues and trainees. I do have hope for astral to come up with one but don't run on rye thinking your life is saved.
- rbanffy 3y ago
  Even on Macs, you can use MacPorts. They provide a lot of versions pre-compiled (and everything is BSD-solid).
  - BiteCode_dev 3y ago
    I know you mean well, but such advice is why so many beginners have painful Python experience. Macports and homebrew Pythons are dependencies of other packages. They can be used by you, but they are not meant for you. This means at some point, they will contain a surprise, and not a good one.
    - rbanffy 3y ago
      > but such advice is why so many beginners have painful Python experience Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
  - hot_gril 3y ago
    Personally I only use MacPorts or Homebrew to install random smaller tools. It's easier to install Python, NodeJS, Postgres, etc binaries from the main website.
rbanffy 3y ago
> Every solution that I know of on Linux requires you to build Python on the machine Unless you need a Python that's not supported by your Linux distribution, you can just use what's available. On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
- SOLAR_FIELDS 3y ago
  If you’re lucky enough to be on a Linux system that uses apt some thankless soul maintains a repo called deadsnakes with all these binaries. Fabulous if you’re using any somewhat old version of Python in CI for instance. Yum based systems are SOL as far as I can tell. Build and host your own binary for that. Apk doesn’t have this either IIRC
  - rbanffy 3y ago
    If you are deploying to such an ancient OS, it's perhaps easier to have the whole OS packaged as a container or a VM and use that. It's not only different Python versions that might bite you.
    - Hackbraten 3y ago
      What makes you think that GP’s comment is talking about older OSes? My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
      - SOLAR_FIELDS 3y ago
        This is not about system packages or OSes, this is about your application needing Python 3.9.7 specifically, and locking to that, and 3.9.7 not being available in repositories anymore (3.9.7 is just an example). So normally you would either need to self host 3.9.7 somewhere or compile it from source on every new machine (which is terrible for CI but fine for local dev, a one off in local dev to build a Python version is nothing but paying 4 minutes every time on ci to compile your version of Python from source is a very angry amount of time for a lot of people).
tgz 3y ago
try pyoxidizer

rcarmo 3y ago

Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now). The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker). Additionally, you can build an _optimized_ Python. I have this set for my builds: env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2 [1]: https://taoofmac.com/space/blog/2015/10/03/1245 https://taoofmac.com/space/blog/2015/10/03/1245

BiteCode_dev 3y ago
That happens way more often than people want to admit, and I think it's not genuine to present the tool that way. E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard machine, I'll get: pyenv install 3.9 -v /tmp/python-build.20240325124651.73089 ~ Downloading Python-3.9.19.tar.xz... -> https://www.python.org/ftp/python/3.9.19/Python-3.9.19.tar.xz ... LD_LIBRARY_PATH=/tmp/python-build.20240325124651.73089/Python-3.9.19 CC='gcc -pthread' LDSHARED='gcc -pthread -shared -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build running build running build_ext ... etc There are obviously binaries for this Python, since I can apt install it and I didn't specify the minor version. But by default it downloads the source, and compiles it. On top of that it will use a shim which comes with its own world of possible pain. Again, I don't want to bash on pyenv. But I do want to lower people's expectations. It's a tool for experts, not beginners.
- orf 3y ago
  I’ve quite literally never seen it download a pre-compiled Python - I didn’t actually know it had that functionality, and have been using it for probably a decade.
  - 3y ago

wodenokoto 3y ago

I generally like bite codes newsletter but this one was a miss for me. I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.

mikemcquaid 3y ago

If you use Homebrew: you can use ‘brew pyenv-sync’ to use Homebrew’s pythons with pyenv. Similar commands are available for rbenv/nodenv (which always feel like it is missing an ‘e’ to me)

lb4r 3y ago

> See also: "Why not tell people to "simply" use pyenv, poetry or anaconda" > https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.

chrisfinazzo 3y ago
I eventually landed on pipx as fighting with Pyenv and Anaconda - via Miniconda - was an exercise in frustration. There's some mucking about in `$HOME/.local`, but this is mostly self-contained and not a huge chore to keep running. Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do. Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
bootsmann 3y ago
Agreed pipx solves a lot of packaging issues with no downside to speak of. Not just with poetry but also for tools like virtualenv, ruff and black and non-dev command line tools.

mwexler 3y ago

So, what is "the tool for beginners fighting with python packaging problems"? That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks. Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?

IshKebab 3y ago
There isn't one. Python distribution and packaging is just fundamentally horribly broken. I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
- notatallshaw 3y ago
  > Python distribution and packaging is just fundamentally horribly broken It's clearly not because most people successfully use it fine. The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware. The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work. Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-getting-better-a-datapoint/ https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
  - Capricorn2481 3y ago
    I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.

cqqxo4zV46cp 3y ago

I think that you are painting a bit of an unreasonably bleak view of pyenv. I think that one can easily get value out of it without being a Python expert. I’m not sure how I can refute your “yes, but you’ll eventually run into trouble. You just haven’t clocked enough hours yet”. I can’t prove a negative. But I’ll say that I’ve been writing Python in my day job for a decade. But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.

belter 3y ago

I call it the curse of Python. When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)

throwup238 3y ago
And then the Devil made CUDA… ”The CUDA version on your system does not match the CUDA version the realm of mortal men was compiled with”

b33j0r 3y ago

If you don’t care which specific version of python you are using, do not use pyenv. You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you: “Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!” Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose. However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway. To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.) For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt. The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)

elevation 3y ago
This kind of error surprises me: > I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2 I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs. But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
- b33j0r 3y ago
  (Edit2 tl:dr: it’s that other package maintainers don’t always keep up with new semver constants in their metadata, particularly on anything cutting-edge) It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always. This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu. Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career. Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.

guappa 3y ago

Python is quite self contained, it's not a chore to build it.

whalesalad 3y ago

I have not suffered any issues due to pyenv compiling. In our prod boxes we compile Python ourselves anyway. It's a really trivial build process tbh.

mrbonner 3y ago

Oh man, you just confirm that I’m not crazy. For every single time I use pyenv it just downloads source and build. I got tangled into so many issues such as TLS headers in my AL2 boxes. I thought the whole thing is about getting the official binary and install and not compile from source even for a 2 year old release.

rewgs 3y ago

What's the problem? The repo clearly states that you need to install such and such build prerequisites. Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.

d0mine 3y ago

Data point: pyenv works just fine for me. It helps me with installing/managing many python versions (multiple years). I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.

Vaslo 3y ago

Can I ask - why the dislike for virtualenvwrapper? If you are saying that it can have occasional problems, I agree with you. But it makes the process so much simpler. The occasional problems I’ve had (had an issue once or twice deleting a venv) pales in comparison to the advantage I get in remembering a couple fast commands. Is there something better or am I relegated to “source what/directory/was/it/again/bin/activate?

Capricorn2481 3y ago

> People that love pyenv are so adamant at telling everybody they should use it. How it solved all their problems, and got rich and lost 5 pounds. > It doesn’t support Windows. That’s game over right there, for half of the community. What do you mean by this? I use pyenv on windows all the time.

hot_gril 3y ago

Why does it compile Python on your machine?