6 ms·

Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumul

Man it's frustrating to see this comic and read the comments here, because it's all so true. Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps. Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers. To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).

devy 9y ago
> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
- mikepurvis 9y ago
  If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
jwilk 9y ago
> Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation. Oh? I don't see any mention of pipenv in the cpython git repo.
- icebraining 9y ago
  https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies https://packaging.python.org/tutorials/managing-dependencies...
- yrro 9y ago
  Nor in the Python Packaging Authority documentation!
  - boxy_brown 9y ago
    Pipenv is mentioned in the PYPA Python Packaging User Guide tool recommendations: > Application dependency management > Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv. https://packaging.python.org/guides/tool-recommendations/ https://packaging.python.org/guides/tool-recommendations/
    - yrro 9y ago
      You're quite right. I skimmed Installing Packages, which mentions it in passing is a tool for managing multiple venvs, and Packaging & Distributing Projects, which doesn't mention it at all.
wintorez 9y ago
For a language that has "There should be one-- and preferably only one --obvious way to do it." as part of its core philosophy, this is very ironic.
- spookthesunset 9y ago
  Was thinking the same thing. When python was getting started it was up against Perl, which toon the opposite approach and there was a myriad of ways to do something in the language. Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager... Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
  - blowski 9y ago
    > Now days you wouldn’t dream of releasing a new language without some kind of standard package manager... Isn't this one of the major criticisms of Go?
    - HeyImAlex 9y ago
      Yeah, and the exact same thing is happening.
    - Dangeranger 9y ago
      My feeling is that a good package manager is the exception to the rule. Particularly around managing versions of the language runtime and the versions of packages available within projects using that runtime. Rust is the exception here in that Cargo is a very nice piece of kit. In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
      - curun1r 9y ago
alexbecker 9y ago
Pipenv solves a very real problem, but it has a lot of problems of its own: - It's very slow: re-locking after updating 1 dependency often takes me ~1 minute. - It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI. - They release multiple times per day, often breaking things in patch releases. - Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues. From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
- dvnguyen 9y ago
  If what you've heard is true, it could be pretty toxic. Do you have any evidences to back up?
- jwilk 9y ago
  The recommendation was added by Jon Wayne Parrott: https://github.com/pypa/python-packaging-user-guide/pull/443 https://github.com/pypa/python-packaging-user-guide/pull/443 He doesn't seem to be a pipenv maintainer: https://github.com/pypa/pipenv/commits?author=jonparrott https://github.com/pypa/pipenv/commits?author=jonparrott
- kennethreitz 9y ago
  I apologize for not meeting perfect standards of mood at all times. I have bipolar disorder, for what it's worth, and working on a packaging tool is very stress-inducing.
  - divbzero 9y ago
    It’s not easy to develop for large communities, and even tougher to speak frankly about oneself. Keep up the good work.
- kennethreitz 9y ago
  Also: > From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation. This is blatantly false.
baldfat 9y ago
Sharing your awesome Python program is its Achilles heel. I have 5 computers I work on a weekly basis. I stopped using Python and Haskell due to all of this. Racket's exe are so simple and quick to share. raco exe foo.rkt https://docs.racket-lang.org/raco/exe.html https://docs.racket-lang.org/raco/exe.html (Edited tendon to heel)
- disconnected 9y ago
  Yes, packaging Python applications for distribution is yet another nightmare situation. Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms. It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
- peatmoss 9y ago
  I've thought about starting a "build an ecosystem for Racket" project where I document gaps in the extended ecosystem and review best-of-breed solutions from other languages as a kind of literature review / review of prior art. Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
  - mark_edward 9y ago
    Would you be willing to collaborate on this? Racket was my first programming language and I miss it dearly. An excuse to write code in it would be great!
    - peatmoss 9y ago
      Oh my yes. Of course. PM me, and let’s figure out the right vehicle to get started.
  - baldfat 9y ago
    The only issues that this really could address. Would be nice to have a "Wall of Shame" for these gaps. A) Racket is so enjoyable everyone builds their own solutions B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
kbenson 9y ago
> Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps. I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this. That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it. (Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
yrro 9y ago
Please forgive me for venting my frustration here, but my reaction to this is basically: oh great, yet another blinking tool to add to the list of misery that is the Python packaging experience. May as well check it out now before I have to do it in a hurry... First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out! $ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done. Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv... $ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right! Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does? $ pipenv install --help * no menition of -e in the output * :sadface: Oh well, let's just run it blind! $ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found. Maybe I screwed up and ran this from the wrong directory? $ ls setup.py setup.py Weird, what's going on here? $ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found. Oh FFS, I give up. I think I'll let pipenv pass me by for now.
collyw 9y ago
As a Python dev it looks like I need to do some reading as I am still using virtualenv (it usually works fine but I have do encounter the occasional issue). Can anyone give me a rundown of the benefits of other systems over virtualenv? On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
- scrollaway 9y ago
  JS isn't in shambles; in fact, their package manager situation is excellent: Use yarn, period. It could be better (yarn isn't default yet), but I would love for the Python ecosystem to get to: "Use pipenv and setup.cfg, period". But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages. But we'll get there. One day. Hopefully it won't be too late.
  - collyw 9y ago
    So what are the benefits over virtualenv? You have given me the downside of another binary to manage. What do I get in return?
    - scrollaway 9y ago
      I've not given you anything other than a warning that it's too young. Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
      - collyw 9y ago
        That still doesn't answer the question of what it will (eventually) give me over virtualenv.
cpburns2009 9y ago
I just started using pipenv for a small personal project, and I like it. The usage is similar to git which I prefer over the weird bash activate mangling that a normal virtualenv does.
bmarkovic 9y ago
There was Zope Buildout for years, that had declarative packaging format, used the cheese shop (biggest issue with conda) and isn't slow as molasses but it was and still is largely ignored by the larger community.
sireat 9y ago
What best practices do I teach to complete beginners to Python? I am teaching adult beginners and it would be best to avoid the messy parts in the beginning. I thought I could get by with just pip and briefly touching on virtualenv. I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project. Theoretically Docker should suffice but it is yet another layer of complexity.

devy 9y ago

> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.

mikepurvis 9y ago
If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.

jwilk 9y ago

> Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation. Oh? I don't see any mention of pipenv in the cpython git repo.

icebraining 9y ago
https://packaging.python.org/tutorials/managing-dependencies/#managing-dependencies https://packaging.python.org/tutorials/managing-dependencies...
yrro 9y ago
Nor in the Python Packaging Authority documentation!
- boxy_brown 9y ago
  Pipenv is mentioned in the PYPA Python Packaging User Guide tool recommendations: > Application dependency management > Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv. https://packaging.python.org/guides/tool-recommendations/ https://packaging.python.org/guides/tool-recommendations/
  - yrro 9y ago
    You're quite right. I skimmed Installing Packages, which mentions it in passing is a tool for managing multiple venvs, and Packaging & Distributing Projects, which doesn't mention it at all.

wintorez 9y ago

For a language that has "There should be one-- and preferably only one --obvious way to do it." as part of its core philosophy, this is very ironic.

spookthesunset 9y ago
Was thinking the same thing. When python was getting started it was up against Perl, which toon the opposite approach and there was a myriad of ways to do something in the language. Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager... Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
- blowski 9y ago
  > Now days you wouldn’t dream of releasing a new language without some kind of standard package manager... Isn't this one of the major criticisms of Go?
  - HeyImAlex 9y ago
    Yeah, and the exact same thing is happening.
  - Dangeranger 9y ago
    My feeling is that a good package manager is the exception to the rule. Particularly around managing versions of the language runtime and the versions of packages available within projects using that runtime. Rust is the exception here in that Cargo is a very nice piece of kit. In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
    - curun1r 9y ago

alexbecker 9y ago

Pipenv solves a very real problem, but it has a lot of problems of its own: - It's very slow: re-locking after updating 1 dependency often takes me ~1 minute. - It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI. - They release multiple times per day, often breaking things in patch releases. - Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues. From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.

dvnguyen 9y ago
If what you've heard is true, it could be pretty toxic. Do you have any evidences to back up?
jwilk 9y ago
The recommendation was added by Jon Wayne Parrott: https://github.com/pypa/python-packaging-user-guide/pull/443 https://github.com/pypa/python-packaging-user-guide/pull/443 He doesn't seem to be a pipenv maintainer: https://github.com/pypa/pipenv/commits?author=jonparrott https://github.com/pypa/pipenv/commits?author=jonparrott
kennethreitz 9y ago
I apologize for not meeting perfect standards of mood at all times. I have bipolar disorder, for what it's worth, and working on a packaging tool is very stress-inducing.
- divbzero 9y ago
  It’s not easy to develop for large communities, and even tougher to speak frankly about oneself. Keep up the good work.
kennethreitz 9y ago
Also: > From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation. This is blatantly false.

baldfat 9y ago

Sharing your awesome Python program is its Achilles heel. I have 5 computers I work on a weekly basis. I stopped using Python and Haskell due to all of this. Racket's exe are so simple and quick to share. raco exe foo.rkt https://docs.racket-lang.org/raco/exe.html https://docs.racket-lang.org/raco/exe.html (Edited tendon to heel)

disconnected 9y ago
Yes, packaging Python applications for distribution is yet another nightmare situation. Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms. It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
peatmoss 9y ago
I've thought about starting a "build an ecosystem for Racket" project where I document gaps in the extended ecosystem and review best-of-breed solutions from other languages as a kind of literature review / review of prior art. Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
- mark_edward 9y ago
  Would you be willing to collaborate on this? Racket was my first programming language and I miss it dearly. An excuse to write code in it would be great!
  - peatmoss 9y ago
    Oh my yes. Of course. PM me, and let’s figure out the right vehicle to get started.
- baldfat 9y ago
  The only issues that this really could address. Would be nice to have a "Wall of Shame" for these gaps. A) Racket is so enjoyable everyone builds their own solutions B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.

kbenson 9y ago

> Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps. I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this. That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it. (Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)

yrro 9y ago

Please forgive me for venting my frustration here, but my reaction to this is basically: oh great, yet another blinking tool to add to the list of misery that is the Python packaging experience. May as well check it out now before I have to do it in a hurry... First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out! $ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done. Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv... $ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right! Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does? $ pipenv install --help * no menition of -e in the output * :sadface: Oh well, let's just run it blind! $ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found. Maybe I screwed up and ran this from the wrong directory? $ ls setup.py setup.py Weird, what's going on here? $ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found. Oh FFS, I give up. I think I'll let pipenv pass me by for now.

collyw 9y ago

As a Python dev it looks like I need to do some reading as I am still using virtualenv (it usually works fine but I have do encounter the occasional issue). Can anyone give me a rundown of the benefits of other systems over virtualenv? On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.

scrollaway 9y ago
JS isn't in shambles; in fact, their package manager situation is excellent: Use yarn, period. It could be better (yarn isn't default yet), but I would love for the Python ecosystem to get to: "Use pipenv and setup.cfg, period". But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages. But we'll get there. One day. Hopefully it won't be too late.
- collyw 9y ago
  So what are the benefits over virtualenv? You have given me the downside of another binary to manage. What do I get in return?
  - scrollaway 9y ago
    I've not given you anything other than a warning that it's too young. Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
    - collyw 9y ago
      That still doesn't answer the question of what it will (eventually) give me over virtualenv.

cpburns2009 9y ago

I just started using pipenv for a small personal project, and I like it. The usage is similar to git which I prefer over the weird bash activate mangling that a normal virtualenv does.

bmarkovic 9y ago

There was Zope Buildout for years, that had declarative packaging format, used the cheese shop (biggest issue with conda) and isn't slow as molasses but it was and still is largely ignored by the larger community.

sireat 9y ago

What best practices do I teach to complete beginners to Python? I am teaching adult beginners and it would be best to avoid the messy parts in the beginning. I thought I could get by with just pip and briefly touching on virtualenv. I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project. Theoretically Docker should suffice but it is yet another layer of complexity.