4 ms·

I hate python package/dependency/virtualenv management so much. Even JavaScript is preferable. I find npm very easy to wrap my head around. I do npm install __

by umvi 8y ago

I hate python package/dependency/virtualenv management so much. Even JavaScript is preferable. I find npm very easy to wrap my head around. I do npm install ___ and it does a lookup in its repository and downloads it and its dependencies to node_modules. If I want to start fresh, I simply delete node_modules. Everything else "just works" when I invoke node myapp.js. Punto e basta. pipenv masquarades as the same thing, but then there is no python_modules folder to be found. Instead it downloads it to some obscure directory halfway across my computer. If I want to start fresh I guess I have to trust the tool's uninstall function? And it's unclear if I need sudo or not. Also I can no longer just run my program, I have to run it with pipenv now? Python requires too much cognitive burden for module/dependency/virtualenv management. For me it's to the point that developing using a docker image with globally installed python modules is easier to manage and wrap my brain around than using pipenv/virtualenv/whatever.

cx1000 8y ago
PEP 582 is in the works to make __pypackages__ the Python equivalent of node_modules. Not sure if it will get accepted but I hope it does. > a mechanism to automatically recognize a __pypackages__ directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate "virtual environments". https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/
- umvi 8y ago
  That would be great! Any way to upvote or otherwise give the equivalent of a GitHub thumbs up to show support for it?
- JNRowe 8y ago
  That’s a great idea. I’ve just been playing with the implementation from the PEP, and it makes things really easy. The speed of simply extracting a bunch of wheels directly in to `__pypackages__/$PYVER/lib/` is a huge benefit. It behaves well if you symlink from a global tree of extracted packages too, like an even simpler version of pundler¹. If others want to play with it without too much breakage, I massaged the patch on to 3.7². The 3.8 base was a little too big a change for my liking ;) 1. https://github.com/Deepwalker/pundler https://github.com/Deepwalker/pundler 2. https://gist.github.com/JNRowe/502f8772861cdd21bff5144c416e4b31 https://gist.github.com/JNRowe/502f8772861cdd21bff5144c416e4... Edit: It’s a third of a second to stand up a complete environment with 63 wheels in its dependency tree for the project I'm playing with right now.
wirrbel 8y ago
Basically it seems you are lacking only two shell commands. virtualenv venv . venv/bin/activate Then everything works with pip pip install numpy I actually prefer the pipenv way (or the virtualenvwrapper way) of putting virtualenvs into a dedicated location, so that I can wipe them off the hard disk if I want to free some space. The real interesting and occasionally bothering point on managing your virtualenv is resolution of dependencies and compatibility ranges (hard to solve in general).
- icebraining 8y ago
  At a previous job, we used an even simpler solution: just download everything to a directory (called "deps") with pip install --target deps -r requirements.txt Then running the application with PYTHONPATH=deps python myapp.py This was also simple to integrate with service supervisors, since most make it easy to configure a environment variable.
  - jazzyjackson 8y ago
    Making a note of this, didn't know about the target flag, thanks !
  - ProblemFactory 8y ago
    Virtualenv tip: you can skip the Bash magic of calling "activate" by providing the path to the python executable. /hello/venvs/myproject/bin/python myscript.py will run myscript.py exactly as if the virtualenv was active, which is easier to configure for cron jobs and services.
  - wirrbel 8y ago
    If this works for you, good. I think I would be cautious giving this out as advice because you can run into problems this way with regards to different python versions. The activate script uses the python version used to setup the environment.
- chatmasta 8y ago
  This doesn’t address his problem of dependencies being “halfway across my computer.”
mlthoughts2018 8y ago
Use conda
androidgirl 8y ago
See, I feel like python has a lot of those benefits and the package management doesn't need to be so complex. Seriously, python package management can be fairly simple. On most of our machines at work, it's just "virtualenv .env && source .env/bin/activate". Then you install your packages and... everything is in one directory, like node_modules in javascript. A clean reinstall is easy from there: remove the .env and just repeat. I feel like pipenv violates KISS, and that a more traditional virtualenv/venv/pyenv setup is the way to go.
- umvi 8y ago
  > A clean reinstall is easy from there: remove the .env and just repeat. Repeat what, manually doing a bazillion `pip install`? Another nice thing about npm is the packages.json file it creates. This allows us to simply add that to version control and then all the new dev has to do is clone and run `npm install` which reads packages.json and installs everything inside of it. I'm sure there's a way to do it in python but, like everything else I bet it's a non-intuitive multistep process. JavaScript ========== First time installation: npm install express --save Subsequent times: npm install Starting clean: rm -rf node_modules Running application: node myapp.js Python ====== First time installation: virtualenv .env source .env/bin/activate pip3 install requests pip3 freeze > requirements.txt Subsequent times: virtualenv .env source .env/bin/activate pip install -r requirements.txt Starting clean: rm -rf .env [exit bash to clear environment] [start bash] Running application: virtualenv .env source .env/bin/activate python3 myapp.py For JavaScript I typed it all out from memory. For Python I had to consult StackOverflow because I couldn't remember "pip freeze". Yes it can be made simple with use of aliases and such, but like I said in my first post, out-of-the-box cognitive burden is several times greater than, say, npm or yarn.
  - toyg 8y ago
    > [exit bash to clear environment] Wat? just deactivate before you rm. Please do not talk about things you clearly don't know well enough (which is patently the case if you can't rememer freeze and deactivate). > Out-of-the-box cognitive burden is several times greater You got any stats on that, or is it just your opinion? Because to me, the completely counter-intuitive --save parameter is much more painful to remember. The venv/pip workflow is not perfect, but what you've described is not the problem.
Evidlo 8y ago
I don't get the node way of doing things. Why should the availability of modules be dependent on the current working directory? Seems brittle to me that your app can't find its libraries because you ran it one directory up/down from where it should be. As for sudo, there is no situation in which you should use sudo with pip.
- Normal_gaussian 8y ago
  Node resolves relative to the path of the file requiring the module, not relative to the process cwd.
erfgh 8y ago
In python you just create a virtualenv with "virtualenv whatever". From then on everything is installed in there. If you want to start fresh, you just delete the virtualenv. How is this difficult?

by umvi 8y ago

cx1000 8y ago

PEP 582 is in the works to make __pypackages__ the Python equivalent of node_modules. Not sure if it will get accepted but I hope it does. > a mechanism to automatically recognize a __pypackages__ directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate "virtual environments". https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/

umvi 8y ago
That would be great! Any way to upvote or otherwise give the equivalent of a GitHub thumbs up to show support for it?
JNRowe 8y ago
That’s a great idea. I’ve just been playing with the implementation from the PEP, and it makes things really easy. The speed of simply extracting a bunch of wheels directly in to `__pypackages__/$PYVER/lib/` is a huge benefit. It behaves well if you symlink from a global tree of extracted packages too, like an even simpler version of pundler¹. If others want to play with it without too much breakage, I massaged the patch on to 3.7². The 3.8 base was a little too big a change for my liking ;) 1. https://github.com/Deepwalker/pundler https://github.com/Deepwalker/pundler 2. https://gist.github.com/JNRowe/502f8772861cdd21bff5144c416e4b31 https://gist.github.com/JNRowe/502f8772861cdd21bff5144c416e4... Edit: It’s a third of a second to stand up a complete environment with 63 wheels in its dependency tree for the project I'm playing with right now.

wirrbel 8y ago

Basically it seems you are lacking only two shell commands. virtualenv venv . venv/bin/activate Then everything works with pip pip install numpy I actually prefer the pipenv way (or the virtualenvwrapper way) of putting virtualenvs into a dedicated location, so that I can wipe them off the hard disk if I want to free some space. The real interesting and occasionally bothering point on managing your virtualenv is resolution of dependencies and compatibility ranges (hard to solve in general).

icebraining 8y ago
At a previous job, we used an even simpler solution: just download everything to a directory (called "deps") with pip install --target deps -r requirements.txt Then running the application with PYTHONPATH=deps python myapp.py This was also simple to integrate with service supervisors, since most make it easy to configure a environment variable.
- jazzyjackson 8y ago
  Making a note of this, didn't know about the target flag, thanks !
- ProblemFactory 8y ago
  Virtualenv tip: you can skip the Bash magic of calling "activate" by providing the path to the python executable. /hello/venvs/myproject/bin/python myscript.py will run myscript.py exactly as if the virtualenv was active, which is easier to configure for cron jobs and services.
- wirrbel 8y ago
  If this works for you, good. I think I would be cautious giving this out as advice because you can run into problems this way with regards to different python versions. The activate script uses the python version used to setup the environment.
chatmasta 8y ago
This doesn’t address his problem of dependencies being “halfway across my computer.”

mlthoughts2018 8y ago

Use conda

androidgirl 8y ago

See, I feel like python has a lot of those benefits and the package management doesn't need to be so complex. Seriously, python package management can be fairly simple. On most of our machines at work, it's just "virtualenv .env && source .env/bin/activate". Then you install your packages and... everything is in one directory, like node_modules in javascript. A clean reinstall is easy from there: remove the .env and just repeat. I feel like pipenv violates KISS, and that a more traditional virtualenv/venv/pyenv setup is the way to go.

umvi 8y ago
> A clean reinstall is easy from there: remove the .env and just repeat. Repeat what, manually doing a bazillion `pip install`? Another nice thing about npm is the packages.json file it creates. This allows us to simply add that to version control and then all the new dev has to do is clone and run `npm install` which reads packages.json and installs everything inside of it. I'm sure there's a way to do it in python but, like everything else I bet it's a non-intuitive multistep process. JavaScript ========== First time installation: npm install express --save Subsequent times: npm install Starting clean: rm -rf node_modules Running application: node myapp.js Python ====== First time installation: virtualenv .env source .env/bin/activate pip3 install requests pip3 freeze > requirements.txt Subsequent times: virtualenv .env source .env/bin/activate pip install -r requirements.txt Starting clean: rm -rf .env [exit bash to clear environment] [start bash] Running application: virtualenv .env source .env/bin/activate python3 myapp.py For JavaScript I typed it all out from memory. For Python I had to consult StackOverflow because I couldn't remember "pip freeze". Yes it can be made simple with use of aliases and such, but like I said in my first post, out-of-the-box cognitive burden is several times greater than, say, npm or yarn.
- toyg 8y ago
  > [exit bash to clear environment] Wat? just deactivate before you rm. Please do not talk about things you clearly don't know well enough (which is patently the case if you can't rememer freeze and deactivate). > Out-of-the-box cognitive burden is several times greater You got any stats on that, or is it just your opinion? Because to me, the completely counter-intuitive --save parameter is much more painful to remember. The venv/pip workflow is not perfect, but what you've described is not the problem.

Evidlo 8y ago

I don't get the node way of doing things. Why should the availability of modules be dependent on the current working directory? Seems brittle to me that your app can't find its libraries because you ran it one directory up/down from where it should be. As for sudo, there is no situation in which you should use sudo with pip.

Normal_gaussian 8y ago
Node resolves relative to the path of the file requiring the module, not relative to the process cwd.

erfgh 8y ago

In python you just create a virtualenv with "virtualenv whatever". From then on everything is installed in there. If you want to start fresh, you just delete the virtualenv. How is this difficult?