5 ms·

I've been using Python as my go-to language for a while. It works well and I feel like I understand the basics, but pipenv, pyenv, pip etc. seem needlessly com

by 7ewis 8y ago

I've been using Python as my go-to language for a while. It works well and I feel like I understand the basics, but pipenv, pyenv, pip etc. seem needlessly complex. Especially when I'm trying to create a serverless framework package, or upload a zip to AWS Lambda. Never know what tool to use to install packages, or how to specify the python version... or even change the python version on my Mac. Homebrew Python, system python, virtualenv? Would love it if I could find one simple way to do all of this, but always seem to run into issues and have to use another method. How do the professionals do it? Do you know if any good guides?

Klathmon 8y ago
This doesn't answer your questions, but I firmly believe this is the major reason node took off in popularity. Npm let you install packages minutes after being installed, and they went right into a folder right there, and they didn't conflict with any other packages on your system. I was like you and played with python for years but never anything larger than small scripts, and every time I went to do "real" python I'd hit a wall where you were. npm was just so easy, so simple, so fast that it won me over. And it was years before I ever setup a virtualenv, and even then I'm still at this point not entirely sure how it works and what it's doing despite having published a pip package.
- qudat 8y ago
  I'll echo this statement as someone who has published packages both to npm as well as pypi. There's no contest, npm is so much simpler as a maintainer. For anyone wondering why, no one should go beyond the confusing requirements.txt and setup.py parity madness. Furthermore, try uploading non-python files to pypi and explain to me why it has to be this difficult? The answer coming swiftly from npm: it doesn't.
- nahtnam 8y ago
  I don't think many people here will agree with you but I do. I use Javascript for almost everything I do because "It just works." Assuming you are running Node 6.0+ and NPM 4+ you'll almost never run into an issue unless you are using something that needs some native packages installed.
  - btashton 8y ago
    But the native library issue is what makes it hard for Python as well. A large part of conda existing is to help data science people not have to worry about compiling native libraries for numpy or whatever ML library they happen to be using, since it bundles these libraries.
  - TeMPOraL 8y ago
    Or update packages? NPM land is the only place I really have to be wary about package versions, because otherwise, even when following 6-months old tutorials you'll end up with nothing working due to breaking changes between library versions. And there are just so many dependencies to keep track of...
- tekromancr 8y ago
  On the other hand, as someone who knows a little js from the bad old days but has been doing python for over a decade, trying to get up and running with a modern js environment seems totally cray cray to me. Usually, for me, I start by installing node. Ok, so I guess I need to install npm now. No problem, I guess that's like pip. Oh, I need yarn. It's better? Ok, I guess python had the same growing pains with easy_install. Oh, looks like when I installed npm I used sudo, so now I need to fix file permissions burried somewhere in my file system. Okay, I guess. At least when I get this running things will be a little better. Oh, I need bundler? I guess that makes sense. Gotta configure a build system. Programing languages need build systems if you are creating distributable assets. Makes sense so far. Oh, I guess I should have used webpack. Okay, lessons learned. I guess since I am using ES6 features I need something called babel too. Cute name for a transpiler, btw. Oh no, I guess that 4 month old tutorial I followed that showed me how to set up babel with webpack used the old plug-in names. They are different now. Thats interesting. Gotta move fast, tho. I guess... Huh, it still isn't working. The 6 different config files all look right. Oh, wow. Turns out that the behavior is different if I wrote the config files in json vs yaml vs pure js. Guess I gotta rewrite them in pure js to get at the real shit. Okay, seems like I got all that mess sorted out. Let's add a few libraries to my packages manifest... Huh, the docker containers are taking an awful long time to build suddenly, I wonder why that -- WHY THE FUCK IS node_modules 7.5GB and contains 82000 files, aghhhhgh?!?! (I kid, I kid. But not really) I guess a lot of that is just cultural differences between js and python communities, but as someone who considers themselves a reasonably competent programmer, trying to get onboard just seems like too much. Contrast that with something like Rust, which I have also been trying to learn. Rust is positively delightful by comparison. I spent a cumulative total of 15 mins getting an environment set up. Which means I get more time to actually learn how to work with the language itself, instead of setting up tooling. Maybe it will sort itself out a little as the node community matures and settles on what the right way to do things is, but for now, it's so much work to even get started.
  - btashton 8y ago
    This is exactly how I feel every time I try and use some project using React or other project using "modern" JavaScript/Typescript. I have seen the impressive things people have done with this to solve real problems, but really I have found it to be super unapproachable, and I say this as someone that has picked up a fair amount of low and high level langauges. I want to like it, but I get turned away before I have written any code.
- dpe82 8y ago
  Along the same line, I think this is also a big part of why Perl was so popular back in the day. CPAN was to Perl what NPM is today. It made finding and installing modules in a local user project directory dead simple which made getting started on a project frictionless. Though I have to say, CPAN was much better organized and documented...
avip 8y ago
Personally, I do it with docker. I don't bother with venvs, because python is just not designed to be self-contained (or maybe it's a failed implementation of such design). As to lambda - using serverless and this https://github.com/UnitedIncome/serverless-python-requirements https://github.com/UnitedIncome/serverless-python-requiremen... may help you.
aerovistae 8y ago
You are not alone, python has been one of my two main languages for 7 years and I still am in total accord with you on everything you just said.
landon32 8y ago
I use pipenv for this. I found it sometimes annoying but generally sufficient. Pyenv and pip both seemed too weak or complex. Pipenv was just simple. I would only write code within my pipenv environment, and then when I deployed the code pipenv would handle package management safely. Never did anything huge with it though. Just small web apps hosted on pythonanywhere.
- dserodio 8y ago
  Specifying dependencies when you're writing a library is still a mess, because pipenv isn't recommended for libraries. npm works the same way for both apps and libraries.
  - mcintyre1994 8y ago
    I saw poetry mentioned a few days ago. Haven't tried it yet but it tries to fix this: https://github.com/sdispater/poetry https://github.com/sdispater/poetry
    - mintplant 8y ago
      Yes, I've uses poetry and it's great. I think it and the pyproject.toml standard will become the solutions to all the problems stated above.
      - dserodio 8y ago
        Thanks, I'll give it a try. That's still a mess compared to npm, which is a single tool you can use for app and libs, in constrast with easy_install/setup.py/pip/pipenv/poetry/...
        mintplant 8y ago
        The benefit of Poetry is that it is a single tool for apps and libs, like npm.
gautamdivgi 8y ago
I use pyenv - https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv. Allows me to manage multiple versions of python and switch between them. The only pain I have encountered is the pip cache.
ploxiln 8y ago
You can get very far with any python on your system, and just the standard library. Don't bother with pipenv or pyenv. Once you have experience solving problems with the standard library, learn where pip puts things (probably /usr/local/lib/pythonX.Y/site-packages), use "pip freeze" to list current versions of stuff, and keep it tidy. Don't make a big mess, it won't be a mess. One nifty trick is using the (built-in) subprocess modules to exec some other tool, like rsync or ssh. I recently had sufficient motivation to remove a dependency on the "awscli" package from my deploy system, and implemented an aws api request signing function in about 50 lines of python3.5 using only the standard library. (For bootstrap convenience, my deploy system depends only python3.5 and some standard cli utilities.) For some non-trivial web services, I use tornado. Running under python3, it depends on nothing else, and can take you very far. (Running under python2.7 it has 4 small dependencies.)
jaxr 8y ago
I find fades awesome for simple scripting in Python. https://fades.readthedocs.io/en/stable/ https://fades.readthedocs.io/en/stable/

by 7ewis 8y ago

Klathmon 8y ago

This doesn't answer your questions, but I firmly believe this is the major reason node took off in popularity. Npm let you install packages minutes after being installed, and they went right into a folder right there, and they didn't conflict with any other packages on your system. I was like you and played with python for years but never anything larger than small scripts, and every time I went to do "real" python I'd hit a wall where you were. npm was just so easy, so simple, so fast that it won me over. And it was years before I ever setup a virtualenv, and even then I'm still at this point not entirely sure how it works and what it's doing despite having published a pip package.

qudat 8y ago
I'll echo this statement as someone who has published packages both to npm as well as pypi. There's no contest, npm is so much simpler as a maintainer. For anyone wondering why, no one should go beyond the confusing requirements.txt and setup.py parity madness. Furthermore, try uploading non-python files to pypi and explain to me why it has to be this difficult? The answer coming swiftly from npm: it doesn't.
nahtnam 8y ago
I don't think many people here will agree with you but I do. I use Javascript for almost everything I do because "It just works." Assuming you are running Node 6.0+ and NPM 4+ you'll almost never run into an issue unless you are using something that needs some native packages installed.
- btashton 8y ago
  But the native library issue is what makes it hard for Python as well. A large part of conda existing is to help data science people not have to worry about compiling native libraries for numpy or whatever ML library they happen to be using, since it bundles these libraries.
- TeMPOraL 8y ago
  Or update packages? NPM land is the only place I really have to be wary about package versions, because otherwise, even when following 6-months old tutorials you'll end up with nothing working due to breaking changes between library versions. And there are just so many dependencies to keep track of...
tekromancr 8y ago
On the other hand, as someone who knows a little js from the bad old days but has been doing python for over a decade, trying to get up and running with a modern js environment seems totally cray cray to me. Usually, for me, I start by installing node. Ok, so I guess I need to install npm now. No problem, I guess that's like pip. Oh, I need yarn. It's better? Ok, I guess python had the same growing pains with easy_install. Oh, looks like when I installed npm I used sudo, so now I need to fix file permissions burried somewhere in my file system. Okay, I guess. At least when I get this running things will be a little better. Oh, I need bundler? I guess that makes sense. Gotta configure a build system. Programing languages need build systems if you are creating distributable assets. Makes sense so far. Oh, I guess I should have used webpack. Okay, lessons learned. I guess since I am using ES6 features I need something called babel too. Cute name for a transpiler, btw. Oh no, I guess that 4 month old tutorial I followed that showed me how to set up babel with webpack used the old plug-in names. They are different now. Thats interesting. Gotta move fast, tho. I guess... Huh, it still isn't working. The 6 different config files all look right. Oh, wow. Turns out that the behavior is different if I wrote the config files in json vs yaml vs pure js. Guess I gotta rewrite them in pure js to get at the real shit. Okay, seems like I got all that mess sorted out. Let's add a few libraries to my packages manifest... Huh, the docker containers are taking an awful long time to build suddenly, I wonder why that -- WHY THE FUCK IS node_modules 7.5GB and contains 82000 files, aghhhhgh?!?! (I kid, I kid. But not really) I guess a lot of that is just cultural differences between js and python communities, but as someone who considers themselves a reasonably competent programmer, trying to get onboard just seems like too much. Contrast that with something like Rust, which I have also been trying to learn. Rust is positively delightful by comparison. I spent a cumulative total of 15 mins getting an environment set up. Which means I get more time to actually learn how to work with the language itself, instead of setting up tooling. Maybe it will sort itself out a little as the node community matures and settles on what the right way to do things is, but for now, it's so much work to even get started.
- btashton 8y ago
  This is exactly how I feel every time I try and use some project using React or other project using "modern" JavaScript/Typescript. I have seen the impressive things people have done with this to solve real problems, but really I have found it to be super unapproachable, and I say this as someone that has picked up a fair amount of low and high level langauges. I want to like it, but I get turned away before I have written any code.
dpe82 8y ago
Along the same line, I think this is also a big part of why Perl was so popular back in the day. CPAN was to Perl what NPM is today. It made finding and installing modules in a local user project directory dead simple which made getting started on a project frictionless. Though I have to say, CPAN was much better organized and documented...

avip 8y ago

Personally, I do it with docker. I don't bother with venvs, because python is just not designed to be self-contained (or maybe it's a failed implementation of such design). As to lambda - using serverless and this https://github.com/UnitedIncome/serverless-python-requirements https://github.com/UnitedIncome/serverless-python-requiremen... may help you.

aerovistae 8y ago

You are not alone, python has been one of my two main languages for 7 years and I still am in total accord with you on everything you just said.

landon32 8y ago

I use pipenv for this. I found it sometimes annoying but generally sufficient. Pyenv and pip both seemed too weak or complex. Pipenv was just simple. I would only write code within my pipenv environment, and then when I deployed the code pipenv would handle package management safely. Never did anything huge with it though. Just small web apps hosted on pythonanywhere.

dserodio 8y ago
Specifying dependencies when you're writing a library is still a mess, because pipenv isn't recommended for libraries. npm works the same way for both apps and libraries.
- mcintyre1994 8y ago
  I saw poetry mentioned a few days ago. Haven't tried it yet but it tries to fix this: https://github.com/sdispater/poetry https://github.com/sdispater/poetry
  - mintplant 8y ago
    Yes, I've uses poetry and it's great. I think it and the pyproject.toml standard will become the solutions to all the problems stated above.
    - dserodio 8y ago
      Thanks, I'll give it a try. That's still a mess compared to npm, which is a single tool you can use for app and libs, in constrast with easy_install/setup.py/pip/pipenv/poetry/...
      - mintplant 8y ago
        The benefit of Poetry is that it is a single tool for apps and libs, like npm.

gautamdivgi 8y ago

I use pyenv - https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv. Allows me to manage multiple versions of python and switch between them. The only pain I have encountered is the pip cache.

ploxiln 8y ago

You can get very far with any python on your system, and just the standard library. Don't bother with pipenv or pyenv. Once you have experience solving problems with the standard library, learn where pip puts things (probably /usr/local/lib/pythonX.Y/site-packages), use "pip freeze" to list current versions of stuff, and keep it tidy. Don't make a big mess, it won't be a mess. One nifty trick is using the (built-in) subprocess modules to exec some other tool, like rsync or ssh. I recently had sufficient motivation to remove a dependency on the "awscli" package from my deploy system, and implemented an aws api request signing function in about 50 lines of python3.5 using only the standard library. (For bootstrap convenience, my deploy system depends only python3.5 and some standard cli utilities.) For some non-trivial web services, I use tornado. Running under python3, it depends on nothing else, and can take you very far. (Running under python2.7 it has 4 small dependencies.)

jaxr 8y ago

I find fades awesome for simple scripting in Python. https://fades.readthedocs.io/en/stable/ https://fades.readthedocs.io/en/stable/