9 ms·

It’s too fucking hard to setup python with all its pyenv, anaconda, poetry, pip requirements.txt and what-not, that’s the best use case for Docker here. Get it

It’s too fucking hard to setup python with all its pyenv, anaconda, poetry, pip requirements.txt and what-not, that’s the best use case for Docker here. Get it to work once and forget about it.

mherrmann 6y ago
I also dislike all those tools you mention. So often, I feel like devs use a tool "because it's cool" but fail to appreciate the complexity it adds - not just for themselves but for others. The frustration expressed by your comment is a perfect example of this. In my ideal world, everybody would just use what comes with vanilla Python: python3 -m venv venv source venv/bin/activate (Or `call` on Win) pip install -Ur requirements.txt
- procinct 6y ago
  Venv is great for keeping dependencies separate but if your local machine doesn’t have the same version of Python as your server then you’ll need pyenv.
  - RMPR 6y ago
    Not necessarily, you can use a CI pipeline to verify that your project can be build with many Python versions. This is the workflow I use here[0]. It makes catching and fixing breaking changes easier. Plus, I'm confident the core team is not likely to introduce a painful breaking change (think Python 2 -> 3) soon[1] 0: https://github.com/rmpr/atbswp https://github.com/rmpr/atbswp 1: https://mobile.twitter.com/gvanrossum/status/1306082472443084801?lang=en https://mobile.twitter.com/gvanrossum/status/130608247244308...
    - imnotjames 6y ago
      That's great for libraries but isn't it a pretty huge cost for an app you only really are going to be running on one version at a time? Dealing with BC breaks for versions you aren't intending to run in production seems like unnecessary overhead.
- 0x426577617265 6y ago
  I agree. Python's virtual environments are awesome and easy to use. On a new server just start with apt install python3-venv Really easy to source the environments as needed during automation, etc.
- ducktective 6y ago
  and pipx for managing python "apps" like youtube-dl, ipython, meson, ... https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/
  - duggan 6y ago
    I’ve been doing a manual version of this for years, creating a shell shim wrapping a venv for cli tools. This is much neater, thanks!
  - mike-cardwell 6y ago
    I prefer my system for running youtube-dl in an isolated environment without having to install all of its dependencies on the host. Ansible writes a bash script at ~/bin/youtube-dl containing: #!/usr/bin/env bash set -e IMAGE_NAME="grepular/youtube-dl" # Build the image if it does not exist if [[ $(podman images --filter "reference=$IMAGE_NAME" -q) == "" ]]; then podman build -t "$IMAGE_NAME" -<<EOF >&2 FROM python:3-slim RUN python3 -m pip --no-cache-dir install youtube_dl ENTRYPOINT ["youtube-dl"] EOF fi podman run --net host -i --rm -v "$PWD:/app" -w /app "$IMAGE_NAME" "$@"
    - ORioN63 6y ago
      QQ, aren't the created files owned by root?
      - mike-cardwell 6y ago
        Nope. I'm using podman rather than docker so they're owned by the user that runs the script. The root user isn't involved. If I were using docker, I'd add `-u "$(id -u):$(id -g)"` to the docker args to deal with that issue.
    - wp381640 6y ago
      Pipx is pretty much that - installs a package into its own venv and links script entry points to .local/bin
- bschwindHN 6y ago
  I also dislike all the commands you just mentioned. In my ideal world, everybody would just ./the-program
  - mherrmann 6y ago
    And then ./the-program breaks and someone has to read its source code. Then, the fewer dependencies the-program has, the easier it will be to fix.
    - bschwindHN 6y ago
      Sounds wonderful compared to dealing with Python tooling.
- avaika 6y ago
  It's all fun until one of your requirements needs a billion build dependencies to be available on a machine. How do you solve that with pyenv? Docker does it for you.
  - mherrmann 6y ago
    I agree. I suspect Docker does make sense in such projects. In my pretty vanilla web development, I just usually don't experience this.
- Aeolun 6y ago
  I dunno, these magic incantations make a lot less sense to me than: pipenv init Pipenv install Maybe it’s because I’m used to npm and composer?
- wp381640 6y ago
  You must be the only person who ever runs that on your own code because you skipped the pyenv step
marban 6y ago
PyEnv + Poetry — Couldn't be easier IMO.
- mherrmann 6y ago
  Except plain old virtual environments instead of poetry. And pyenv only if really really necessary, and not in prod.
  - raverbashing 6y ago
    After knowing Poetry, I can honestly say I don't miss requirements.txts at all But true, you weren't supposed to need it. But managing projects with interlocked dependencies get old fast.
  - imiric 6y ago
    Ah, see, this is the perfect example of why Docker helps, especially with Python: consistency. The team would no longer have to make these deployment decisions and argue about which tool is a better fit. They'd make the decision once, hopefully follow best Docker practices (unprivileged user, multi-stage builds, etc.), and have documentation available for how to integrate the setup with IDEs, work with volumes, etc. Once this initial adoption hurdle is overcome, IME the productivity gains are greater than the issues of dealing with Docker. It becomes trivial to setup CI/CD, onboard new developers and integrate the app into other workflows. Docker and containers in general have become mature enough to prove their use case and benefits, so the cargo cult argument doesn't hold weight for me.
ramraj07 6y ago
Few hours of annoyance per developer in the beginning vs. a few minutes every time you run tests or when you debug. Also add in the overhead of configuring a debugging system where you work in an ide and debug code running in a docker image. Doable (though many of my fellow Devs have just given up) but also takes time, probably more than to run pip and get psycopg2 installed on Macs. Also the majority of engineers probably don't even know docker that we'll (definitely far worse than they think they do, just like with k8s). The number of times I've seen Devs running webservers as root in docker containers is amazing. My team insists on using Docker and against my better personal judgement I let it, but I set up the code to run locally without it. If I call the shots I'll not use docker absolutely.
- matsemann 6y ago
  If you think it's okay to spend a few hours setting stuff up per developer, why can't you just invest an hour in learning how to debug with docker?
  - ramraj07 6y ago
    Because I have, my colleagues have, in fact we probably spent more than an hour and just decided it's not worth it. I decided it's not worth it because I had a way to debug, while others decided they will just code without proper debugging tools because they only want to work through docker.
    - hurflmurfl 6y ago
      My (limited) experience debugging Typescript/Node, Typescript/Browser, PHP using IntelliJ and Docker has been pretty much like this: 1. Open a port on the container for debugging 2. Tell the debugger in the container what's the port it should use (if you're not using the default one) 3. Tell the IDE what port it can use to connect to the debugger (if you're not using the default). 4. Debug it! For things like Typescript you might need some extra trickery because it's all transpiled, so you'll need to make sure your sourcemaps are set up correctly, but that's not overly difficult. It did take me several hours to work everything out and write a README so that every time we get a new hire / someone sets up a new PC they can just follow the instructions. I'd say it was time well spent.
odiroot 6y ago
Is it? I've been riding on bare virtualenv and pip since around 2014, maybe longer. Inside containers and without Docker alike.
- stefano 6y ago
  Same. The only issue I had with virtualenv was when I copied one to a different directory and it didn't work. It turns out you can't do that. Everything else has always worked fine, and I've been using it professionally for 10 years.
kissgyorgy 6y ago
No it's not. It's literally just two commands to get started with poetry, and after that, one command to add a dependency and it's pretty easy to run simple projects locally.

It’s too fucking hard to setup python with all its pyenv, anaconda, poetry, pip requirements.txt and what-not, that’s the best use case for Docker here. Get it to work once and forget about it.

mherrmann 6y ago

I also dislike all those tools you mention. So often, I feel like devs use a tool "because it's cool" but fail to appreciate the complexity it adds - not just for themselves but for others. The frustration expressed by your comment is a perfect example of this. In my ideal world, everybody would just use what comes with vanilla Python: python3 -m venv venv source venv/bin/activate (Or `call` on Win) pip install -Ur requirements.txt

procinct 6y ago
Venv is great for keeping dependencies separate but if your local machine doesn’t have the same version of Python as your server then you’ll need pyenv.
- RMPR 6y ago
  Not necessarily, you can use a CI pipeline to verify that your project can be build with many Python versions. This is the workflow I use here[0]. It makes catching and fixing breaking changes easier. Plus, I'm confident the core team is not likely to introduce a painful breaking change (think Python 2 -> 3) soon[1] 0: https://github.com/rmpr/atbswp https://github.com/rmpr/atbswp 1: https://mobile.twitter.com/gvanrossum/status/1306082472443084801?lang=en https://mobile.twitter.com/gvanrossum/status/130608247244308...
  - imnotjames 6y ago
    That's great for libraries but isn't it a pretty huge cost for an app you only really are going to be running on one version at a time? Dealing with BC breaks for versions you aren't intending to run in production seems like unnecessary overhead.
0x426577617265 6y ago
I agree. Python's virtual environments are awesome and easy to use. On a new server just start with apt install python3-venv Really easy to source the environments as needed during automation, etc.
ducktective 6y ago
and pipx for managing python "apps" like youtube-dl, ipython, meson, ... https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/
- duggan 6y ago
  I’ve been doing a manual version of this for years, creating a shell shim wrapping a venv for cli tools. This is much neater, thanks!
- mike-cardwell 6y ago
  I prefer my system for running youtube-dl in an isolated environment without having to install all of its dependencies on the host. Ansible writes a bash script at ~/bin/youtube-dl containing: #!/usr/bin/env bash set -e IMAGE_NAME="grepular/youtube-dl" # Build the image if it does not exist if [[ $(podman images --filter "reference=$IMAGE_NAME" -q) == "" ]]; then podman build -t "$IMAGE_NAME" -<<EOF >&2 FROM python:3-slim RUN python3 -m pip --no-cache-dir install youtube_dl ENTRYPOINT ["youtube-dl"] EOF fi podman run --net host -i --rm -v "$PWD:/app" -w /app "$IMAGE_NAME" "$@"
  - ORioN63 6y ago
    QQ, aren't the created files owned by root?
    - mike-cardwell 6y ago
      Nope. I'm using podman rather than docker so they're owned by the user that runs the script. The root user isn't involved. If I were using docker, I'd add `-u "$(id -u):$(id -g)"` to the docker args to deal with that issue.
  - wp381640 6y ago
    Pipx is pretty much that - installs a package into its own venv and links script entry points to .local/bin
bschwindHN 6y ago
I also dislike all the commands you just mentioned. In my ideal world, everybody would just ./the-program
- mherrmann 6y ago
  And then ./the-program breaks and someone has to read its source code. Then, the fewer dependencies the-program has, the easier it will be to fix.
  - bschwindHN 6y ago
    Sounds wonderful compared to dealing with Python tooling.
avaika 6y ago
It's all fun until one of your requirements needs a billion build dependencies to be available on a machine. How do you solve that with pyenv? Docker does it for you.
- mherrmann 6y ago
  I agree. I suspect Docker does make sense in such projects. In my pretty vanilla web development, I just usually don't experience this.
Aeolun 6y ago
I dunno, these magic incantations make a lot less sense to me than: pipenv init Pipenv install Maybe it’s because I’m used to npm and composer?
wp381640 6y ago
You must be the only person who ever runs that on your own code because you skipped the pyenv step

marban 6y ago

PyEnv + Poetry — Couldn't be easier IMO.

mherrmann 6y ago
Except plain old virtual environments instead of poetry. And pyenv only if really really necessary, and not in prod.
- raverbashing 6y ago
  After knowing Poetry, I can honestly say I don't miss requirements.txts at all But true, you weren't supposed to need it. But managing projects with interlocked dependencies get old fast.
- imiric 6y ago
  Ah, see, this is the perfect example of why Docker helps, especially with Python: consistency. The team would no longer have to make these deployment decisions and argue about which tool is a better fit. They'd make the decision once, hopefully follow best Docker practices (unprivileged user, multi-stage builds, etc.), and have documentation available for how to integrate the setup with IDEs, work with volumes, etc. Once this initial adoption hurdle is overcome, IME the productivity gains are greater than the issues of dealing with Docker. It becomes trivial to setup CI/CD, onboard new developers and integrate the app into other workflows. Docker and containers in general have become mature enough to prove their use case and benefits, so the cargo cult argument doesn't hold weight for me.

ramraj07 6y ago

Few hours of annoyance per developer in the beginning vs. a few minutes every time you run tests or when you debug. Also add in the overhead of configuring a debugging system where you work in an ide and debug code running in a docker image. Doable (though many of my fellow Devs have just given up) but also takes time, probably more than to run pip and get psycopg2 installed on Macs. Also the majority of engineers probably don't even know docker that we'll (definitely far worse than they think they do, just like with k8s). The number of times I've seen Devs running webservers as root in docker containers is amazing. My team insists on using Docker and against my better personal judgement I let it, but I set up the code to run locally without it. If I call the shots I'll not use docker absolutely.

matsemann 6y ago
If you think it's okay to spend a few hours setting stuff up per developer, why can't you just invest an hour in learning how to debug with docker?
- ramraj07 6y ago
  Because I have, my colleagues have, in fact we probably spent more than an hour and just decided it's not worth it. I decided it's not worth it because I had a way to debug, while others decided they will just code without proper debugging tools because they only want to work through docker.
  - hurflmurfl 6y ago
    My (limited) experience debugging Typescript/Node, Typescript/Browser, PHP using IntelliJ and Docker has been pretty much like this: 1. Open a port on the container for debugging 2. Tell the debugger in the container what's the port it should use (if you're not using the default one) 3. Tell the IDE what port it can use to connect to the debugger (if you're not using the default). 4. Debug it! For things like Typescript you might need some extra trickery because it's all transpiled, so you'll need to make sure your sourcemaps are set up correctly, but that's not overly difficult. It did take me several hours to work everything out and write a README so that every time we get a new hire / someone sets up a new PC they can just follow the instructions. I'd say it was time well spent.

odiroot 6y ago

Is it? I've been riding on bare virtualenv and pip since around 2014, maybe longer. Inside containers and without Docker alike.

stefano 6y ago
Same. The only issue I had with virtualenv was when I copied one to a different directory and it didn't work. It turns out you can't do that. Everything else has always worked fine, and I've been using it professionally for 10 years.

kissgyorgy 6y ago

No it's not. It's literally just two commands to get started with poetry, and after that, one command to add a dependency and it's pretty easy to run simple projects locally.