4 ms·

On my computer, I only use Python software that is in the Debian repos. So all dependencies are handled by Debian. Everything else I run in a Docker container.

On my computer, I only use Python software that is in the Debian repos. So all dependencies are handled by Debian. Everything else I run in a Docker container. This way I never need a venv. I hope these changes will not make my workflow harder. Running 3rd party software in containers not only makes dependency handling easier. It also is a security thing. Can you really trust the whole dependency tree below any python code that you want to run? I prefer to keep it separate from my main OS.

BiteCode_dev 5y ago
No, it fact it will make your workflow easier: no matter what, no script using pip will break your install.
- JonathanBeuys 5y ago
  I don't know if I actually have something you would call an "install". I run 3rd party software in fresh Debian containers. So there is nothing in there I would be afraid to "break".
  - BiteCode_dev 5y ago
    The container has a system python stdlib, and it's used by apt. If it breaks, everything breaks. I've destroyed it twice, with style I must say. If any script, or yourself, run "pip install" using admin rights (the default on a lot of container techs), you have a chance to break it.
    - JonathanBeuys 5y ago
      "Chance to break" what? It is not as if I manually work in a terminal inside a container that is somehow permanent and valuable. I just once create a dockerfile with the commands to set up the application I want to use. That's it. From then on, I use that container whenever I want to use that application.
      - BiteCode_dev 5y ago
        If pip installs something that has a dependancy with the same name than any of the packages you install, it can cause a conflict, because by default, they will override each other.
- quietbritishjim 5y ago
  A simple workflow that doesn't use Docker: * Exactly as parent comment says, feel free to use the system Python but install packages with the system package manager (sudo apt install python3-numpy etc.). NEVER run sudo pip anything. * Use virtual environments (either venv or virtualenv) and do not use sudo for this. Keep these within your home directory (either within or next to relevant project directory if that's your preference, or within something like e.g. ~/envs/). So using Docker is making your workflow harder IMO because it's harder than the above and doesn't offer any benefits. Not sure what you mean by "no script using pip will break your install". Why would you run pip from a script? Once you've activated your virtual environment, you'd just run pip install -r requirements.txt. And what is "your install" that you'd break? You can't break your system Python with pip because pip doesn't have sudo access to break it. And it can't contaminate any virtual environments except the one that's active because it doesn't even know about them.
  - BiteCode_dev 5y ago
    While the best practice is to use a venv even in docker images, docker users often believe that because they are into an isolated container, they can pip install their heart away directly in it. Also, docker image default users are often root. They may also want to install tooling that they want to use outside of a venv, such as a command line tools, and do so outside of venv, not even using pipx. Or they may just don't know about virtualenv, or follow a bad tutorial/documentation. Or just do one of those curl stuff | bash that pip install something. Or think they are in a venv, but they are not (because they don't use -m) and it doesn't work so they try to solve their problem with a bazooka. This is also true outside of docker. Bottom line, plenty of people do the wrong thing, and since we can't prevent all of them from doing so, or educate the entire world, the next best thing is to mitigate the impact. All, in all, I still think out best bet is something like pyflow (see my comment about it). It solves so many problems at once, including this once. I hope it becomes the de factor standard.
    - quietbritishjim 5y ago
      I think you misunderstood the original comment. They seem to be using Docker in the first place because they think it will help simplify managing Python dependencies. The answer is surely that their life would be simpler if they didn't use Docker at all, and instead use venv/virtalenv on directories within their home directory. Your comment about how to use pip if you're going to use Docker doesn't apply - the best answer is not to use Docker at all (if you weren't going to anyway). There's not point talking about pyflow or some other tool to simplify your Python dependencies until you've said me _why_ I need another tool. What's wrong with the plan I listed above (with no Docker)? If I want a new set of dependencies, I just make a new virtal env, activate it and pip install away. If I want to get rid of it, I `rm -r ~/envs/foo`. What could be simpler than that? There's just no problem to solve in the first place.
dagw 5y ago
Running 3rd party software in containers not only makes dependency handling easier. True, but it does making using the software together with any other software harder. Any variation on foo data.csv | bar becomes a lot more of a pain when foo and bar live in two different Docker containers.

BiteCode_dev 5y ago

No, it fact it will make your workflow easier: no matter what, no script using pip will break your install.

JonathanBeuys 5y ago
I don't know if I actually have something you would call an "install". I run 3rd party software in fresh Debian containers. So there is nothing in there I would be afraid to "break".
- BiteCode_dev 5y ago
  The container has a system python stdlib, and it's used by apt. If it breaks, everything breaks. I've destroyed it twice, with style I must say. If any script, or yourself, run "pip install" using admin rights (the default on a lot of container techs), you have a chance to break it.
  - JonathanBeuys 5y ago
    "Chance to break" what? It is not as if I manually work in a terminal inside a container that is somehow permanent and valuable. I just once create a dockerfile with the commands to set up the application I want to use. That's it. From then on, I use that container whenever I want to use that application.
    - BiteCode_dev 5y ago
      If pip installs something that has a dependancy with the same name than any of the packages you install, it can cause a conflict, because by default, they will override each other.
quietbritishjim 5y ago
A simple workflow that doesn't use Docker: * Exactly as parent comment says, feel free to use the system Python but install packages with the system package manager (sudo apt install python3-numpy etc.). NEVER run sudo pip anything. * Use virtual environments (either venv or virtualenv) and do not use sudo for this. Keep these within your home directory (either within or next to relevant project directory if that's your preference, or within something like e.g. ~/envs/). So using Docker is making your workflow harder IMO because it's harder than the above and doesn't offer any benefits. Not sure what you mean by "no script using pip will break your install". Why would you run pip from a script? Once you've activated your virtual environment, you'd just run pip install -r requirements.txt. And what is "your install" that you'd break? You can't break your system Python with pip because pip doesn't have sudo access to break it. And it can't contaminate any virtual environments except the one that's active because it doesn't even know about them.
- BiteCode_dev 5y ago
  While the best practice is to use a venv even in docker images, docker users often believe that because they are into an isolated container, they can pip install their heart away directly in it. Also, docker image default users are often root. They may also want to install tooling that they want to use outside of a venv, such as a command line tools, and do so outside of venv, not even using pipx. Or they may just don't know about virtualenv, or follow a bad tutorial/documentation. Or just do one of those curl stuff | bash that pip install something. Or think they are in a venv, but they are not (because they don't use -m) and it doesn't work so they try to solve their problem with a bazooka. This is also true outside of docker. Bottom line, plenty of people do the wrong thing, and since we can't prevent all of them from doing so, or educate the entire world, the next best thing is to mitigate the impact. All, in all, I still think out best bet is something like pyflow (see my comment about it). It solves so many problems at once, including this once. I hope it becomes the de factor standard.
  - quietbritishjim 5y ago
    I think you misunderstood the original comment. They seem to be using Docker in the first place because they think it will help simplify managing Python dependencies. The answer is surely that their life would be simpler if they didn't use Docker at all, and instead use venv/virtalenv on directories within their home directory. Your comment about how to use pip if you're going to use Docker doesn't apply - the best answer is not to use Docker at all (if you weren't going to anyway). There's not point talking about pyflow or some other tool to simplify your Python dependencies until you've said me _why_ I need another tool. What's wrong with the plan I listed above (with no Docker)? If I want a new set of dependencies, I just make a new virtal env, activate it and pip install away. If I want to get rid of it, I `rm -r ~/envs/foo`. What could be simpler than that? There's just no problem to solve in the first place.

dagw 5y ago

Running 3rd party software in containers not only makes dependency handling easier. True, but it does making using the software together with any other software harder. Any variation on foo data.csv | bar becomes a lot more of a pain when foo and bar live in two different Docker containers.