7 ms·

Show HN: Python virtual environment, but backed by Docker

78 points by se7entyse7en 7y ago

ryall 7y ago
I've tried going the docker environment path before but always hit a roadblock at the actual development stage. VSCode has no idea about your env because it's effectively non-local. So you lose auto-complete, linting etc. How well does pydocenv handle those kind of issues?
- se7entyse7en 7y ago
  First of all, thanks for your interest! > VSCode has no idea about your env because it's effectively non-local. This is an issue that I'm aware of, and I actually didn't think about a possible solution yet. Currently, it is in a VERY early stage, but I wanted to share it to get feedback from the community and how much interest the project could raise. The only things that I thought were adding support for example for aliasing of some commands such as running `flake8` will actually call `flake8` inside the container. But it's still very fuzzy in my head.
  - pbalau 7y ago
    Not sure how hard this would be, instead of running "flake8 blabla", one can run "docker run bla bla". Or even docker exec bla bla if you have the container already running. And by "you run" I mean vscode does that for you, via an extension.
    - se7entyse7en 7y ago
      That could be an idea, so in practice, you move the responsibility to the editor which I think is actually the correct thing to do. Regarding VSCode specifically I don't know, as I don't use it.
    - haney 7y ago
      This is almost exactly how Pycharm works with testing/auto complete/coverage. It has support to use a a particular interpreter, specific virtual env, or it lets you say “my interpreter is inside this docker container”, which is even allowed to be on a remote host (although I’ve always used docker running locally).
      - lpasselin 7y ago
        I find using pycharm with remote docker painful. If you need unusual docker arguments not included in the pycharm Gui, it sucks. They did a good job but it doesn't seem complete yet.
  - ericol 7y ago
    What I've seen around this last days, and seems that is not an easy feature to implement, is providing Python the posibility of live debugging (Something that, for instance, PHP has long ago acquired with xDebug). Im not saying one is better than the other, I'm not going into the discussion "print is better than live" (Which I don't agree with, BTW). I just find it amusing that a language as mature as Python does not have this.
  - pard68 7y ago
    Would vim or emacs remote editing options work? Another option is to use an editor that has a server/client model. And for VSCode there is always code-server or a similar project that runs vscode remotely.
- henryfjordan 7y ago
  Consider running VSCode in docker? https://coder.com/ https://coder.com/
- BerislavLopac 7y ago
  I'm not quite sure what is really the advantage of this approach. Docker is essentially a process isolation tool, and is well suited for scenarios when you need to isolate a running process, but less when you need to isolate an environment. If you need to isolate a part of the file system, e.g. if your environment only concerns Python files, the standard virtualenv (or venv) is perfectly powerful; if you need to isolate other parts of the system, such as memory, installed binaries and the such, your best bet is vagrant (combined with something like VirtualBox). And both of them work fine with a local IDE. Docker works best for deployment, when you want to set up a running process in a consistent manner, and its closed-box approach works really well in that scenario. But in development you generally want more flexibility.
catern 7y ago
Personally, I use Nix to create my Python virtual environments. That gets the isolation, replicability, and deployability that the author wants, without the heavyweight infrastructure of Docker.
- ggm 7y ago
  Got a show HN ?
  - yorwba 7y ago
    Here's a post by tweag.io about using Nix to manage dependencies of Jupyter notebooks: https://www.tweag.io/posts/2019-02-28-jupyter-with.html https://www.tweag.io/posts/2019-02-28-jupyter-with.html
    - ggm 7y ago
      Awesome! Thanks!
ben509 7y ago
That looks pretty cool! My immediate use for it would be in replacing tox. It's a lot of downloading, but "docker pull python:3.7.3-alpine3.8" is far more reliable and simple than building from source. Especially for someone who just wants to test out a small contribution to a library.
- se7entyse7en 7y ago
  That's a cool use case that I didn't think about! I'm really glad that this project that I started only a couple of weeks ago is actually useful.
- fock 7y ago
  Seems like someone is indeed compiling: https://github.com/docker-library/python/blob/65b5603fdf177c6d90df6ea6f3bd6e23839c63a4/3.7/alpine3.8/Dockerfile https://github.com/docker-library/python/blob/65b5603fdf177c...
Townley 7y ago
I didn't follow how this relates to docker. Does running the pydocenv command modify dependencies stored in the Dockerfile (or a file that pydocenv references within a "pip install -r ___" command inside the Dockerfile)? It seems interesting, in that I respect any library (eg pyenv or pipenv) that tries to solve the issue with pip has compared to npm or cargo, where the installation of a library isn't coupled to the dependency tracking. I just can't see from the examples how this is preferable to changing a requirements.txt file and then rebuilding the docker image
- se7entyse7en 7y ago
  When I have some time I could try adding a `How does it work` section. Under the hood the creation of an environment corresponds to the creation of a docker container (that's how it relates to docker). The docker container is built starting from a the official python docker images as you can see here https://github.com/se7entyse7en/pydockenv/blob/master/pydockenv/commands/pydockenv.py#L32 https://github.com/se7entyse7en/pydockenv/blob/master/pydock.... The activation of the environment corresponds to the start of the container. Hence inside the container, you don't have (don't need) to have any virtual env as the isolation is already provided by the container itself. Every time you install a dependency, this is actually installed inside the container.
  - Spivak 7y ago
    Once I get my environment set up just the way I want in the container how do give it to someone else or make another one? Seems wild to pass around the actual container rather than an image that spawns your environment.
    - se7entyse7en 7y ago
      To share your environment with someone else you can run `pydockenv save` (see here: https://github.com/se7entyse7en/pydockenv/blob/master/pydockenv/commands/pydockenv.py#L98 https://github.com/se7entyse7en/pydockenv/blob/master/pydock...). I'm sorry that any example is present in the README so far. Under the hood, a new docker image is saved from the container corresponding to the currently active environment. A `*.tar.gz` file is created with the image the can be now loaded using `pydockenv load`. Alternatively, you could push that image to Dockerhub for example and pull it from somewhere else. The problem is that currently there's no mechanism to make it recognize to `pydockenv`, but I'm also planning to add the possibility to create an environment from a user-defined image. Using `pydockenv save` will ensure that the environment you'll share will be exactly the same. An alternative that I'm also planning to implement is to use a `toml` file that will simply create a new environment from it. Obviously, in your running container, there could be something not captured in the original `toml` file if you, for example, installed something manually inside the container. But again, it's really in early stage, so all these feedbacks are really welcome!
pritambaral 7y ago
Personally, I use LXC (via LXD) and Emacs's TRAMP. Tramp is smart enough to run development tools (like flake8) on the remote, and I get a full OS to play with, with isolation.
rcarmo 7y ago
I’ve been using and maintaining a set of Python images targeted at doing builds and packaging for a while now: https://github.com/insightfulsystems/alpine-python https://github.com/insightfulsystems/alpine-python (There is also an Ubuntu variant, but it isn’t fully multiarch yet) This kind of thing is really useful when you have to build and deploy more modern/well-known versions of Python than what ships with your distribution, but I also use them as virtualenv replacements (although the workflow is different, since you have to derive a new image for each project)
rich-tea 7y ago
What I've wanted for a while now is to be able to run a single jupyter server but have many kernels running inside different containers. It's possible to have the kernels in different virtualenvs. I wonder if I could achieve that with this.

78 points by se7entyse7en 7y ago

I've tried going the docker environment path before but always hit a roadblock at the actual development stage. VSCode has no idea about your env because it's effectively non-local. So you lose auto-complete, linting etc. How well does pydocenv handle those kind of issues?

se7entyse7en 7y ago
First of all, thanks for your interest! > VSCode has no idea about your env because it's effectively non-local. This is an issue that I'm aware of, and I actually didn't think about a possible solution yet. Currently, it is in a VERY early stage, but I wanted to share it to get feedback from the community and how much interest the project could raise. The only things that I thought were adding support for example for aliasing of some commands such as running `flake8` will actually call `flake8` inside the container. But it's still very fuzzy in my head.
- pbalau 7y ago
  Not sure how hard this would be, instead of running "flake8 blabla", one can run "docker run bla bla". Or even docker exec bla bla if you have the container already running. And by "you run" I mean vscode does that for you, via an extension.
  - se7entyse7en 7y ago
    That could be an idea, so in practice, you move the responsibility to the editor which I think is actually the correct thing to do. Regarding VSCode specifically I don't know, as I don't use it.
  - haney 7y ago
    This is almost exactly how Pycharm works with testing/auto complete/coverage. It has support to use a a particular interpreter, specific virtual env, or it lets you say “my interpreter is inside this docker container”, which is even allowed to be on a remote host (although I’ve always used docker running locally).
    - lpasselin 7y ago
      I find using pycharm with remote docker painful. If you need unusual docker arguments not included in the pycharm Gui, it sucks. They did a good job but it doesn't seem complete yet.
- ericol 7y ago
  What I've seen around this last days, and seems that is not an easy feature to implement, is providing Python the posibility of live debugging (Something that, for instance, PHP has long ago acquired with xDebug). Im not saying one is better than the other, I'm not going into the discussion "print is better than live" (Which I don't agree with, BTW). I just find it amusing that a language as mature as Python does not have this.
- pard68 7y ago
  Would vim or emacs remote editing options work? Another option is to use an editor that has a server/client model. And for VSCode there is always code-server or a similar project that runs vscode remotely.
henryfjordan 7y ago
Consider running VSCode in docker? https://coder.com/ https://coder.com/
BerislavLopac 7y ago
I'm not quite sure what is really the advantage of this approach. Docker is essentially a process isolation tool, and is well suited for scenarios when you need to isolate a running process, but less when you need to isolate an environment. If you need to isolate a part of the file system, e.g. if your environment only concerns Python files, the standard virtualenv (or venv) is perfectly powerful; if you need to isolate other parts of the system, such as memory, installed binaries and the such, your best bet is vagrant (combined with something like VirtualBox). And both of them work fine with a local IDE. Docker works best for deployment, when you want to set up a running process in a consistent manner, and its closed-box approach works really well in that scenario. But in development you generally want more flexibility.

catern 7y ago

Personally, I use Nix to create my Python virtual environments. That gets the isolation, replicability, and deployability that the author wants, without the heavyweight infrastructure of Docker.

ggm 7y ago
Got a show HN ?
- yorwba 7y ago
  Here's a post by tweag.io about using Nix to manage dependencies of Jupyter notebooks: https://www.tweag.io/posts/2019-02-28-jupyter-with.html https://www.tweag.io/posts/2019-02-28-jupyter-with.html
  - ggm 7y ago
    Awesome! Thanks!

ben509 7y ago

That looks pretty cool! My immediate use for it would be in replacing tox. It's a lot of downloading, but "docker pull python:3.7.3-alpine3.8" is far more reliable and simple than building from source. Especially for someone who just wants to test out a small contribution to a library.

se7entyse7en 7y ago
That's a cool use case that I didn't think about! I'm really glad that this project that I started only a couple of weeks ago is actually useful.
fock 7y ago
Seems like someone is indeed compiling: https://github.com/docker-library/python/blob/65b5603fdf177c6d90df6ea6f3bd6e23839c63a4/3.7/alpine3.8/Dockerfile https://github.com/docker-library/python/blob/65b5603fdf177c...

Townley 7y ago

I didn't follow how this relates to docker. Does running the pydocenv command modify dependencies stored in the Dockerfile (or a file that pydocenv references within a "pip install -r ___" command inside the Dockerfile)? It seems interesting, in that I respect any library (eg pyenv or pipenv) that tries to solve the issue with pip has compared to npm or cargo, where the installation of a library isn't coupled to the dependency tracking. I just can't see from the examples how this is preferable to changing a requirements.txt file and then rebuilding the docker image

se7entyse7en 7y ago
When I have some time I could try adding a `How does it work` section. Under the hood the creation of an environment corresponds to the creation of a docker container (that's how it relates to docker). The docker container is built starting from a the official python docker images as you can see here https://github.com/se7entyse7en/pydockenv/blob/master/pydockenv/commands/pydockenv.py#L32 https://github.com/se7entyse7en/pydockenv/blob/master/pydock.... The activation of the environment corresponds to the start of the container. Hence inside the container, you don't have (don't need) to have any virtual env as the isolation is already provided by the container itself. Every time you install a dependency, this is actually installed inside the container.
- Spivak 7y ago
  Once I get my environment set up just the way I want in the container how do give it to someone else or make another one? Seems wild to pass around the actual container rather than an image that spawns your environment.
  - se7entyse7en 7y ago
    To share your environment with someone else you can run `pydockenv save` (see here: https://github.com/se7entyse7en/pydockenv/blob/master/pydockenv/commands/pydockenv.py#L98 https://github.com/se7entyse7en/pydockenv/blob/master/pydock...). I'm sorry that any example is present in the README so far. Under the hood, a new docker image is saved from the container corresponding to the currently active environment. A `*.tar.gz` file is created with the image the can be now loaded using `pydockenv load`. Alternatively, you could push that image to Dockerhub for example and pull it from somewhere else. The problem is that currently there's no mechanism to make it recognize to `pydockenv`, but I'm also planning to add the possibility to create an environment from a user-defined image. Using `pydockenv save` will ensure that the environment you'll share will be exactly the same. An alternative that I'm also planning to implement is to use a `toml` file that will simply create a new environment from it. Obviously, in your running container, there could be something not captured in the original `toml` file if you, for example, installed something manually inside the container. But again, it's really in early stage, so all these feedbacks are really welcome!

pritambaral 7y ago

Personally, I use LXC (via LXD) and Emacs's TRAMP. Tramp is smart enough to run development tools (like flake8) on the remote, and I get a full OS to play with, with isolation.

rcarmo 7y ago

I’ve been using and maintaining a set of Python images targeted at doing builds and packaging for a while now: https://github.com/insightfulsystems/alpine-python https://github.com/insightfulsystems/alpine-python (There is also an Ubuntu variant, but it isn’t fully multiarch yet) This kind of thing is really useful when you have to build and deploy more modern/well-known versions of Python than what ships with your distribution, but I also use them as virtualenv replacements (although the workflow is different, since you have to derive a new image for each project)

rich-tea 7y ago

What I've wanted for a while now is to be able to run a single jupyter server but have many kernels running inside different containers. It's possible to have the kernels in different virtualenvs. I wonder if I could achieve that with this.