6 ms·

This, me too. I never understood the need for virtualenv and similar. Do people really encounter trouble with conflicting packages that often? I try to write s

by amenod 9y ago

This, me too. I never understood the need for virtualenv and similar. Do people really encounter trouble with conflicting packages that often? I try to write scripts so they run on different versions of python anyway, unless there is a very specific reason why that is not possible; and even then you can run python versions in parallel on a Debian/Ubuntu box, with different pip installs for each of them. As for production, I usually ship things in a Docker container anyway, so there is no chance of mismatched libraries. I guess I just never saw a problem that virtualenv solves.

stctgion 9y ago
I experience version conflicts all the time. I work on multiple projects and virtual envs are life savers.
eesmith 9y ago
The linked-to essay describes several reasons why the author uses virtualenv. Which of them don't you understand? My package supports several optional back-ends, selectable at run-time, and it runs under Python 2.7 and 3.6+. I use tox to manage different virtualenvs for the combination of {backend X but not Y or Z, Python 2.7}, {backend X but not Y or Z, Python 3.6}, etc. for Y, and Z, as well as {backend X and Y and Z} for the two versions. Oh, and I support several releases of each of the X, Y, and Z. That's a lot of virtualenvs. Usually I only develop on the X+Y+Z version because full test suite across all the combinations takes about 15 minutes. I also do coverage testing, and the Python coverage tool isn't hard to use under this setup to combine tox results across multiple virturalenvs. I don't know if Docker is better. I've been using this setup for some years.
landr0id 9y ago
I use virtualenv so I can just easily `pip freeze > requirements.txt` for individual projects.
- acdha 9y ago
  pipenv is basically the next level of this process: it records every install and locks only the things you installed rather than their dependencies which often change over time.
  - falcolas 9y ago
    I'm a bit confused by this statement, do you mean: I install foo 1.0, which depends on frob >= 1.0 (which happens to be frob 1.1 when I installed it). Foo 1.1 comes out, as does frob 1.2 and 1.3. If I reinstall from the pipfile, do I get foo 1.0 with frob 1.3? I ask because that sounds like a bug waiting to happen. IMO, frozen requirements should remain frozen.
    - acdha 9y ago
      Here's what happens: You type `pipenv install foo`. It will create the virtualenv if necessary and adds `foo = " * "` to the packages section of the Pipfile. The Pipfile.lock file will add a section for the package you installed _and_ all of its dependencies, including the hashes of the downloaded packages. That avoids accidental breakage if the package you depend on doesn't set their versions correctly but it also means that your main Pipfile documents the things which you intentionally installed so a year from now you're not wondering why all of your servers have frob 1.2 installed, which only works on Python 2.7, even though nothing you're using now depends on it. As a concrete example, here's what `pipenv install requests` looks like in a clean project: Pipfile: [[source]] url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi" [packages] requests = " * " [dev-packages] (had I used `--python $(which python2.7)` it'd have recorded that as well) Pipfile.lock: { "_meta": { "hash": { "sha256": "a0e63f8a0d1e3df046dc19b3ffbaaedfa151afc12af5a5b960ae7393952f8679" }, "host-environment-markers": { "implementation_name": "cpython", "implementation_version": "3.6.4", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "17.4.0", "platform_system": "Darwin", "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64", "python_full_version": "3.6.4", "python_version": "3.6", "sys_platform": "darwin" }, "pipfile-spec": 6, "requires": {}, "sources": [ { "name": "pypi", "url": "https://pypi.python.org/simple", "verify_ssl": true } ] }, "default": { "certifi": { "hashes": [ "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296", "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d" ], "version": "==2018.1.18" }, "chardet": { "hashes": [ "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691", "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae" ], "version": "==3.0.4" }, "idna": { "hashes": [ "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4", "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f" ], "version": "==2.6" }, "requests": { "hashes": [ "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b", "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e" ], "version": "==2.18.4" }, "urllib3": { "hashes": [ "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b", "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f" ], "version": "==1.22" } }, "develop": {} } (EDITED: the HN Markdown parser appears to be a simple regex match and breaks formatting with a * and uses only the ASCII definition of whitespace so I couldn't use a zero-width space. The real output doesn't have spaces around the asterisks).
randlet 9y ago
> Do people really encounter trouble with conflicting packages that often? Yes it's quite common. For example I work on multiple projects with different versions of Django being used in each one.
krautsourced 9y ago
A rather simple practical example I'm working on right now: moving a Django app from 1.9 to 2.0. There are a few code breaking changes in 2.0. All I need to do is create a new virtualenv with Django 2.0 and I can just switch between both versions simply by changing the environment. I may even need to change some further packages that do not work with 2.0 yet and swap them for something else. I can do that in the new env. Then, when all is working, I export a requirements.txt from my virtual env and can quickly set up the very same virtual env on my production machine.
reachtarunhere 9y ago
Yes, it is very common. I work a lot with ML code released as part of research papers. Everyone uses a different version of TF/PyTorch or whatever framework they are using. I can't imagine working without something like virtualenv (though I use pipenv)
odonnellryan 9y ago
Do you just use one environment for every project? That gets messy. What if you want to setup a new machine? Or a new employee's machine? Etc.. It isn't only about conflicting packages.
- amenod 9y ago
  Yes - but then again, I don't have that many projects and I tend to use the same libraries across them anyway. Installing on another machine is just a matter of pip install -r requirements.txt, if it's for a dev. other wise it's just a docker run .... I do know that most Python devs use virtualenv, I just never understood what all the fuss is about. Of course I don't mind if the others in team use it, I just never saw the appeal so I'm curious what other peoples' reasons are. Maybe I don't see the need because I came to Python quite late, a little before Docker came around, and we were early adopters for Docker. So we solved testing and production with Docker (and thus never needed to support multiple Python versions in parallel). It's really one of those small mysteries I don't understand. :)
  - odonnellryan 9y ago
    Yeah I understand what you mean. I have around ~12 projects, for different clients, that are just recent enough that I might need to access the environments. If that was one environment (forget some of them being stuck on 2.x) the number of damn dependencies would be monstrous! When I need to bring a partner onto the project, I can't give them a requirements file that's 5x what it should be! > I just never understood what all the fuss is about. It's to keep environments clean + easy to maintain. Virtual is probably a bad word for it. It isn't like Docker or a VM. I don't think too many Python projects that are not libraries support many versions, that isn't the point of virtualenv or conda anyway.
falcolas 9y ago
I've found that the library dependency issues appear in direct proportion to the number 3rd party libraries you use. If you're mostly developing against the standard library with a handful other stable libraries, it's not an issue. If you're developing with a requirements.txt (or pipfile) with 10 or more entries, you'll start running into conflicts on a regular basis. It can also occur as projects age, especially with 3rd party libraries who don't provide API backwards compatibility (which I fully acknowledge is a PITA to develop for, more effort than is frequently justifiable). Both of these are why I prefer to use the standard library when possible. It's going to remain API stable for a very long time, and the occasional 2-3 lines of boilerplate to do HTTPS requests (and other similar convenience functions) is a cost I'm willing to bear for that stability.
- speedplane 9y ago
  I think pretty much everyone would "prefer to use the standard library where possible". But try building a modern web app with that. No one uses libraries because they want to, they use b/c they need to.
  - falcolas 9y ago
    > No one uses libraries because they want to, they use b/c they need to. I'll have to respectfully disagree with this one. Everyone pulls in requests the moment they have to make any kind of http request for the sole reason that it's more ergonomic, not because it's "needed". And requests brings in 4 of its own dependencies. Right there you've created a prime chance for everything to go sideways (and I've watched it happen explicitly with the requests library as it bolts on more and more ergonomic features). For what it's worth, the last web app I built had a DB library from the OS vendor, flask, and gunicorn. All of which, since they were quite stable, never introduced library conflicts.
wirrbel 9y ago
Docker containers and virtual envs are in some way both capsules that can serve a similar purpose. virtual envs are in general more lightweight, but limited to the Python world, whereas Docker containers are heavier, but enable you to also include other dependencies in that capsule. I sometimes feel that Docker containers are a bit misused if they are just used as environment capsules. They provide more isolation (process space, etc.). But of course, that would be still fine. The software I write mostly does not require databases and other supporting services to run on my laptop. So virtualenvs are all I need (most of the time). There is a docker tax that I don't want to pay, unless I heavily benefit from it. I guess, if I could deploy docker images, the return of invest would be much better and I might use it more.

by amenod 9y ago

stctgion 9y ago

I experience version conflicts all the time. I work on multiple projects and virtual envs are life savers.

eesmith 9y ago

The linked-to essay describes several reasons why the author uses virtualenv. Which of them don't you understand? My package supports several optional back-ends, selectable at run-time, and it runs under Python 2.7 and 3.6+. I use tox to manage different virtualenvs for the combination of {backend X but not Y or Z, Python 2.7}, {backend X but not Y or Z, Python 3.6}, etc. for Y, and Z, as well as {backend X and Y and Z} for the two versions. Oh, and I support several releases of each of the X, Y, and Z. That's a lot of virtualenvs. Usually I only develop on the X+Y+Z version because full test suite across all the combinations takes about 15 minutes. I also do coverage testing, and the Python coverage tool isn't hard to use under this setup to combine tox results across multiple virturalenvs. I don't know if Docker is better. I've been using this setup for some years.

landr0id 9y ago

I use virtualenv so I can just easily `pip freeze > requirements.txt` for individual projects.

acdha 9y ago
pipenv is basically the next level of this process: it records every install and locks only the things you installed rather than their dependencies which often change over time.
- falcolas 9y ago
  I'm a bit confused by this statement, do you mean: I install foo 1.0, which depends on frob >= 1.0 (which happens to be frob 1.1 when I installed it). Foo 1.1 comes out, as does frob 1.2 and 1.3. If I reinstall from the pipfile, do I get foo 1.0 with frob 1.3? I ask because that sounds like a bug waiting to happen. IMO, frozen requirements should remain frozen.
  - acdha 9y ago
    Here's what happens: You type `pipenv install foo`. It will create the virtualenv if necessary and adds `foo = " * "` to the packages section of the Pipfile. The Pipfile.lock file will add a section for the package you installed _and_ all of its dependencies, including the hashes of the downloaded packages. That avoids accidental breakage if the package you depend on doesn't set their versions correctly but it also means that your main Pipfile documents the things which you intentionally installed so a year from now you're not wondering why all of your servers have frob 1.2 installed, which only works on Python 2.7, even though nothing you're using now depends on it. As a concrete example, here's what `pipenv install requests` looks like in a clean project: Pipfile: [[source]] url = "https://pypi.python.org/simple" verify_ssl = true name = "pypi" [packages] requests = " * " [dev-packages] (had I used `--python $(which python2.7)` it'd have recorded that as well) Pipfile.lock: { "_meta": { "hash": { "sha256": "a0e63f8a0d1e3df046dc19b3ffbaaedfa151afc12af5a5b960ae7393952f8679" }, "host-environment-markers": { "implementation_name": "cpython", "implementation_version": "3.6.4", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "17.4.0", "platform_system": "Darwin", "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64", "python_full_version": "3.6.4", "python_version": "3.6", "sys_platform": "darwin" }, "pipfile-spec": 6, "requires": {}, "sources": [ { "name": "pypi", "url": "https://pypi.python.org/simple", "verify_ssl": true } ] }, "default": { "certifi": { "hashes": [ "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296", "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d" ], "version": "==2018.1.18" }, "chardet": { "hashes": [ "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691", "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae" ], "version": "==3.0.4" }, "idna": { "hashes": [ "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4", "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f" ], "version": "==2.6" }, "requests": { "hashes": [ "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b", "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e" ], "version": "==2.18.4" }, "urllib3": { "hashes": [ "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b", "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f" ], "version": "==1.22" } }, "develop": {} } (EDITED: the HN Markdown parser appears to be a simple regex match and breaks formatting with a * and uses only the ASCII definition of whitespace so I couldn't use a zero-width space. The real output doesn't have spaces around the asterisks).

randlet 9y ago

> Do people really encounter trouble with conflicting packages that often? Yes it's quite common. For example I work on multiple projects with different versions of Django being used in each one.

krautsourced 9y ago

A rather simple practical example I'm working on right now: moving a Django app from 1.9 to 2.0. There are a few code breaking changes in 2.0. All I need to do is create a new virtualenv with Django 2.0 and I can just switch between both versions simply by changing the environment. I may even need to change some further packages that do not work with 2.0 yet and swap them for something else. I can do that in the new env. Then, when all is working, I export a requirements.txt from my virtual env and can quickly set up the very same virtual env on my production machine.

reachtarunhere 9y ago

Yes, it is very common. I work a lot with ML code released as part of research papers. Everyone uses a different version of TF/PyTorch or whatever framework they are using. I can't imagine working without something like virtualenv (though I use pipenv)

odonnellryan 9y ago

Do you just use one environment for every project? That gets messy. What if you want to setup a new machine? Or a new employee's machine? Etc.. It isn't only about conflicting packages.

amenod 9y ago
Yes - but then again, I don't have that many projects and I tend to use the same libraries across them anyway. Installing on another machine is just a matter of pip install -r requirements.txt, if it's for a dev. other wise it's just a docker run .... I do know that most Python devs use virtualenv, I just never understood what all the fuss is about. Of course I don't mind if the others in team use it, I just never saw the appeal so I'm curious what other peoples' reasons are. Maybe I don't see the need because I came to Python quite late, a little before Docker came around, and we were early adopters for Docker. So we solved testing and production with Docker (and thus never needed to support multiple Python versions in parallel). It's really one of those small mysteries I don't understand. :)
- odonnellryan 9y ago
  Yeah I understand what you mean. I have around ~12 projects, for different clients, that are just recent enough that I might need to access the environments. If that was one environment (forget some of them being stuck on 2.x) the number of damn dependencies would be monstrous! When I need to bring a partner onto the project, I can't give them a requirements file that's 5x what it should be! > I just never understood what all the fuss is about. It's to keep environments clean + easy to maintain. Virtual is probably a bad word for it. It isn't like Docker or a VM. I don't think too many Python projects that are not libraries support many versions, that isn't the point of virtualenv or conda anyway.

falcolas 9y ago

I've found that the library dependency issues appear in direct proportion to the number 3rd party libraries you use. If you're mostly developing against the standard library with a handful other stable libraries, it's not an issue. If you're developing with a requirements.txt (or pipfile) with 10 or more entries, you'll start running into conflicts on a regular basis. It can also occur as projects age, especially with 3rd party libraries who don't provide API backwards compatibility (which I fully acknowledge is a PITA to develop for, more effort than is frequently justifiable). Both of these are why I prefer to use the standard library when possible. It's going to remain API stable for a very long time, and the occasional 2-3 lines of boilerplate to do HTTPS requests (and other similar convenience functions) is a cost I'm willing to bear for that stability.

speedplane 9y ago
I think pretty much everyone would "prefer to use the standard library where possible". But try building a modern web app with that. No one uses libraries because they want to, they use b/c they need to.
- falcolas 9y ago
  > No one uses libraries because they want to, they use b/c they need to. I'll have to respectfully disagree with this one. Everyone pulls in requests the moment they have to make any kind of http request for the sole reason that it's more ergonomic, not because it's "needed". And requests brings in 4 of its own dependencies. Right there you've created a prime chance for everything to go sideways (and I've watched it happen explicitly with the requests library as it bolts on more and more ergonomic features). For what it's worth, the last web app I built had a DB library from the OS vendor, flask, and gunicorn. All of which, since they were quite stable, never introduced library conflicts.

wirrbel 9y ago

Docker containers and virtual envs are in some way both capsules that can serve a similar purpose. virtual envs are in general more lightweight, but limited to the Python world, whereas Docker containers are heavier, but enable you to also include other dependencies in that capsule. I sometimes feel that Docker containers are a bit misused if they are just used as environment capsules. They provide more isolation (process space, etc.). But of course, that would be still fine. The software I write mostly does not require databases and other supporting services to run on my laptop. So virtualenvs are all I need (most of the time). There is a docker tax that I don't want to pay, unless I heavily benefit from it. I guess, if I could deploy docker images, the return of invest would be much better and I might use it more.