6 ms·

It does a bad job of dealing with versioning conflicts and multiple projects, so Python developers resort to hacks like virtual environments to get work done. C

by snazz 6y ago

It does a bad job of dealing with versioning conflicts and multiple projects, so Python developers resort to hacks like virtual environments to get work done. Compared to Cargo or even Go modules, it's not a great solution. It's also missing lots of features that are standard in other package managers.

vmchale 6y ago
> so Python developers resort to hacks like virtual environments to get work done It's really hard with many deps, it's why cabal (for instance) moved away from a global model.
deleted 6y ago
[deleted]
orf 6y ago
The need for a virtualenv has nothing to do with pip. Python only has “global” dependencies due to the way its import system works.
- jrochkind1 6y ago
  Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics). Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).
  - orf 6y ago
    so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.
    - jrochkind1 6y ago
      Sort of. The equivalent of "virtualenv" would be more "rvm gemsets", which is what people did before bundler. Bundler is doing something different. Bundler is not a "node_modules" style setup. It does not require dependencies to be in a local path (although they can be, the default is they live in a system-wide location, and this does not limit functionality). It also does not support more than one version of a dependency in the same execution environment (as node_modules does) -- that really would be impossible in ruby too. It's possible something about python's design would make the bundler approach impossible, I don't know. But it's not "dependencies are installed globally" alone, as that's true of ruby too. We would probably all benefit in understanding better how these things are handled in other environments. And I include myself here. I think ruby's bundler really set a new standard for best practices here, and many subsequent managers (like cargo) were heavily influenced by it, although many don't realize it. But meanwhile many don't even realize what they are missing or what's possible. Like the basic idea of having a specification of top-level dependencies (including allowable ranges) separate from a "lockfile" of exact versions in use of ALL dependencies... is just so hugely useful I never want to do without it, and I think is compatible with just about any architecture, and yet somehow JS is still only slowly catching on to it.
  - dragonwriter 6y ago
    > Ruby works the same way Kind of, if you ignore Rubygems, which is also part of stdlib at a lower level then bundler (and also, originally wasn't.) > but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. It can do that because rubygems manages multiple installed versions of packages and allows per-project ("per call to require", potentially, IIRC) specification of which one to pull from the globally-installed versions (this was originally done by monkey patching require when rubygems was an add-on.) This lets bundler easily live on top of it providing per-project dependencies somewhat more smoothly than Rubygems does without requiring anything like a venv. > Perhaps ruby was more "hackable" by bundler. Ruby is ludicrously hackable, yes. > (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked). Rubygems also wasn't part of stdlib originally, and started out relying on hacking around the way Kernel#require works.
    - jrochkind1 6y ago
      > It can do that because rubygems manages multiple installed versions of packages Oh wow, the default python dependency manager only lets you have one version of each package installed system-wide? Yeah, that is a limitation. As opposed to rubygems (the first dependency manager although as you say not originally built-in to ruby) which has system-wide install, but always let you have more than one version installed. Without fixing that one way or another, there's no sensible way, true. virtualenv is certainly one way to fix it. I wonder if there would have been a more rubygems way to fix it.
      - orf 6y ago
        Honestly I found the multiple versions approach more complex, confusing and more hacky. A virtualenv is just “node_modules” that also contains a Python executable. It’s a directory - you delete and create them at will, fast, and don’t worry or care about the system Python. Having some crazy setup that patches “require” to handle concurrently installed package versions seems insane, especially if you cannot actually use them concurrently in the same Ruby process. So, segmenting them by project (aka virtualenv) seems like the best solution.
- zo1 6y ago
  Not quite true. Python import mechanics are quite hackable and controllable programmatically and externally via controlling the relevant PATH env variables. It's just that no one bothers with it and instead seems to rely on the set of global folder-path lookup mechanics that are standard. But nothing stops you from masking global-libraries with local library versions (similar to node_modules). Why hasn't anyone done this you may ask? I don't know the answer to that.
  - orf 6y ago
    There are a few reasons you can’t have multiple versions of the same module at the same time. Consider a simple enum class defined in a package: two versions now have two different enum objects which may not compare equally. Maybe a function from module A would return Enum.A and pass it to module B, which would then compare A.Enum.A to B.Enum.A, which fails. Super confusing. So yes, Python’s import system is dynamic enough to do crazy things, but I don’t see how we can ever retrofit that into Python. Regarding dependencies installed into a project-local directory (node_modules): that’s a virtual environment. Just more flexible.
d0mine 6y ago
It may depend on you experience with each language. I have much more experience with Python than Go (almost none with Rust) and therefore I have much better time with python packaging tools (I don't remember a single issue, that I didn't find a satisfactory solution -- as much as possible in the packaging world with myriad conflicting use-cases) For example, my experience with `cargo` (that I mostly use to install command-line utilities such as rg, fd, dust): it is great when it works as written in the instructions but sometimes it doesn't (running `cargo` may involve a lot of compiling -- in contrast to `pip` which can use wheels transparently and avoid compiling even for modules with C extensions -- I guess there might be a way to do something similar with `cargo` though not by default).
ptx 6y ago
What's wrong with virtual environments? The required tools come bundled with Python nowadays and are super easy to use.
- jtdev 6y ago
  Agreed 100%,in fact of the languages I work with regularly (Python, Java, C#, JavaScript, Go) Python has the simplest dependency management solution via virtualenv + pypi + pip. Not sure why every Python thread turns into a conversation about the pain of Python dep management... seems overblown.
  - JAlexoid 6y ago
    Hm.... Let me see. The number of operations I have to perform in Maven vs Python for independent packages: Maven Scala project - create skeleton, add libraries to POM, write app, run app PIP Venv Python project - create venv, enable venv, create requirements file, write app, run pip to install dependencies(possibly install GCC and extra libraries), run app (Oh... and god forbid that you forget to deactivate venv) You're lying when you say that library management is easier in Python. It's just factually untrue.
    - Dylan16807 6y ago
      Most of the steps you're listing take only a few seconds. They're talking about the actual management, not how long it takes you to type . venv/bin/activate
      - JAlexoid 6y ago
        A few seconds here and a few seconds there - it's death by a thousand papercuts. There's no community consensus - that keeps Python from advancing to where it needs to be. I said it once and I'll say it again - Python lacks mature tooling.
        Dylan16807 6y ago

by snazz 6y ago

vmchale 6y ago

> so Python developers resort to hacks like virtual environments to get work done It's really hard with many deps, it's why cabal (for instance) moved away from a global model.

deleted 6y ago

[deleted]

orf 6y ago

The need for a virtualenv has nothing to do with pip. Python only has “global” dependencies due to the way its import system works.

jrochkind1 6y ago
Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics). Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).
- orf 6y ago
  so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.
  - jrochkind1 6y ago
    Sort of. The equivalent of "virtualenv" would be more "rvm gemsets", which is what people did before bundler. Bundler is doing something different. Bundler is not a "node_modules" style setup. It does not require dependencies to be in a local path (although they can be, the default is they live in a system-wide location, and this does not limit functionality). It also does not support more than one version of a dependency in the same execution environment (as node_modules does) -- that really would be impossible in ruby too. It's possible something about python's design would make the bundler approach impossible, I don't know. But it's not "dependencies are installed globally" alone, as that's true of ruby too. We would probably all benefit in understanding better how these things are handled in other environments. And I include myself here. I think ruby's bundler really set a new standard for best practices here, and many subsequent managers (like cargo) were heavily influenced by it, although many don't realize it. But meanwhile many don't even realize what they are missing or what's possible. Like the basic idea of having a specification of top-level dependencies (including allowable ranges) separate from a "lockfile" of exact versions in use of ALL dependencies... is just so hugely useful I never want to do without it, and I think is compatible with just about any architecture, and yet somehow JS is still only slowly catching on to it.
- dragonwriter 6y ago
  > Ruby works the same way Kind of, if you ignore Rubygems, which is also part of stdlib at a lower level then bundler (and also, originally wasn't.) > but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. It can do that because rubygems manages multiple installed versions of packages and allows per-project ("per call to require", potentially, IIRC) specification of which one to pull from the globally-installed versions (this was originally done by monkey patching require when rubygems was an add-on.) This lets bundler easily live on top of it providing per-project dependencies somewhat more smoothly than Rubygems does without requiring anything like a venv. > Perhaps ruby was more "hackable" by bundler. Ruby is ludicrously hackable, yes. > (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked). Rubygems also wasn't part of stdlib originally, and started out relying on hacking around the way Kernel#require works.
  - jrochkind1 6y ago
    > It can do that because rubygems manages multiple installed versions of packages Oh wow, the default python dependency manager only lets you have one version of each package installed system-wide? Yeah, that is a limitation. As opposed to rubygems (the first dependency manager although as you say not originally built-in to ruby) which has system-wide install, but always let you have more than one version installed. Without fixing that one way or another, there's no sensible way, true. virtualenv is certainly one way to fix it. I wonder if there would have been a more rubygems way to fix it.
    - orf 6y ago
      Honestly I found the multiple versions approach more complex, confusing and more hacky. A virtualenv is just “node_modules” that also contains a Python executable. It’s a directory - you delete and create them at will, fast, and don’t worry or care about the system Python. Having some crazy setup that patches “require” to handle concurrently installed package versions seems insane, especially if you cannot actually use them concurrently in the same Ruby process. So, segmenting them by project (aka virtualenv) seems like the best solution.
zo1 6y ago
Not quite true. Python import mechanics are quite hackable and controllable programmatically and externally via controlling the relevant PATH env variables. It's just that no one bothers with it and instead seems to rely on the set of global folder-path lookup mechanics that are standard. But nothing stops you from masking global-libraries with local library versions (similar to node_modules). Why hasn't anyone done this you may ask? I don't know the answer to that.
- orf 6y ago
  There are a few reasons you can’t have multiple versions of the same module at the same time. Consider a simple enum class defined in a package: two versions now have two different enum objects which may not compare equally. Maybe a function from module A would return Enum.A and pass it to module B, which would then compare A.Enum.A to B.Enum.A, which fails. Super confusing. So yes, Python’s import system is dynamic enough to do crazy things, but I don’t see how we can ever retrofit that into Python. Regarding dependencies installed into a project-local directory (node_modules): that’s a virtual environment. Just more flexible.

d0mine 6y ago

It may depend on you experience with each language. I have much more experience with Python than Go (almost none with Rust) and therefore I have much better time with python packaging tools (I don't remember a single issue, that I didn't find a satisfactory solution -- as much as possible in the packaging world with myriad conflicting use-cases) For example, my experience with `cargo` (that I mostly use to install command-line utilities such as rg, fd, dust): it is great when it works as written in the instructions but sometimes it doesn't (running `cargo` may involve a lot of compiling -- in contrast to `pip` which can use wheels transparently and avoid compiling even for modules with C extensions -- I guess there might be a way to do something similar with `cargo` though not by default).

ptx 6y ago

What's wrong with virtual environments? The required tools come bundled with Python nowadays and are super easy to use.

jtdev 6y ago
Agreed 100%,in fact of the languages I work with regularly (Python, Java, C#, JavaScript, Go) Python has the simplest dependency management solution via virtualenv + pypi + pip. Not sure why every Python thread turns into a conversation about the pain of Python dep management... seems overblown.
- JAlexoid 6y ago
  Hm.... Let me see. The number of operations I have to perform in Maven vs Python for independent packages: Maven Scala project - create skeleton, add libraries to POM, write app, run app PIP Venv Python project - create venv, enable venv, create requirements file, write app, run pip to install dependencies(possibly install GCC and extra libraries), run app (Oh... and god forbid that you forget to deactivate venv) You're lying when you say that library management is easier in Python. It's just factually untrue.
  - Dylan16807 6y ago
    Most of the steps you're listing take only a few seconds. They're talking about the actual management, not how long it takes you to type . venv/bin/activate
    - JAlexoid 6y ago
      A few seconds here and a few seconds there - it's death by a thousand papercuts. There's no community consensus - that keeps Python from advancing to where it needs to be. I said it once and I'll say it again - Python lacks mature tooling.
      - Dylan16807 6y ago