5 ms·

I prefer containers over pyenv and poetry. This way not only python version and dependencies are "in one place" but also all other stuff that comes along with a

I prefer containers over pyenv and poetry. This way not only python version and dependencies are "in one place" but also all other stuff that comes along with a new project. The OS, the database etc. The one thing I dislike about Python projects is that Python plasters the compile cache files all over the place. Is there a reason to change that? Currently I use the -B flag for all my scripts. But that makes it slow. I wish Python would have an option to perform like PHP and keep cached compilations in memory instead on disk. Or at least somewhere in /tmp/.

willis936 5y ago
If you need a specific OS then isn't a VM the solution rather than a container? Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language?
- skohan 5y ago
  Yeah to be honest if you need containers to make a project reproducible this is just a sign of failure. You're basically saying you need to encapsulate the entire system for your code to run correctly.
  - cpfohl 5y ago
    Loads of tools have external dependencies that are hard dependencies. And library search paths is my machine vary from those on prod... I could go on, but I'm not sure I understand why using a container to manage all that is a failure?
    - skohan 5y ago
      There are plenty of reasons why you might want to containerize a project. If you have a lot of system dependencies, for example, you might want to consider including a Dockerfile in your project to make it portable. However what makes Python a failure is that people feel they need this to dependably run a python program which only has pure-python dependencies. Compare this to a language like Rust, or the NPM ecosystem. In those cases, the tools have managed to dependably encapsulate projects such that you only need the package manager to make a project fully repeatable. With either of those ecosystems, there's basically one system dependency, and you can find any repository online and dependably do `git clone ...` then `cargo build` etc. to make it work. With Python, you effectively have to reproduce the original developer's system, and that is a failure.
      - Spivak 5y ago
        Huh? Either something is really weird about your env or we have different ideas about what counts as a pure Python package. Because if you don’t rely on Python packages with extensions that farm out to external libs it’s as easy as git clone, pyenv virtualenv, pip install -r, and python -m build.
        skohan 5y ago
        The think that makes this worse than other ecosystems is: 1. virtualenv shouldn't be necessary. This is more or less the same concept as containerization. This is only needed because python has a fractured ecosystem, and setting up your environment for one project can break another. 2. you also have to know which environment encapsulation and package management solution the library author is using - this is not standardized
        zo1 5y ago
        1. Virtualenv is essentially the same as node_modules, yet everyone rants and raves and loves that. And the kind of breakage you're talking about is astronomically rare in my experience. 2. No you don't - what makes you say that?
        skohan 5y ago
        virtualenv is so much less user friendly than npm. Like why do I have to run a `source` command to make virtualenv work? I don't use either often, but I can remember how to use npm if I haven't used it in like 6 months, but I have to look up the right commands virtualenv if I haven't used it for like 2 weeks.
        Daishiman 5y ago
        If you want to run multiple versions of node you're going to need to use some sort of version manager too.
- Tipewryter 5y ago
  A Docker container starts in two seconds or so. And gives me everything I need. So no need to dabble with a VM. There is not much overhead in running a project in a container. The project has a setup file that turns a fresh Debian 10 into whatever environment it needs. And thats it. Run that setup script in your Dockerfile to create a container and you are all set. Want to run the project in a VM or on bare metal? Just install Debian 10, run the setup script and you all set.
- evgen 5y ago
  > Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language? Probably some time shortly after your developer time costs less than your cloud compute time. Until you hit that point (if ever) there are few options as cost-effective as Python.
- Spivak 5y ago
  In any language ever if you use non-vendored shared libs you will hit this problem. Certainly not specific to Python, in fact the reason package managers on *nix are necessary (and not just a nice to have) is because of this.
Tenoke 5y ago
Yeah, I don't think I've created a single venv in the last 2 years. I don't need them for basic stuff (e.g. a quick script) and for anything more I'd rather have a container so I can deal with all dependencies in one place and use it elsewhere quicker if I need to.
bpicolo 5y ago
Poetry is a great way to manage library dependencies in containerized apps
- Tipewryter 5y ago
  Why would I need poetry? Doesn't "pip3 install -r requirements.txt" do everything I need?
  - deleted 5y ago
    [deleted]
  - bpicolo 5y ago
    Pip is fine, it depends on your goals. I've found requirements.txt less enjoyable to maintain for several reasons – you need to separate dev, test dependencies on your own time, there's no notion of a lockfile for transitive dependencies (`pip freeze` notably doesn't separate actual dependencies from transitive dependencies). pip is also darn slow at installing dependencies once you hit a certain scale, and poetry outperforms it pretty substantially. Poetry does I expect a package manager to do, and does it well, especially when working with a team of developers on an application versus individually. There's not a compelling reason for me to use pip directly as a less functional alternative.
    - Tipewryter 5y ago
      Additional requirement files for dev and test don't seem like a burden to me. Can you describe an issue that you had by not locking transitive dependencies?
      - bpicolo 5y ago
        Bit rot, "it works on my machine"-style issues, cache misses on dependency installation (which can really bloat deploy times in deploy pipelines by busting Docker caches across machines, too). Can be a security issue if a vulnerable library version is pushed and one installs it as a consequence of having non-locked dependencies, especially in python where package install scripts have a lot of power. Lock files help solve for these. You can build software without solving them, but it makes my life easier.
ed_balls 5y ago
How do you make sure your dependencies are not tampered with? https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...
- Tipewryter 5y ago
  That is a very broad question. Can you mention a specific attack vector? Then I might be able to explain how I do or do not avoid it.
  - ed_balls 5y ago
    The link describes the attack vector. pipenv locks the dependencies using hash. if you company has my-company-py-lib then pip could install public library that pretends to be internal.
0x008 5y ago
yes, you can set PYTHONDONTWRITEBYTECODE=1 in your environment but it is equivalent to -B, I think.

willis936 5y ago

If you need a specific OS then isn't a VM the solution rather than a container? Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language?

skohan 5y ago
Yeah to be honest if you need containers to make a project reproducible this is just a sign of failure. You're basically saying you need to encapsulate the entire system for your code to run correctly.
- cpfohl 5y ago
  Loads of tools have external dependencies that are hard dependencies. And library search paths is my machine vary from those on prod... I could go on, but I'm not sure I understand why using a container to manage all that is a failure?
  - skohan 5y ago
    There are plenty of reasons why you might want to containerize a project. If you have a lot of system dependencies, for example, you might want to consider including a Dockerfile in your project to make it portable. However what makes Python a failure is that people feel they need this to dependably run a python program which only has pure-python dependencies. Compare this to a language like Rust, or the NPM ecosystem. In those cases, the tools have managed to dependably encapsulate projects such that you only need the package manager to make a project fully repeatable. With either of those ecosystems, there's basically one system dependency, and you can find any repository online and dependably do `git clone ...` then `cargo build` etc. to make it work. With Python, you effectively have to reproduce the original developer's system, and that is a failure.
    - Spivak 5y ago
      Huh? Either something is really weird about your env or we have different ideas about what counts as a pure Python package. Because if you don’t rely on Python packages with extensions that farm out to external libs it’s as easy as git clone, pyenv virtualenv, pip install -r, and python -m build.
      - skohan 5y ago
        The think that makes this worse than other ecosystems is: 1. virtualenv shouldn't be necessary. This is more or less the same concept as containerization. This is only needed because python has a fractured ecosystem, and setting up your environment for one project can break another. 2. you also have to know which environment encapsulation and package management solution the library author is using - this is not standardized
        zo1 5y ago
        1. Virtualenv is essentially the same as node_modules, yet everyone rants and raves and loves that. And the kind of breakage you're talking about is astronomically rare in my experience. 2. No you don't - what makes you say that?
        skohan 5y ago
        virtualenv is so much less user friendly than npm. Like why do I have to run a `source` command to make virtualenv work? I don't use either often, but I can remember how to use npm if I haven't used it in like 6 months, but I have to look up the right commands virtualenv if I haven't used it for like 2 weeks.
        Daishiman 5y ago
        If you want to run multiple versions of node you're going to need to use some sort of version manager too.
Tipewryter 5y ago
A Docker container starts in two seconds or so. And gives me everything I need. So no need to dabble with a VM. There is not much overhead in running a project in a container. The project has a setup file that turns a fresh Debian 10 into whatever environment it needs. And thats it. Run that setup script in your Dockerfile to create a container and you are all set. Want to run the project in a VM or on bare metal? Just install Debian 10, run the setup script and you all set.
evgen 5y ago
> Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language? Probably some time shortly after your developer time costs less than your cloud compute time. Until you hit that point (if ever) there are few options as cost-effective as Python.
Spivak 5y ago
In any language ever if you use non-vendored shared libs you will hit this problem. Certainly not specific to Python, in fact the reason package managers on *nix are necessary (and not just a nice to have) is because of this.

Tenoke 5y ago

Yeah, I don't think I've created a single venv in the last 2 years. I don't need them for basic stuff (e.g. a quick script) and for anything more I'd rather have a container so I can deal with all dependencies in one place and use it elsewhere quicker if I need to.

bpicolo 5y ago

Poetry is a great way to manage library dependencies in containerized apps

Tipewryter 5y ago
Why would I need poetry? Doesn't "pip3 install -r requirements.txt" do everything I need?
- deleted 5y ago
  [deleted]
- bpicolo 5y ago
  Pip is fine, it depends on your goals. I've found requirements.txt less enjoyable to maintain for several reasons – you need to separate dev, test dependencies on your own time, there's no notion of a lockfile for transitive dependencies (`pip freeze` notably doesn't separate actual dependencies from transitive dependencies). pip is also darn slow at installing dependencies once you hit a certain scale, and poetry outperforms it pretty substantially. Poetry does I expect a package manager to do, and does it well, especially when working with a team of developers on an application versus individually. There's not a compelling reason for me to use pip directly as a less functional alternative.
  - Tipewryter 5y ago
    Additional requirement files for dev and test don't seem like a burden to me. Can you describe an issue that you had by not locking transitive dependencies?
    - bpicolo 5y ago
      Bit rot, "it works on my machine"-style issues, cache misses on dependency installation (which can really bloat deploy times in deploy pipelines by busting Docker caches across machines, too). Can be a security issue if a vulnerable library version is pushed and one installs it as a consequence of having non-locked dependencies, especially in python where package install scripts have a lot of power. Lock files help solve for these. You can build software without solving them, but it makes my life easier.

ed_balls 5y ago

How do you make sure your dependencies are not tampered with? https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...

Tipewryter 5y ago
That is a very broad question. Can you mention a specific attack vector? Then I might be able to explain how I do or do not avoid it.
- ed_balls 5y ago
  The link describes the attack vector. pipenv locks the dependencies using hash. if you company has my-company-py-lib then pip could install public library that pretends to be internal.

0x008 5y ago

yes, you can set PYTHONDONTWRITEBYTECODE=1 in your environment but it is equivalent to -B, I think.