4 ms·

> While I get that you may have been frustrated with a Python setup It's not so much that I was frustrated by this, it's more the case that given the other eco

by skohan 7y ago

> While I get that you may have been frustrated with a Python setup It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling. > This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary I don't agree. I think npm does an excellent job of making js packages repeatable for example. > In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime. And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.

silvester23 7y ago
I haven't worked with npm in quite a while but as far as I remember, it does pretty much the same as pip with a virtual env and a requirements file -- it installs all dependencies into a directory for that specific project. The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
- skohan 7y ago
  The big difference I have seen is that with npm I have never run into issues because the code is being executed with the incorrect version of the interpreter, or something weird like python being installed at an unexpected path. Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task. In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies. Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
  - wrmsr 7y ago
    As below node has the luxuries of both having been able to learn lessons from python and ruby ~and~ not being a victim of its own success of being already integrated with and a dependency of most major operating systems. (That said javascript the language definitely ~is~ a victim of its own success in browsers - there's probably some metaphorical parallel here..) But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing. If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
- mumblemumble 7y ago
  I'm guessing npm also has a nice luxury that nobody writes core parts of the operating system in JavaScript. Versus, one of Python's bigger challenges with environment management is that you need to come up with an environment isolation mechanism that won't also interfere with the operation of the OS itself, or cut you off from being able to use userland tools that rely on it. Which. . . Python being an essential part of virtually every production system out there being one of the root causes behind a situation that makes people feel skittish about its appropriateness for use in production is a fun thought. It's a particularly delightfully Unixy brand of ouroboros.
  - skohan 7y ago
    Yes this seems like a maassive problem with a band-aid solution. Like the fact that I might try to change my system Python version to get my Python project working, and that could cause side-effects in completely un-related areas - like maybe my C++ build system is now broken - just seems like an absolutely absurd state of affairs. It seems crazy to me that the precise Python version isn't defined in the Python file itself, and the interpreter is responsible for sorting out compatibility issues.
danpalmer 7y ago
With the pip-tools package and pip-compile, having a stable requirements system is pretty straightforward. We have no problems with pyenv and pip locally, and dh-virtualenv in production. We're considering moving to poetry as it brings a load of nice features and more automation, but really it's not been a problem for us. Node I'd say is about as complicated. You still need to care about the Node version, about as much as the Python version. a node_modules directory is pretty similar to a virtualenv, it's not portable, maybe a little more reproducible by default, but no better than a pip-tools managed requirements file in my experience. Package upgrades are a little trickier than with pip-tools. You still need to care about things like what version of OpenSSL is on your machine, or do you have libpq installed, etc. One problem with JS is whether to compile and if so, how, what do you run, etc. You probably don't want to ship a node_modules to production with all the random package build scripts that it includes, so there's some additional complexity there introducing a build process.

by skohan 7y ago

silvester23 7y ago

I haven't worked with npm in quite a while but as far as I remember, it does pretty much the same as pip with a virtual env and a requirements file -- it installs all dependencies into a directory for that specific project. The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.

skohan 7y ago
The big difference I have seen is that with npm I have never run into issues because the code is being executed with the incorrect version of the interpreter, or something weird like python being installed at an unexpected path. Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task. In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies. Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
- wrmsr 7y ago
  As below node has the luxuries of both having been able to learn lessons from python and ruby ~and~ not being a victim of its own success of being already integrated with and a dependency of most major operating systems. (That said javascript the language definitely ~is~ a victim of its own success in browsers - there's probably some metaphorical parallel here..) But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing. If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
mumblemumble 7y ago
I'm guessing npm also has a nice luxury that nobody writes core parts of the operating system in JavaScript. Versus, one of Python's bigger challenges with environment management is that you need to come up with an environment isolation mechanism that won't also interfere with the operation of the OS itself, or cut you off from being able to use userland tools that rely on it. Which. . . Python being an essential part of virtually every production system out there being one of the root causes behind a situation that makes people feel skittish about its appropriateness for use in production is a fun thought. It's a particularly delightfully Unixy brand of ouroboros.
- skohan 7y ago
  Yes this seems like a maassive problem with a band-aid solution. Like the fact that I might try to change my system Python version to get my Python project working, and that could cause side-effects in completely un-related areas - like maybe my C++ build system is now broken - just seems like an absolutely absurd state of affairs. It seems crazy to me that the precise Python version isn't defined in the Python file itself, and the interpreter is responsible for sorting out compatibility issues.

danpalmer 7y ago

With the pip-tools package and pip-compile, having a stable requirements system is pretty straightforward. We have no problems with pyenv and pip locally, and dh-virtualenv in production. We're considering moving to poetry as it brings a load of nice features and more automation, but really it's not been a problem for us. Node I'd say is about as complicated. You still need to care about the Node version, about as much as the Python version. a node_modules directory is pretty similar to a virtualenv, it's not portable, maybe a little more reproducible by default, but no better than a pip-tools managed requirements file in my experience. Package upgrades are a little trickier than with pip-tools. You still need to care about things like what version of OpenSSL is on your machine, or do you have libpq installed, etc. One problem with JS is whether to compile and if so, how, what do you run, etc. You probably don't want to ship a node_modules to production with all the random package build scripts that it includes, so there's some additional complexity there introducing a build process.