6 ms·

The python environment story has led me to the conclusion that it's not really suited for serious production software. I was recently onboarded into a project

by skohan 7y ago

The python environment story has led me to the conclusion that it's not really suited for serious production software. I was recently onboarded into a project which is using Python in a server-less setting, and when my colleague walked me through the concept of setting up a virtual environment for each one of my python projects I could not really believe it. It seems like you practically need to containerize a python code-base to make it repeatable.

danpalmer 7y ago
> led me to the conclusion that it's not really suited for serious production software. While I get that you may have been frustrated with a Python setup, concluding that it's not suitable for "serious" production software isn't a proportionate response, and the evidence of many companies using it for "serious" production systems suggests that it is in fact fine. Examples include Instagram and Dropbox. > It seems like you practically need to containerize a python code-base to make it repeatable. This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary. Arguably even for anything that doesn't produce completely static binaries. Yes Python has a learning curve for production use, all languages and ecosystems do. In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. Python is not out of the ordinary, it's not the worst. It's not the best either, it's fairly reasonable for its age, complexity, and typical uses.
- skohan 7y ago
  > While I get that you may have been frustrated with a Python setup It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling. > This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary I don't agree. I think npm does an excellent job of making js packages repeatable for example. > In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime. And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.
  - silvester23 7y ago
    I haven't worked with npm in quite a while but as far as I remember, it does pretty much the same as pip with a virtual env and a requirements file -- it installs all dependencies into a directory for that specific project. The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
    - skohan 7y ago
      The big difference I have seen is that with npm I have never run into issues because the code is being executed with the incorrect version of the interpreter, or something weird like python being installed at an unexpected path. Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task. In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies. Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
      - wrmsr 7y ago
        As below node has the luxuries of both having been able to learn lessons from python and ruby ~and~ not being a victim of its own success of being already integrated with and a dependency of most major operating systems. (That said javascript the language definitely ~is~ a victim of its own success in browsers - there's probably some metaphorical parallel here..) But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing. If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.
wwright 7y ago
Have you worked with C++ much? It has the same problems to a much, much worse extent.
- skohan 7y ago
  I probably would not choose C++ either unless I really needed its USP's like extreme memory/performance optimization or required compatibility with exotic hardware. But at least with C++ you can compile a binary which won't behave differently on a different system most of the time. But there are a lot of other languages out there besides C++ and Python.
  - fnord123 7y ago
    > But at least with C++ you can compile a binary which won't behave differently on a different system most of the time. Not really, no. Only if the compiler, compiler version, compilation flags, kernel version, platform, platform version, cpu version, directory structure, shared object versions, locale, environment (LD_PRELOAD, LD_LIBRARY_PATH, TZ), config in pwd, .local, and /etc or maybe $XDG_*, "~/Library/Application Support/" are all the same.
    - skohan 7y ago
      Fair but I think my other points still stand
    - KMag 7y ago
      The GP specifically was talking about the binary behaving differently, so the compiler version and compilation flags would be irrelevant. The kernel, system libraries, environment variables, etc. would all still be relevant, though. I think they're referring to unpredictable behavior under future versions of language VMs and garbage collectors vs. statically compiled binaries with either no GC or a statically linked GC library. Most of the time, all of that machinery going on behind the scenes is very convenient, but coding strictly to POSIX and the C/C++ standard libraries / STL and keeping your compilation artifacts gives you a lot more stability than coding to some version of the Java standard library (or other language VM) and keeping around your JVM installer with hopes that the installer will agree to install and run on future systems. I remember being bitten by testing our product against the earliest JRE we supported and also the latest JRE sun had released at the time. I was aware that reading environment variables was deprecated in the earliest JRE I tested with, and I knew it had been un-deprecated in the latest JRE I tested, and the methods I was using were present in all of the JRE versions in between. What I had missed was that there was at least one JRE version in the middle where the deprecated environment variable reading methods actually threw runtime errors. I've also run into issues in the past with the version of MSVC I was required to use having an installer that depended upon some ancient version of the CLR which would no longer install on my machine. There were some workarounds, but I needed to talk to several of our helpdesk people before I found someone who knew the required magic incantations. Had the MSVC installer been coded strictly to C++ stdlib, STL, Win32 and MFC, it would have run just fine on my modern desktop.
mumblemumble 7y ago
I used to think that, end then I just realized that this is really just Python showing its roots as a relatively old programming language with Unix roots. C and C++ have a a fairly similar problem with dependency hell due to shared libraries being installed in a single system-wide location. They have their own hacks for getting dealing with it. I don't know which way of doing it is technically superior, all I know is none of them offer a developer experience that I'd choose for myself if I were designing a new programming language in this day and age. FWIW, I give Python some credit for coming up with a solution that doesn't rely on containerization, and therefore will work consistently on many OSes, and not just Linux. Containerization is an option, too. It's really down to whether you need a system that works for one language on many OSes, or one that works for many languages on one OS. Or, if you'd rather keep it really simple, there are zipapps, which are Python's answer to fat binaries. Similar to uberjars in Java. Or to good ol' static compilation in most the languages I like to use even when someone isn't paying me to do it. (That said, there is one thing about environment management in Python that absolutely drives me up the wall: There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Which sucks for everyone, and makes learning the ropes without a good mentor way more of a chore than it needs to be. That's not too far off from C, either, though.)
- skohan 7y ago
  > There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way Yeah I guess this is really my main issue. Like it seems like every Python project needs to come with a README to explain which tools are required, and maybe even which order they have to be applied to get the desired result. It just feels like an almost absurd step back when compared to other language ecosystems where these are well-solved problems.
- qqssccfftt 7y ago
  > There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Generally, newer projects are settling around Poetry.
klyrs 7y ago
> It seems like you practically need to containerize a python code-base to make it repeatable. Most serious production workflows are containerized these days -- are you sure that this supports your conclusion?
- skohan 7y ago
  It makes sense to me that I might want to containerize a project when deploying it to production to guarantee a consistent environment. It seems strange to me that I effectively have to containerize all projects of a certain language on my own workstation because they might have side-effects on each other.
  - klyrs 7y ago
    Don't get me wrong, I hate that too. I simply disagree with your assertion that Python isn't production-ready on this basis, because of what a "production" environment typically looks like. Very different from your workstation and mine.
    - skohan 7y ago
      Yeah I mean that's just part of the story. I am also considering the fact that Python's type system and the fact that it's an interpreted language mean that you are going to have to put a lot of effort into testing for issues which can be caught by other languages at compile time, and the fact that you're leaving a lot of performance on the table. It's not that Python is a "bad language", but all these things taken together just basically work against everything I am trying to achieve in a solid production environment.

by skohan 7y ago

danpalmer 7y ago

> led me to the conclusion that it's not really suited for serious production software. While I get that you may have been frustrated with a Python setup, concluding that it's not suitable for "serious" production software isn't a proportionate response, and the evidence of many companies using it for "serious" production systems suggests that it is in fact fine. Examples include Instagram and Dropbox. > It seems like you practically need to containerize a python code-base to make it repeatable. This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary. Arguably even for anything that doesn't produce completely static binaries. Yes Python has a learning curve for production use, all languages and ecosystems do. In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. Python is not out of the ordinary, it's not the worst. It's not the best either, it's fairly reasonable for its age, complexity, and typical uses.

skohan 7y ago
> While I get that you may have been frustrated with a Python setup It's not so much that I was frustrated by this, it's more the case that given the other ecosystems I have worked with, I was kind of shocked that people actually manage to ship projects with this type of tooling. > This is pretty much the case for any interpreted language that has dependencies that aren't compiled into the binary I don't agree. I think npm does an excellent job of making js packages repeatable for example. > In Python a fair bit of this happens to be on deployment, rather than in, say, compilation stages. To be honest, that's a big part of the reason I'm not a fan of interpreted languages in general in production environments. I think something like Python is fine for a setting like a notebook where you only have to worry about your own workstation, and you just want to iterate quickly without a ton of constraints on performance or robustness, but for running code in an un-supervised environment? I would much rather have a compiler checking every possible issue ahead of time rather than relying on testing and monitoring at runtime. And Python seems particularly bad, because if I have to use a virtual environment to get the same code to run consistently on my workstation and my colleagues, that just gives me no confidence that I know how it's going to behave in some cloud execution environment. I'm not saying Python is "the worst" but basically all the properties of Python as a language ecosystem seem to run counter to what you want in order to produce robust, reliable production software.
- silvester23 7y ago
  I haven't worked with npm in quite a while but as far as I remember, it does pretty much the same as pip with a virtual env and a requirements file -- it installs all dependencies into a directory for that specific project. The ergonomics might be better with npm but they both solve the problem of consistent execution across different machines pretty similarly.
  - skohan 7y ago
    The big difference I have seen is that with npm I have never run into issues because the code is being executed with the incorrect version of the interpreter, or something weird like python being installed at an unexpected path. Also npm is just one tool which behaves consistently and takes full responsibility of making a project portable. With Python it seems as if it's at the whim of the developer to choose from a variety of tools to complete this task. In other words, if I see `pacakage.json` in my project directory, I know it will tell me everything I need to know to run the project. With Python, if the previous developer hasn't documented what tools they were using, I might have to do some archeology to understand how they have encapsulated the project's dependencies. Or worse, if they never encapsulated their project's dependencies, I might just have to guess how the environment was set up on their workstation and try to recreate it through trial and error.
    - wrmsr 7y ago
      As below node has the luxuries of both having been able to learn lessons from python and ruby ~and~ not being a victim of its own success of being already integrated with and a dependency of most major operating systems. (That said javascript the language definitely ~is~ a victim of its own success in browsers - there's probably some metaphorical parallel here..) But the 'python version' problem (and really just any notion of a 'system python' in general) is still absolutely one of the bigger pain points of the ecosystem. I can say that for nix OS's pyenv ( https://github.com/pyenv/pyenv/ https://github.com/pyenv/pyenv/ ) is a real godsend for this though. Pure-bash, no bootstrap problem, doesn't need to be 'installed' / can just be cloned and used, and has been able to reliably install any version of python I've needed into any of my relevant platforms in a well isolated ~/.pyenv/version/v3.x.y from which I can '~/.pyenv/versions/v3.x.y/bin/python -m venv .venv' in my project's dev bootstrap/Makefile. It's not part of my production-bound Dockerfiles which just base off of the official python images but it's invaluable for development and testing. If you are going to use it I definitely recommend cloning it directly to ~/.pyenv rather than installing it via brew or somesuch as those tend to lag further behind the most up-to-date python versions than I'd like. And this is of course in no way a general solution to all of python's many packaging woes, and it'll probably never really be as simple as packages.json for many very valid if unfortunate reasons, but the 'python version' problem is at least pretty well handled by pyenv.

wwright 7y ago

Have you worked with C++ much? It has the same problems to a much, much worse extent.

skohan 7y ago
I probably would not choose C++ either unless I really needed its USP's like extreme memory/performance optimization or required compatibility with exotic hardware. But at least with C++ you can compile a binary which won't behave differently on a different system most of the time. But there are a lot of other languages out there besides C++ and Python.
- fnord123 7y ago
  > But at least with C++ you can compile a binary which won't behave differently on a different system most of the time. Not really, no. Only if the compiler, compiler version, compilation flags, kernel version, platform, platform version, cpu version, directory structure, shared object versions, locale, environment (LD_PRELOAD, LD_LIBRARY_PATH, TZ), config in pwd, .local, and /etc or maybe $XDG_*, "~/Library/Application Support/" are all the same.
  - skohan 7y ago
    Fair but I think my other points still stand
  - KMag 7y ago
    The GP specifically was talking about the binary behaving differently, so the compiler version and compilation flags would be irrelevant. The kernel, system libraries, environment variables, etc. would all still be relevant, though. I think they're referring to unpredictable behavior under future versions of language VMs and garbage collectors vs. statically compiled binaries with either no GC or a statically linked GC library. Most of the time, all of that machinery going on behind the scenes is very convenient, but coding strictly to POSIX and the C/C++ standard libraries / STL and keeping your compilation artifacts gives you a lot more stability than coding to some version of the Java standard library (or other language VM) and keeping around your JVM installer with hopes that the installer will agree to install and run on future systems. I remember being bitten by testing our product against the earliest JRE we supported and also the latest JRE sun had released at the time. I was aware that reading environment variables was deprecated in the earliest JRE I tested with, and I knew it had been un-deprecated in the latest JRE I tested, and the methods I was using were present in all of the JRE versions in between. What I had missed was that there was at least one JRE version in the middle where the deprecated environment variable reading methods actually threw runtime errors. I've also run into issues in the past with the version of MSVC I was required to use having an installer that depended upon some ancient version of the CLR which would no longer install on my machine. There were some workarounds, but I needed to talk to several of our helpdesk people before I found someone who knew the required magic incantations. Had the MSVC installer been coded strictly to C++ stdlib, STL, Win32 and MFC, it would have run just fine on my modern desktop.

mumblemumble 7y ago

I used to think that, end then I just realized that this is really just Python showing its roots as a relatively old programming language with Unix roots. C and C++ have a a fairly similar problem with dependency hell due to shared libraries being installed in a single system-wide location. They have their own hacks for getting dealing with it. I don't know which way of doing it is technically superior, all I know is none of them offer a developer experience that I'd choose for myself if I were designing a new programming language in this day and age. FWIW, I give Python some credit for coming up with a solution that doesn't rely on containerization, and therefore will work consistently on many OSes, and not just Linux. Containerization is an option, too. It's really down to whether you need a system that works for one language on many OSes, or one that works for many languages on one OS. Or, if you'd rather keep it really simple, there are zipapps, which are Python's answer to fat binaries. Similar to uberjars in Java. Or to good ol' static compilation in most the languages I like to use even when someone isn't paying me to do it. (That said, there is one thing about environment management in Python that absolutely drives me up the wall: There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Which sucks for everyone, and makes learning the ropes without a good mentor way more of a chore than it needs to be. That's not too far off from C, either, though.)

skohan 7y ago
> There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way Yeah I guess this is really my main issue. Like it seems like every Python project needs to come with a README to explain which tools are required, and maybe even which order they have to be applied to get the desired result. It just feels like an almost absurd step back when compared to other language ecosystems where these are well-solved problems.
qqssccfftt 7y ago
> There are more than a fistful of competing tools for doing it, and a zillion different ways to work with them, and no clear preferred way. Generally, newer projects are settling around Poetry.

klyrs 7y ago

> It seems like you practically need to containerize a python code-base to make it repeatable. Most serious production workflows are containerized these days -- are you sure that this supports your conclusion?

skohan 7y ago
It makes sense to me that I might want to containerize a project when deploying it to production to guarantee a consistent environment. It seems strange to me that I effectively have to containerize all projects of a certain language on my own workstation because they might have side-effects on each other.
- klyrs 7y ago
  Don't get me wrong, I hate that too. I simply disagree with your assertion that Python isn't production-ready on this basis, because of what a "production" environment typically looks like. Very different from your workstation and mine.
  - skohan 7y ago
    Yeah I mean that's just part of the story. I am also considering the fact that Python's type system and the fact that it's an interpreted language mean that you are going to have to put a lot of effort into testing for issues which can be caught by other languages at compile time, and the fact that you're leaving a lot of performance on the table. It's not that Python is a "bad language", but all these things taken together just basically work against everything I am trying to achieve in a solid production environment.