6 ms·

I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static comp

by mqus 6y ago

I have the opinion that this is because "Deployment" is not a solved issue (or clearly defined and enforced) for Python. C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy). Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.

nicoburns 6y ago
Python should just do what JavaScript does and install all dependencies to a project-local folder.
- StreamBright 6y ago
  Not sure if JavaScript has anything that any other language or development tool should copy.
  - wilsonrocks 6y ago
    It is a mess... But you can often get a project running an awful lot quicker than in python
    - notdonspaulding 6y ago
      True, but if that's the metric we're basing our language choice off of then we should all go back to PHP.
      - PalzyPunk 6y ago
        Yeah no kidding. I prefer it as a prototyping language. Unfortunately, I work in networking, and it's become the raison d'etre for our "DevOps". At least we're not using Ansible anymore... :|
  - nicoburns 6y ago
    JavaScript's package management is significantly better than Pythons: - There's one tool to use, and it comes pre-installed - You don't have to deal with virtualenvs or packages from different projects conflicting. - Native code dependencies "just work".
    - dboreham 6y ago
      That one tool must be yarnnpmgypbabelwebpack?
      - nicoburns 6y ago
        Babel and Webpack aren't package management tools. I guess there is now yarn in addition to NPM. But NPM works just fine. And furthermore they interoperate.
      - beckingz 6y ago
        which gets defined in Yet Another YAML Language
    - toyg 6y ago
      > You don't have to deal with virtualenvs or packages from different projects conflicting. That's only because you're basically assured there will be conflicts inside any single project. Every time I install anything via npm I get a boatload of warnings about insecure dependencies, which are effectively impossible to fix without breaking the whole mountain of hacks. There are good deployment stories out there. JS is not one of them.
      - nicoburns 6y ago
        > That's only because you're basically assured there will be conflicts inside any single project. That's not an issue with the package management though, that's an issue with the quality of the packages themselves. The JS ecosystem does have issues there. But the tooling itself is good.
- HelloNurse 6y ago
  You are assuming there is a "project", but Python code with massive dependencies can (and often should) be a bare script in an arbitrary folder that runs by invoking the Python interpreter on it, without "installing" anything except missing dependencies.
  - hctaw 6y ago
    That's not a bare script then, it's a project with dependencies that need to be installed. This misunderstanding is how you fuck up a developer environment.
- IgorPartola 6y ago
  Almost nothing should do what JavaScript does, not even JavaScript. Of all the interpreted languages I’ve used, JS has the single worst ecosystem for running production code. If your experience is primarily with JS, do yourself a favor and try to grok other languages and ecosystems.
- StavrosK 6y ago
  It does: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ Though, as the sibling comments say, it's debatable whether it should.
- BiteCode_dev 6y ago
  That's literally what the article is about.
- hctaw 6y ago
  I'm surprised this is getting downvoted. This is the only sane way to manage dependencies. They must be relative to the thing you care about. There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
  - paulie_a 6y ago
    They essentially are. Not in the actual directory but in the virtual env and not cluttering up your project directory. This is superior to JavaScript in my opinion No language should replicate anything JavaScript does when it comes to package management
pjfin123 6y ago
This video really helped me understand Python packaging of binaries: https://youtu.be/02aAZ8u3wEQ https://youtu.be/02aAZ8u3wEQ
BerislavLopac 6y ago
Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it. On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.
- isignal 6y ago
  All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make). The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
  - curryst 6y ago
    I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller <spec file>" and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test. > All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.

by mqus 6y ago

nicoburns 6y ago

Python should just do what JavaScript does and install all dependencies to a project-local folder.

StreamBright 6y ago
Not sure if JavaScript has anything that any other language or development tool should copy.
- wilsonrocks 6y ago
  It is a mess... But you can often get a project running an awful lot quicker than in python
  - notdonspaulding 6y ago
    True, but if that's the metric we're basing our language choice off of then we should all go back to PHP.
    - PalzyPunk 6y ago
      Yeah no kidding. I prefer it as a prototyping language. Unfortunately, I work in networking, and it's become the raison d'etre for our "DevOps". At least we're not using Ansible anymore... :|
- nicoburns 6y ago
  JavaScript's package management is significantly better than Pythons: - There's one tool to use, and it comes pre-installed - You don't have to deal with virtualenvs or packages from different projects conflicting. - Native code dependencies "just work".
  - dboreham 6y ago
    That one tool must be yarnnpmgypbabelwebpack?
    - nicoburns 6y ago
      Babel and Webpack aren't package management tools. I guess there is now yarn in addition to NPM. But NPM works just fine. And furthermore they interoperate.
    - beckingz 6y ago
      which gets defined in Yet Another YAML Language
  - toyg 6y ago
    > You don't have to deal with virtualenvs or packages from different projects conflicting. That's only because you're basically assured there will be conflicts inside any single project. Every time I install anything via npm I get a boatload of warnings about insecure dependencies, which are effectively impossible to fix without breaking the whole mountain of hacks. There are good deployment stories out there. JS is not one of them.
    - nicoburns 6y ago
      > That's only because you're basically assured there will be conflicts inside any single project. That's not an issue with the package management though, that's an issue with the quality of the packages themselves. The JS ecosystem does have issues there. But the tooling itself is good.
HelloNurse 6y ago
You are assuming there is a "project", but Python code with massive dependencies can (and often should) be a bare script in an arbitrary folder that runs by invoking the Python interpreter on it, without "installing" anything except missing dependencies.
- hctaw 6y ago
  That's not a bare script then, it's a project with dependencies that need to be installed. This misunderstanding is how you fuck up a developer environment.
IgorPartola 6y ago
Almost nothing should do what JavaScript does, not even JavaScript. Of all the interpreted languages I’ve used, JS has the single worst ecosystem for running production code. If your experience is primarily with JS, do yourself a favor and try to grok other languages and ecosystems.
StavrosK 6y ago
It does: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ Though, as the sibling comments say, it's debatable whether it should.
BiteCode_dev 6y ago
That's literally what the article is about.
hctaw 6y ago
I'm surprised this is getting downvoted. This is the only sane way to manage dependencies. They must be relative to the thing you care about. There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
- paulie_a 6y ago
  They essentially are. Not in the actual directory but in the virtual env and not cluttering up your project directory. This is superior to JavaScript in my opinion No language should replicate anything JavaScript does when it comes to package management

pjfin123 6y ago

This video really helped me understand Python packaging of binaries: https://youtu.be/02aAZ8u3wEQ https://youtu.be/02aAZ8u3wEQ

BerislavLopac 6y ago

Or to be precise, it is a solved issue -- several times over. After years of meandering, I think it is safe to say that deployment of Python libraries is pretty much resolved in one, general way: we have the wheel/PyPI/pip toolchain, and that's about it. On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.

isignal 6y ago
All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. End users cannot set up virtual envs and pip install stuff into them (which sometimes needs external tools like make). The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
- curryst 6y ago
  I've used pyinstaller. My experience was that pure Python was easy; things that needed to use C shared objects increased in complexity exponentially based on the number of C dependencies. If it's pure Python, you can basically run "pyinstaller <spec file>" and be done. I tried to make PyQt5 work in it, and it was extremely painful. I don't think I ever actually made it work, I think I just gave up. That being said, PyQt5 is probably the most complicated thing you'd ever package with pyinstaller, so it may not be a great test. > All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users. virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.