4 ms·

I have to ask, did you read https://packaging.python.org/ https://packaging.python.org/ ? I find that the majority of the people who find difficulty with packa

I have to ask, did you read https://packaging.python.org/ https://packaging.python.org/ ? I find that the majority of the people who find difficulty with packaging are really just struggling with documentation and ecosystem fragmentation, as you hinted at. For example, that page I linked weirdly focuses on one specific tool (Pipenv) and doesn't attempt to survey the landscape of packaging and dependency management tools. There tends to be a poor balance of "explanation", "example", and "overview" in these documents. I would say that I don't quite know why, but I know that writing good docs is incredibly difficult, so I have sympathy for people who've tried and didn't quite succeed. One thing I'll offer is that, in the Python world, "managing dependencies" is distinct from "packaging". This seems to be different from other language ecosystems, where there is 1 package management tool that does everything. That said, I take serious issue with the claim in the article that you quoted: > Nobody knows how to correctly install and package Python apps. This simply is not true. There are lots and lots of Python applications that are packaged and that you can install without a problem. You can complain about the need for virtual environments instead of having "app-local" packages by default, sure. But that's not the issue in most cases. I think ultimately this amounts to FUD. Not because your experience is invalid, but because the blame is put on the wrong thing. The fact that Python packaging documentation is generally dogshit doesn't mean that the actual process of making a Python package is hard. If someone sat with you and showed you how to set things up the first time, I am confident that you would have no trouble cranking out Python packages that you can distribute to your engineering team without any problems. > I wasn’t even able to find some kind of best practice to manage friggin dependencies Again, this is a documentation problem. > There’s like a myriad of package managers, all of them work differently. Not really, Pip is still mostly the only package manager in town. But there are several tools like Pipenv and Poetry that replace the old Setuptools when it comes to actually building and installing packages, as well as generating lockfiles for dependencies (which Setuptools does not do at all). I don't find it particularly bad that there are a few different tools to do similar jobs. I do however find it upsetting that the Python packaging document I linked doesn't even attempt to describe them or explain when/why you would use one or the other. > nobody seemed to had something like redistributing an app to other people on their mind. That's just outright untrue. Otherwise, PyPI wouldn't exist. Again, this sounds like a documentation problem. > machine learning Jupyter notebook The other issue here is that you're looking at a machine learning research script, not an "application" as such. Machine learning libraries tend to have complicated dependencies with bindings to C and C++ libraries that might or might not be bundled with the packages themselves. This would be and is a problem in any language ecosystem, e.g. Node or Ruby or Perl. These are NOT easy applications when it comes to packaging. Another issue with your scientist's code is that they were very likely using Conda, which is kind of its own thing. Conda is somewhat language-agnostic, and largely bypasses all the established Python "developer" tooling, in pursuit of somewhat-reproducible computing environments in support of reproducible research. Conda is very popular precisely because machine learning libraries tend to be so complicated to package correctly, and Conda solves that problem as a kind of portable alternative to Deb, RPM, et al. Unfortunately, Conda poses some challenges if/when you need to take a project that lives in a Conda environment and transfer it to a more typical Python package, mostly because Conda environments can control things like the C compiler, while Python package managers cannot. However, I will insist that this is not a Python-specific problem. If they wrote their application in Clojure, for example, you would have the exact same issue in migrating it from Conda to the usual suite of Clojure tools. Oh, and speaking of Clojure: Python isn't the only language with the "which tool do I use and why are they all so complicated?" problem. I don't have a goddamn clue how to package for Clojure (I tried!), but I'm not going off a forum about how nobody should use Clojure for internal tools.

xcambar 5y ago
After reading the linked documentation, it seems that it is fairly easy to package a python package. But building a python application (CLI) has another dedicated page that is much scarier and much less easy to follow: https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/
disgruntledphd2 5y ago
> Unfortunately, Conda poses some challenges if/when you need to take a project that lives in a Conda environment and transfer it to a more typical Python package, mostly because Conda environments can control things like the C compiler, while Python package managers cannot. To be honest, having been in this situation it generally makes more sense to implement the pip dependencies inside the conda venv (by running pip), as conda can handle the C-level dependencies and pip is mostly perfect (modulo back when it didn't actually manage dependencies) for pure python stuff. Also, it sounds like you know a lot about python packaging, please by the love of all that's holy, write this stuff down somewhere other than HN. There really isn't a lot of good docs around this.
- nerdponx 5y ago
  That's pretty much how I've done the Conda transition too. Switch to using Pip inside the Conda env to suss out what the non-Python deps are. Also a lot of packages are distributed on PyPI as binary packages now, so as long as you aren't using Musl in production you should just be able to use those instead. > Also, it sounds like you know a lot about python packaging, please by the love of all that's holy, write this stuff down somewhere other than HN. There really isn't a lot of good docs around this. Believe me, I want to! I am pretty bad at managing my "project/writing wishlist".
9dev 5y ago
Let me preface this by saying thank you for your insightful response, and the many suggestions - I'm definitely going to revalidate some of my assumptions and try to package that app again! I peaked at the documentation, but found it both overwhelming and unfocused, thus hard to follow. As you say, it's focused on specific tools and feels more like a reference than actual instructions for what I was aiming to do. > One thing I'll offer is that, in the Python world, "managing dependencies" is distinct from "packaging". This seems to be different from other language ecosystems, where there is 1 package management tool that does everything. That might be the actual thing that stumped me: I don't intend on publishing my package somewhere publicly, but simply enable other devs on the team to install the required dependencies with a single command, without any assumptions on their environment, which seems awkward to me. > That said, I take serious issue with the claim in the article that you quoted: >> Nobody knows how to correctly install and package Python apps. > This simply is not true. There are lots and lots of Python applications that are packaged and that you can install without a problem. You can complain about the need for virtual environments instead of having "app-local" packages by default, sure. But that's not the issue in most cases. That is because we understood that quote differently. In your post you mention Pip, Pipenv, Poetry, Setuptools, PyPI, and Conda. Additionally I came across Virtualenv, and Pyenv. All of these tools are somehow related to python packages and dependency management, and everyone on the internet seems to have their own opinion one which combination is the best. There seems to be no consensus on one of the most foundational tasks in software engineering, namely sharing code with other people. To me that counts as nobody knowing how to it correctly. That said, reading the docs again, which clearly recommend using Pipenv - and that seems to be simple enough - I probably should have headed for the official docs right away. > These are NOT easy applications when it comes to packaging. I noticed that :) Caused me to hit the wall multiple times, for example while using Alpine Linux/MUSL and wondering why the C bindings had to be compiled on every install. But you're right, that would bring its own complexities in every language and might be partially responsible for my general frustration with python dependencies. While the scientist didn't use Conda (I think he just happily installs everything he needs globally), that was the recommendation I found on one of his primary packages' website (I think it was pandas), so I followed through and tried to set things up using Conda. It was a nightmare, frankly. I'm really not opposed to learning new things. And I get that my assumption that packages live inside an application directory by default might be just that, and colored by other ecosystems I'm more used to. And, as I said earlier, I'm definitely going to check out Pipenv and try to refactor that thing into something more standards-compliant. Still, I think Python places more roadblocks than necessary in your way, and the combination of bad docs, several opinions on how to do basic stuff, as well as the bad habit of global dependencies, make it difficult for newbies to deal with Python. Which, all in all, confirms the statement in TFA for me: Don't use Python for internal tools. And frankly, as someone affected by Python packaging, I'm definitely going to complain on forums as much as I want if things are harder than they have to be. Just because other systems have their warts too doesn't mean there's nothing to improve.

xcambar 5y ago

After reading the linked documentation, it seems that it is fairly easy to package a python package. But building a python application (CLI) has another dedicated page that is much scarier and much less easy to follow: https://packaging.python.org/tutorials/packaging-projects/ https://packaging.python.org/tutorials/packaging-projects/

disgruntledphd2 5y ago

> Unfortunately, Conda poses some challenges if/when you need to take a project that lives in a Conda environment and transfer it to a more typical Python package, mostly because Conda environments can control things like the C compiler, while Python package managers cannot. To be honest, having been in this situation it generally makes more sense to implement the pip dependencies inside the conda venv (by running pip), as conda can handle the C-level dependencies and pip is mostly perfect (modulo back when it didn't actually manage dependencies) for pure python stuff. Also, it sounds like you know a lot about python packaging, please by the love of all that's holy, write this stuff down somewhere other than HN. There really isn't a lot of good docs around this.

nerdponx 5y ago
That's pretty much how I've done the Conda transition too. Switch to using Pip inside the Conda env to suss out what the non-Python deps are. Also a lot of packages are distributed on PyPI as binary packages now, so as long as you aren't using Musl in production you should just be able to use those instead. > Also, it sounds like you know a lot about python packaging, please by the love of all that's holy, write this stuff down somewhere other than HN. There really isn't a lot of good docs around this. Believe me, I want to! I am pretty bad at managing my "project/writing wishlist".

9dev 5y ago

Let me preface this by saying thank you for your insightful response, and the many suggestions - I'm definitely going to revalidate some of my assumptions and try to package that app again! I peaked at the documentation, but found it both overwhelming and unfocused, thus hard to follow. As you say, it's focused on specific tools and feels more like a reference than actual instructions for what I was aiming to do. > One thing I'll offer is that, in the Python world, "managing dependencies" is distinct from "packaging". This seems to be different from other language ecosystems, where there is 1 package management tool that does everything. That might be the actual thing that stumped me: I don't intend on publishing my package somewhere publicly, but simply enable other devs on the team to install the required dependencies with a single command, without any assumptions on their environment, which seems awkward to me. > That said, I take serious issue with the claim in the article that you quoted: >> Nobody knows how to correctly install and package Python apps. > This simply is not true. There are lots and lots of Python applications that are packaged and that you can install without a problem. You can complain about the need for virtual environments instead of having "app-local" packages by default, sure. But that's not the issue in most cases. That is because we understood that quote differently. In your post you mention Pip, Pipenv, Poetry, Setuptools, PyPI, and Conda. Additionally I came across Virtualenv, and Pyenv. All of these tools are somehow related to python packages and dependency management, and everyone on the internet seems to have their own opinion one which combination is the best. There seems to be no consensus on one of the most foundational tasks in software engineering, namely sharing code with other people. To me that counts as nobody knowing how to it correctly. That said, reading the docs again, which clearly recommend using Pipenv - and that seems to be simple enough - I probably should have headed for the official docs right away. > These are NOT easy applications when it comes to packaging. I noticed that :) Caused me to hit the wall multiple times, for example while using Alpine Linux/MUSL and wondering why the C bindings had to be compiled on every install. But you're right, that would bring its own complexities in every language and might be partially responsible for my general frustration with python dependencies. While the scientist didn't use Conda (I think he just happily installs everything he needs globally), that was the recommendation I found on one of his primary packages' website (I think it was pandas), so I followed through and tried to set things up using Conda. It was a nightmare, frankly. I'm really not opposed to learning new things. And I get that my assumption that packages live inside an application directory by default might be just that, and colored by other ecosystems I'm more used to. And, as I said earlier, I'm definitely going to check out Pipenv and try to refactor that thing into something more standards-compliant. Still, I think Python places more roadblocks than necessary in your way, and the combination of bad docs, several opinions on how to do basic stuff, as well as the bad habit of global dependencies, make it difficult for newbies to deal with Python. Which, all in all, confirms the statement in TFA for me: Don't use Python for internal tools. And frankly, as someone affected by Python packaging, I'm definitely going to complain on forums as much as I want if things are harder than they have to be. Just because other systems have their warts too doesn't mean there's nothing to improve.