4 ms·

I recently had to package a machine learning Jupyter notebook created by a very smart person, albeit a scientist, not a developer. Getting the tool somehow into

by 9dev 5y ago

I recently had to package a machine learning Jupyter notebook created by a very smart person, albeit a scientist, not a developer. Getting the tool somehow into production, making it reproducible, testable, and maintainable, has proven to be a major headache. Before, I only casually dabbled in the python world, but this was the first time I had to care about packaging, dependencies, CI and the like. Turns out, as TFA said, nobody knows how to package python apps right. For what it’s worth, I wasn’t even able to find some kind of best practice to manage friggin dependencies. There’s like a myriad of package managers, all of them work differently, and nobody seemed to had something like redistributing an app to other people on their mind. Coming from PHP, JavaScript and Go, this was utterly ridiculous to me. Go ahead, tell me I got it all wrong and it’s really easy using tool Xyz, but if a developer with some experience under their belt isn’t able to figure this out in a few days, things are just broken.

worik 5y ago
It has taken everybody else thirty years to realise: Python is not very good....
- burnished 5y ago
  Python is fucking phenomenal. Packaging python is an exercise in frustration until you become a level 3 wizard.
  - nerdponx 5y ago
    It really isn't that hard to package for Python. Certainly a lot easier than packaging for Debian, and not much different from packaging for Ruby or Node. The bad part is the Setuptools documentation, which is slowly improving, but is (and has been for years) so bad that almost nobody can learn from reading it.
    - burnished 5y ago
      I think we might be in agreement because I see that these >>Packaging python is an exercise in frustration >>Setuptools documentation [...] is [...] so bad that almost nobody can learn from reading it and these >> until you become a level 3 wizard >> It really isn't that hard to package for Python. as being fundamentally equivalent statements.
nerdponx 5y ago
I have to ask, did you read https://packaging.python.org/ https://packaging.python.org/ ? I find that the majority of the people who find difficulty with packaging are really just struggling with documentation and ecosystem fragmentation, as you hinted at. For example, that page I linked weirdly focuses on one specific tool (Pipenv) and doesn't attempt to survey the landscape of packaging and dependency management tools. There tends to be a poor balance of "explanation", "example", and "overview" in these documents. I would say that I don't quite know why, but I know that writing good docs is incredibly difficult, so I have sympathy for people who've tried and didn't quite succeed. One thing I'll offer is that, in the Python world, "managing dependencies" is distinct from "packaging". This seems to be different from other language ecosystems, where there is 1 package management tool that does everything. That said, I take serious issue with the claim in the article that you quoted: > Nobody knows how to correctly install and package Python apps. This simply is not true. There are lots and lots of Python applications that are packaged and that you can install without a problem. You can complain about the need for virtual environments instead of having "app-local" packages by default, sure. But that's not the issue in most cases. I think ultimately this amounts to FUD. Not because your experience is invalid, but because the blame is put on the wrong thing. The fact that Python packaging documentation is generally dogshit doesn't mean that the actual process of making a Python package is hard. If someone sat with you and showed you how to set things up the first time, I am confident that you would have no trouble cranking out Python packages that you can distribute to your engineering team without any problems. > I wasn’t even able to find some kind of best practice to manage friggin dependencies Again, this is a documentation problem. > There’s like a myriad of package managers, all of them work differently. Not really, Pip is still mostly the only package manager in town. But there are several tools like Pipenv and Poetry that replace the old Setuptools when it comes to actually building and installing packages, as well as generating lockfiles for dependencies (which Setuptools does not do at all). I don't find it particularly bad that there are a few different tools to do similar jobs. I do however find it upsetting that the Python packaging document I linked doesn't even attempt to describe them or explain when/why you would use one or the other. > nobody seemed to had something like redistributing an app to other people on their mind. That's just outright untrue. Otherwise, PyPI wouldn't exist. Again, this sounds like a documentation problem. > machine learning Jupyter notebook The other issue here is that you're looking at a machine learning research script, not an "application" as such. Machine learning libraries tend to have complicated dependencies with bindings to C and C++ libraries that might or might not be bundled with the packages themselves. This would be and is a problem in any language ecosystem, e.g. Node or Ruby or Perl. These are NOT easy applications when it comes to packaging. Another issue with your scientist's code is that they were very likely using Conda, which is kind of its own thing. Conda is somewhat language-agnostic, and largely bypasses all the established Python "developer" tooling, in pursuit of somewhat-reproducible computing environments in support of reproducible research. Conda is very popular precisely because machine learning libraries tend to be so complicated to package correctly, and Conda solves that problem as a kind of portable alternative to Deb, RPM, et al. Unfortunately, Conda poses some challenges if/when you need to take a project that lives in a Conda environment and transfer it to a more typical Python package, mostly because Conda environments can control things like the C compiler, while Python package managers cannot. However, I will insist that this is not a Python-specific problem. If they wrote their application in Clojure, for example, you would have the exact same issue in migrating it from Conda to the usual suite of Clojure tools. Oh, and speaking of Clojure: Python isn't the only language with the "which tool do I use and why are they all so complicated?" problem. I don't have a goddamn clue how to package for Clojure (I tried!), but I'm not going off a forum about how nobody should use Clojure for internal tools.

by 9dev 5y ago

worik 5y ago

It has taken everybody else thirty years to realise: Python is not very good....

burnished 5y ago
Python is fucking phenomenal. Packaging python is an exercise in frustration until you become a level 3 wizard.
- nerdponx 5y ago
  It really isn't that hard to package for Python. Certainly a lot easier than packaging for Debian, and not much different from packaging for Ruby or Node. The bad part is the Setuptools documentation, which is slowly improving, but is (and has been for years) so bad that almost nobody can learn from reading it.
  - burnished 5y ago
    I think we might be in agreement because I see that these >>Packaging python is an exercise in frustration >>Setuptools documentation [...] is [...] so bad that almost nobody can learn from reading it and these >> until you become a level 3 wizard >> It really isn't that hard to package for Python. as being fundamentally equivalent statements.

nerdponx 5y ago

I have to ask, did you read https://packaging.python.org/ https://packaging.python.org/ ? I find that the majority of the people who find difficulty with packaging are really just struggling with documentation and ecosystem fragmentation, as you hinted at. For example, that page I linked weirdly focuses on one specific tool (Pipenv) and doesn't attempt to survey the landscape of packaging and dependency management tools. There tends to be a poor balance of "explanation", "example", and "overview" in these documents. I would say that I don't quite know why, but I know that writing good docs is incredibly difficult, so I have sympathy for people who've tried and didn't quite succeed. One thing I'll offer is that, in the Python world, "managing dependencies" is distinct from "packaging". This seems to be different from other language ecosystems, where there is 1 package management tool that does everything. That said, I take serious issue with the claim in the article that you quoted: > Nobody knows how to correctly install and package Python apps. This simply is not true. There are lots and lots of Python applications that are packaged and that you can install without a problem. You can complain about the need for virtual environments instead of having "app-local" packages by default, sure. But that's not the issue in most cases. I think ultimately this amounts to FUD. Not because your experience is invalid, but because the blame is put on the wrong thing. The fact that Python packaging documentation is generally dogshit doesn't mean that the actual process of making a Python package is hard. If someone sat with you and showed you how to set things up the first time, I am confident that you would have no trouble cranking out Python packages that you can distribute to your engineering team without any problems. > I wasn’t even able to find some kind of best practice to manage friggin dependencies Again, this is a documentation problem. > There’s like a myriad of package managers, all of them work differently. Not really, Pip is still mostly the only package manager in town. But there are several tools like Pipenv and Poetry that replace the old Setuptools when it comes to actually building and installing packages, as well as generating lockfiles for dependencies (which Setuptools does not do at all). I don't find it particularly bad that there are a few different tools to do similar jobs. I do however find it upsetting that the Python packaging document I linked doesn't even attempt to describe them or explain when/why you would use one or the other. > nobody seemed to had something like redistributing an app to other people on their mind. That's just outright untrue. Otherwise, PyPI wouldn't exist. Again, this sounds like a documentation problem. > machine learning Jupyter notebook The other issue here is that you're looking at a machine learning research script, not an "application" as such. Machine learning libraries tend to have complicated dependencies with bindings to C and C++ libraries that might or might not be bundled with the packages themselves. This would be and is a problem in any language ecosystem, e.g. Node or Ruby or Perl. These are NOT easy applications when it comes to packaging. Another issue with your scientist's code is that they were very likely using Conda, which is kind of its own thing. Conda is somewhat language-agnostic, and largely bypasses all the established Python "developer" tooling, in pursuit of somewhat-reproducible computing environments in support of reproducible research. Conda is very popular precisely because machine learning libraries tend to be so complicated to package correctly, and Conda solves that problem as a kind of portable alternative to Deb, RPM, et al. Unfortunately, Conda poses some challenges if/when you need to take a project that lives in a Conda environment and transfer it to a more typical Python package, mostly because Conda environments can control things like the C compiler, while Python package managers cannot. However, I will insist that this is not a Python-specific problem. If they wrote their application in Clojure, for example, you would have the exact same issue in migrating it from Conda to the usual suite of Clojure tools. Oh, and speaking of Clojure: Python isn't the only language with the "which tool do I use and why are they all so complicated?" problem. I don't have a goddamn clue how to package for Clojure (I tried!), but I'm not going off a forum about how nobody should use Clojure for internal tools.