5 ms·

I'm militantly disinterested in the performance of Python until the deployment style of most Python projects is no longer "hunter-gatherer-style installs".

by bborud 5y ago

aazaa 5y ago
What are "hunter-gatherer-style installs?"
- taeric 5y ago
  I take that as "keep running pip install on yet another discovered dep until the script works." And heaven help you if you accidentally get conflicting versions installed in the same venv. (By conflicting, I mean two libraries that both depend on different versions of the same underlying library. Odds of this increase dramatically the more you use.)
  - deleted 5y ago
    [deleted]
  - orf 5y ago
    Pip now checks this, but you usually use something like Poetry to give you a concrete lockfile with the correct transient dependencies locked in.
    - taeric 5y ago
      My "heaven help you" was more that, if you are in that situation, you don't really have an exit strategy in python. Do you?
  - singularity2001 5y ago
    yeah people are often lazy about adding a requirements.txt
  - joconde 5y ago
    Neither of these problems have happened to me since pip introduced their new dependency resolver (I think sometime during 2020). All dependencies are installed, and packages in the default repo usually have consistent dependency lists. It's another story for conda, which flat-out refused to install my list of around 10 packages (that seems to stems from the fact that most packages are in conda-forge, and requirements are often inconsistent with the default repo).
nisegami 5y ago
Well, we have both Pipenv and Poetry now. It's a just a matter of people using them.
- bborud 5y ago
  You are kind of proving my point there.
cybernautique 5y ago
I highly recommend Poetry as a way to manage Python projects. It's my favorite bit of software I've had the pleasure of using, period; if any Poetry devs see this, kudos. However, with pip's native freeze and install -r abilities, I'm also curious what you mean by hunter gatherer style installs. I've not yet had a problem with installing Python modules.
m4rtink 5y ago
Is it really a problem of Python itself rather than project developers being too lazy to do proper package management?
- bborud 5y ago
  Yes. If you are going to get predictable, consistent practices for this you need a more or less official way of doing this or projects will just end up choosing one of several ways of doing this. Or worse: they will invent their own. And this is demonstrated in the thread: people start listing ways you can streamline installs and try to avoid version conflicts. And there are, as you can see, several. Some don't even anticipate what kinds of problems the user will encounter with versioning conflicts, old Python versions coming with the OS etc. What is a bit disheartening is that people actually thought they were providing helpful suggestions for solutions when what they did was only to prove my point. But they won't necessarily see it that way.
  - m4rtink 5y ago
    I'm not sure one can really end up with one true way of packaging per language - sooner or later someone will think "this is too complicated, I can do it better and simpler!" and you end up with another way of software distribution for that language (which is just as complicated as the previous one, once the author actually notices all the usecases). This is why I prefer using the target OS dependent package management - there is usually just a single one such official system, which is also language agnostic and much more robust than the various per language kludge.
    - bborud 5y ago
      It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)). Then you have languages that are "half way there", like Java. I suppose there are still people who make what I refer to as "splat style projects", where you unpack some horror-zip containing thousands of files and litter your surroundings with JAR files, config files and other detritus. But you can build everything into a single JAR file that contains all the dependencies and can be run without any other requirement than a sufficiently new Java runtime. And when shown how to do this, most people tend to adopt it as their default build product. Languages like Python don't have a well-established build product that is self-contained. This places an undue burden on those who package software for distribution. It forces them to involve themselves in concerns that should be contained within the project - not leak out onto everyone's floor and potentially cause accidents. Packagers have to make sure that "all arms and legs are inside the vehicle" for all possible permutations of system configurations. Which is easy when you have a statically linked binary. Less so when the software in question is more like a cranky, writhing toddler. What really made the problem visible for me was when I worked in an organization where suddenly the place was filled with "data scientists". Mostly math or statistics people. Or more precisely: when more non-engineers started writing code and proved to be not only unable, but unwilling, to learn sufficient software engineering to ensure other people than themselves could actually run the code they wrote. Which was kind of unfortunate because researchers who were unable to reproduce their own computations was a _regular_ occurrence. They simply couldn't figure out how to get their old code to run on their new computer, for instance. And it isn't because they are jerks. It is because they use tools to get work done. They are not as interested in the tools as the people who make those tools are. To them the rest is noise that wastes their time. This highlighted the fact that you shouldn't have to be a software engineer in order to produce programs that are easy to distribute - it has to be part of the path of least resistance. The fact that Python says "not my problem" isn't helpful. (I think there are interesting lessons to learn from Go. There are a lot of things I don't like about Go, but almost every instance where the language developers put their foot down and said "this is how we do it" actually made things better for everyone. It might not be your favorite way of doing things, but someone has made a choice - which is better than "I don't know...do whatever you like")
BiteCode_dev 5y ago
Use shiv: http://shiv.readthedocs.io http://shiv.readthedocs.io Installs are now an scp. Now of course, you gotta remember that installing something is usually more than copying the code, and there is nothing to help you with that, in any language. Welcome to the world of deployment, which is why things like ansible and docker exist.

I'm militantly disinterested in the performance of Python until the deployment style of most Python projects is no longer "hunter-gatherer-style installs".

by bborud 5y ago

I'm militantly disinterested in the performance of Python until the deployment style of most Python projects is no longer "hunter-gatherer-style installs".

aazaa 5y ago

What are "hunter-gatherer-style installs?"

taeric 5y ago
I take that as "keep running pip install on yet another discovered dep until the script works." And heaven help you if you accidentally get conflicting versions installed in the same venv. (By conflicting, I mean two libraries that both depend on different versions of the same underlying library. Odds of this increase dramatically the more you use.)
- deleted 5y ago
  [deleted]
- orf 5y ago
  Pip now checks this, but you usually use something like Poetry to give you a concrete lockfile with the correct transient dependencies locked in.
  - taeric 5y ago
    My "heaven help you" was more that, if you are in that situation, you don't really have an exit strategy in python. Do you?
- singularity2001 5y ago
  yeah people are often lazy about adding a requirements.txt
- joconde 5y ago
  Neither of these problems have happened to me since pip introduced their new dependency resolver (I think sometime during 2020). All dependencies are installed, and packages in the default repo usually have consistent dependency lists. It's another story for conda, which flat-out refused to install my list of around 10 packages (that seems to stems from the fact that most packages are in conda-forge, and requirements are often inconsistent with the default repo).

nisegami 5y ago

Well, we have both Pipenv and Poetry now. It's a just a matter of people using them.

bborud 5y ago
You are kind of proving my point there.

cybernautique 5y ago

I highly recommend Poetry as a way to manage Python projects. It's my favorite bit of software I've had the pleasure of using, period; if any Poetry devs see this, kudos. However, with pip's native freeze and install -r abilities, I'm also curious what you mean by hunter gatherer style installs. I've not yet had a problem with installing Python modules.

m4rtink 5y ago

Is it really a problem of Python itself rather than project developers being too lazy to do proper package management?

bborud 5y ago
Yes. If you are going to get predictable, consistent practices for this you need a more or less official way of doing this or projects will just end up choosing one of several ways of doing this. Or worse: they will invent their own. And this is demonstrated in the thread: people start listing ways you can streamline installs and try to avoid version conflicts. And there are, as you can see, several. Some don't even anticipate what kinds of problems the user will encounter with versioning conflicts, old Python versions coming with the OS etc. What is a bit disheartening is that people actually thought they were providing helpful suggestions for solutions when what they did was only to prove my point. But they won't necessarily see it that way.
- m4rtink 5y ago
  I'm not sure one can really end up with one true way of packaging per language - sooner or later someone will think "this is too complicated, I can do it better and simpler!" and you end up with another way of software distribution for that language (which is just as complicated as the previous one, once the author actually notices all the usecases). This is why I prefer using the target OS dependent package management - there is usually just a single one such official system, which is also language agnostic and much more robust than the various per language kludge.
  - bborud 5y ago
    It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)). Then you have languages that are "half way there", like Java. I suppose there are still people who make what I refer to as "splat style projects", where you unpack some horror-zip containing thousands of files and litter your surroundings with JAR files, config files and other detritus. But you can build everything into a single JAR file that contains all the dependencies and can be run without any other requirement than a sufficiently new Java runtime. And when shown how to do this, most people tend to adopt it as their default build product. Languages like Python don't have a well-established build product that is self-contained. This places an undue burden on those who package software for distribution. It forces them to involve themselves in concerns that should be contained within the project - not leak out onto everyone's floor and potentially cause accidents. Packagers have to make sure that "all arms and legs are inside the vehicle" for all possible permutations of system configurations. Which is easy when you have a statically linked binary. Less so when the software in question is more like a cranky, writhing toddler. What really made the problem visible for me was when I worked in an organization where suddenly the place was filled with "data scientists". Mostly math or statistics people. Or more precisely: when more non-engineers started writing code and proved to be not only unable, but unwilling, to learn sufficient software engineering to ensure other people than themselves could actually run the code they wrote. Which was kind of unfortunate because researchers who were unable to reproduce their own computations was a _regular_ occurrence. They simply couldn't figure out how to get their old code to run on their new computer, for instance. And it isn't because they are jerks. It is because they use tools to get work done. They are not as interested in the tools as the people who make those tools are. To them the rest is noise that wastes their time. This highlighted the fact that you shouldn't have to be a software engineer in order to produce programs that are easy to distribute - it has to be part of the path of least resistance. The fact that Python says "not my problem" isn't helpful. (I think there are interesting lessons to learn from Go. There are a lot of things I don't like about Go, but almost every instance where the language developers put their foot down and said "this is how we do it" actually made things better for everyone. It might not be your favorite way of doing things, but someone has made a choice - which is better than "I don't know...do whatever you like")

BiteCode_dev 5y ago

Use shiv: http://shiv.readthedocs.io http://shiv.readthedocs.io Installs are now an scp. Now of course, you gotta remember that installing something is usually more than copying the code, and there is nothing to help you with that, in any language. Welcome to the world of deployment, which is why things like ansible and docker exist.