5 ms·

Do we live in different worlds? Virtual environments solve 99% of all python packaging and installation use cases.

YetAnotherNick 3y ago
I guess so. Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. Try running any of the program which is >3 years old. I remember in one of the programs I needed to pin 10 dependency versions manually to make it run.
- biorach 3y ago
  > I needed to pin 10 dependency versions manually to make it run um... You should have been pinning all your dependencies from the very start
  - noitpmeder 3y ago
    Agreed, or if you dont, you should run pip-compile during CI/merge in order to pin every dependency in the entire tree. These are very solveable (and have been solved for a while) problems. It really pains me that most people are either unaware or do not dive deep enough below the surface to find them.
  - YetAnotherNick 3y ago
    It wasn't my code. I pin all my dependencies, but lot of python code has been written by people without software engineering experience like university students and to an extent ML engineers.
    - biorach 3y ago
      OK but your occasional contention was that > Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. This is a problem of lack of training or willful poor practice, not an issue with venvs themselves which absolutely do solve this issue. Agreed, the problem would not exist at all if an approach like that of say, node development, existed, but here we are.
  - baq 3y ago
    But only in what you deploy. For the love of God don’t pin versions in libraries unless you’re really sure it has a completely different api. That’s how you get a dependency graph with no solutions.
    - biorach 3y ago
      > But only in what you deploy You mean only your direct dependencies, right? And yes, of course
      - baq 3y ago
        I mean the whole pip freeze output, if you’re still using that (this works like a lock file).
- mrweasel 3y ago
  Dependency pinning is one of those things where I do see valid use cases, but "making it run" isn't one of them. It should primarily be use to deal with incompatible version until you can make the necessary changes. If you depend on dependency pinning due to unmaintained code then you should go deal with the problem directly. Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to? That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year.
  - jabradoodle 3y ago
    In this case your accidentally upgrading your dependencies only when rebuilt. Especially with micro services some things can be ran for years without being rebuilt.
  - biorach 3y ago
    > Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to? > That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year You seem to be suggesting not pinning and just pulling in the latest (minor, I presume?) versions every time you redeploy. A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis, especially for publically available services. A proper CI system should allow you to do this with confidence. Occasionally you will be forced into major version upgrades, but in my experience this is rare. This should be a regular and essential part of software maintenance and security vigilance.
    - mrweasel 3y ago
      > A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis I'd agree with that, I just don't believe that to be happening on any significant scale. Reasonably I think you should be able to do pinning like: library>=3.0,<4 Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages. Pinning is fine is you manage updates, but if you're then pulling in three year old libraries, which may come with their own pinned third party dependencies, I still think you're doing it wrong. In the example from GP it appears that they are pinning versions as a way to avoid forking and patching a deprecated and unmaintained package. I'm not oppose to version pinning as such, but if you don't have a plan to stay on top of security updates, then you're better off pulling in any minor update. It's not just at redeploy, you may reasonably need to pull dependency updates more frequently than you redeploy. If you deploy using Docker e.g. you're CI system needs to be constantly be pulling in updates, rebuilding and testing container images. People just don't do that. Most developers I worked with don't even care to update their container images if they aren't updating their own code. I've on multiple occasions had to deal with developers who absolutely lost their mind because we didn't automatically pull in OS updates, yet they themselves shipped outdated Java libraries or at one point even relied on an out date version of an alpha release of Tomcat that hadn't been updated for three years. The only different was that their dependencies where in containers and therefor, in their mind, "safe".

I guess so. Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. Try running any of the program which is >3 years old. I remember in one of the programs I needed to pin 10 dependency versions manually to make it run.

biorach 3y ago
> I needed to pin 10 dependency versions manually to make it run um... You should have been pinning all your dependencies from the very start
- noitpmeder 3y ago
  Agreed, or if you dont, you should run pip-compile during CI/merge in order to pin every dependency in the entire tree. These are very solveable (and have been solved for a while) problems. It really pains me that most people are either unaware or do not dive deep enough below the surface to find them.
- YetAnotherNick 3y ago
  It wasn't my code. I pin all my dependencies, but lot of python code has been written by people without software engineering experience like university students and to an extent ML engineers.
  - biorach 3y ago
    OK but your occasional contention was that > Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. This is a problem of lack of training or willful poor practice, not an issue with venvs themselves which absolutely do solve this issue. Agreed, the problem would not exist at all if an approach like that of say, node development, existed, but here we are.
- baq 3y ago
  But only in what you deploy. For the love of God don’t pin versions in libraries unless you’re really sure it has a completely different api. That’s how you get a dependency graph with no solutions.
  - biorach 3y ago
    > But only in what you deploy You mean only your direct dependencies, right? And yes, of course
    - baq 3y ago
      I mean the whole pip freeze output, if you’re still using that (this works like a lock file).
mrweasel 3y ago
Dependency pinning is one of those things where I do see valid use cases, but "making it run" isn't one of them. It should primarily be use to deal with incompatible version until you can make the necessary changes. If you depend on dependency pinning due to unmaintained code then you should go deal with the problem directly. Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to? That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year.
- jabradoodle 3y ago
  In this case your accidentally upgrading your dependencies only when rebuilt. Especially with micro services some things can be ran for years without being rebuilt.
- biorach 3y ago
  > Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to? > That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year You seem to be suggesting not pinning and just pulling in the latest (minor, I presume?) versions every time you redeploy. A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis, especially for publically available services. A proper CI system should allow you to do this with confidence. Occasionally you will be forced into major version upgrades, but in my experience this is rare. This should be a regular and essential part of software maintenance and security vigilance.
  - mrweasel 3y ago
    > A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis I'd agree with that, I just don't believe that to be happening on any significant scale. Reasonably I think you should be able to do pinning like: library>=3.0,<4 Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages. Pinning is fine is you manage updates, but if you're then pulling in three year old libraries, which may come with their own pinned third party dependencies, I still think you're doing it wrong. In the example from GP it appears that they are pinning versions as a way to avoid forking and patching a deprecated and unmaintained package. I'm not oppose to version pinning as such, but if you don't have a plan to stay on top of security updates, then you're better off pulling in any minor update. It's not just at redeploy, you may reasonably need to pull dependency updates more frequently than you redeploy. If you deploy using Docker e.g. you're CI system needs to be constantly be pulling in updates, rebuilding and testing container images. People just don't do that. Most developers I worked with don't even care to update their container images if they aren't updating their own code. I've on multiple occasions had to deal with developers who absolutely lost their mind because we didn't automatically pull in OS updates, yet they themselves shipped outdated Java libraries or at one point even relied on an out date version of an alpha release of Tomcat that hadn't been updated for three years. The only different was that their dependencies where in containers and therefor, in their mind, "safe".