4 ms·

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking

I do a lot of maintenance work and every time I've encountered a (complex) project set up in this way that's older than ~6 months, it's bitrotted from breaking changes in the dependencies. The python ecosystem does not stand still and seems quite happy to introduce breaking changes in non-major versions. To my mind, there are 3 ways to make sure your python project of today will work in 2 years time: 1. Have no dependencies 2. Destroy your virtual environment and reinstall it every day for the next 2 years and fix anything that breaks 3. Freeze your entire dependency tree where it stands and occasionally do large breaking change updates All 3 options suck but #3 is the least worst

turndown 2y ago
Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this
- lexicality 2y ago
  It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees: > I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade
- kelseyfrog 2y ago
  3 is exactly what you're supposed to do. A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies. The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.
  - LarsDu88 2y ago
    I mention this ceiling pinning footgun in the article. It's an enormous pain in the ass to explain to folks, and some software engineers I've met are totally incredulous that that's "not the right thing to do" Poetry makes it 10x worse with its `^` operator
    - kelseyfrog 2y ago
      Definitely! And the article you link to in this section is great too
- daemonologist 2y ago
  It is, but Python software tends towards large dependency graphs which will quickly accumulate CVEs, and so this strategy greatly upsets your security/audit people. (Obviously patching vulnerabilities is good and proper, but automatically flagged CVEs are only ever _potential_ at best; in many contexts most of them are not actual vulnerabilities.)
- alexey-salmin 2y ago
  Yes. But then any transitive dependency stays there forever even when not needed anymore. Initial requirements (only first level, version ranges) and dependency resolution results (all transitive packages, exact versions or hashes) are two very different things and should be treated separately. You can implement and maintain it by hand with two requirements.txt files but it's rarely done this way. And really at this level you're better off with a normal package manager.
GenerocUsername 2y ago
I work in JS and the issues are the same. It's why NPM creates lockfiles that can freeze the version of direct and indirect deps.
kapilvt 2y ago
fwiw, we built an oss plugin for poetry on option #3 when publishing for repeatable installs to automate translation of published wheel metadata from lock file, also supports mono repos. https://github.com/cloud-custodian/poetry-plugin-freeze https://github.com/cloud-custodian/poetry-plugin-freeze
deleted 2y ago
[deleted]
yjftsjthsd-h 2y ago
IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv could manage it like everything else). And also as you note there's just ecosystem churn.
- eternityforest 2y ago
  If we could reboot all of computer science, I'd want one unified cross platform package manager with the same market takeover level as Git, and all languages designed to be aware of it. No file system overlay stacking or anything, no file system hierarchy of different folders for different types of resources, just python style search paths and virtual environments, and no global environment allowed, ever. But Python development is pretty nice at the moment so I can't complain.
JackSlateur 2y ago
What sucks is when you imply that your project will be unmaintained for 2 years, without any kind of update (not even basic security upgrades). This is a not a project, this is abandonware from day 1.
- rusticpenn 2y ago
  There was a time when you wrote a software for a version of the OS and it was done (except security fixes).
- lexicality 2y ago
  Obviously my profession gives me a ton of bias but in general it seems that for most people the squeaky wheel gets the grease and once a project is working it is ignored until it stops working again because there are only so many hours in the day and management has an infinite list of demands... And then it breaks 2 years later and I'm called in because the person who wrote it left and no one else knows how it works because they didn't write any documentation etc etc etc
- theshrike79 2y ago
  It's not like that simple 50 line Python script taking stuff from API A and stuffing them to API B is a full ass project with a JIRA project ID and a Product Manager. It works, people forget it exists. Then something breaks and I need to start fixing it. And the amount of work it takes to just update the packages is more than it'd take me to rewrite the whole thing in a language with a sane distribution mechanism (Go).

turndown 2y ago

Is option 3 exactly what you’re supposed to do? Freezing your dependency graph and/or explicitly denoting what version of the dependency you want are your best bets for avoiding problems like this

lexicality 2y ago
It's certainly what I'd go for and what I recommend to everyone, but the GP disagrees: > I never mention transitive dependencies in my requirements.txt file, just direct dependencies and rely on pip to install all transitive libs. > You dont even have to freeze the version, just list the name and pull up latest version whenever you run pip upgrade
kelseyfrog 2y ago
3 is exactly what you're supposed to do. A lot of people will assume that specifying major version upper bounds on dependencies is what you're supposed to do, but I've seen this fail more often than freezing dependencies. The problem with major version upper bounds is that if it's possible to write a test case for a bug, it's possible to depend on broken behavior. Changing behavior in a way that breaks users should be a major version bump, but that's not actually how people use semver and semver isn't really described that way either. It's described in a way that makes people think that changes in type signatures are the predominant impetus to bump major versions.
- LarsDu88 2y ago
  I mention this ceiling pinning footgun in the article. It's an enormous pain in the ass to explain to folks, and some software engineers I've met are totally incredulous that that's "not the right thing to do" Poetry makes it 10x worse with its `^` operator
  - kelseyfrog 2y ago
    Definitely! And the article you link to in this section is great too
daemonologist 2y ago
It is, but Python software tends towards large dependency graphs which will quickly accumulate CVEs, and so this strategy greatly upsets your security/audit people. (Obviously patching vulnerabilities is good and proper, but automatically flagged CVEs are only ever _potential_ at best; in many contexts most of them are not actual vulnerabilities.)
alexey-salmin 2y ago
Yes. But then any transitive dependency stays there forever even when not needed anymore. Initial requirements (only first level, version ranges) and dependency resolution results (all transitive packages, exact versions or hashes) are two very different things and should be treated separately. You can implement and maintain it by hand with two requirements.txt files but it's rarely done this way. And really at this level you're better off with a normal package manager.

GenerocUsername 2y ago

I work in JS and the issues are the same. It's why NPM creates lockfiles that can freeze the version of direct and indirect deps.

kapilvt 2y ago

fwiw, we built an oss plugin for poetry on option #3 when publishing for repeatable installs to automate translation of published wheel metadata from lock file, also supports mono repos. https://github.com/cloud-custodian/poetry-plugin-freeze https://github.com/cloud-custodian/poetry-plugin-freeze

deleted 2y ago

[deleted]

yjftsjthsd-h 2y ago

IMHO the problem with python package management isn't python package management. The biggest problem I've hit is when python depends on non-python - `pip install` failing because I need to `apt install libsomething-dev` is the big one, but also managing multiple versions of python itself (I count this as a dependency on non-python because the python3 binary isn't written in python - if it was, then pip+virtualenv could manage it like everything else). And also as you note there's just ecosystem churn.

eternityforest 2y ago
If we could reboot all of computer science, I'd want one unified cross platform package manager with the same market takeover level as Git, and all languages designed to be aware of it. No file system overlay stacking or anything, no file system hierarchy of different folders for different types of resources, just python style search paths and virtual environments, and no global environment allowed, ever. But Python development is pretty nice at the moment so I can't complain.

JackSlateur 2y ago

What sucks is when you imply that your project will be unmaintained for 2 years, without any kind of update (not even basic security upgrades). This is a not a project, this is abandonware from day 1.

rusticpenn 2y ago
There was a time when you wrote a software for a version of the OS and it was done (except security fixes).
lexicality 2y ago
Obviously my profession gives me a ton of bias but in general it seems that for most people the squeaky wheel gets the grease and once a project is working it is ignored until it stops working again because there are only so many hours in the day and management has an infinite list of demands... And then it breaks 2 years later and I'm called in because the person who wrote it left and no one else knows how it works because they didn't write any documentation etc etc etc
theshrike79 2y ago
It's not like that simple 50 line Python script taking stuff from API A and stuffing them to API B is a full ass project with a JIRA project ID and a Product Manager. It works, people forget it exists. Then something breaks and I need to start fixing it. And the amount of work it takes to just update the packages is more than it'd take me to rewrite the whole thing in a language with a sane distribution mechanism (Go).