3 ms·

Oh I have 100% run into dependency hell many times in the Python ecosystem! I am well aware. But the overall force multiplication of that ecosystem has far out

by szvsw 2y ago

Oh I have 100% run into dependency hell many times in the Python ecosystem! I am well aware. But the overall force multiplication of that ecosystem has far outweighed the cumulative day or two per year spent in dependency hell. As someone who works in both academia (currently a PhD student!) and industry (at the aforementioned startup), it just seems obvious to me that the situation you are describing is less of an innate issue with incorporating dependencies and more an issue with the lack of training on environment management / containerization and lack of experience with software development in general that you see in academic projects. In a relatively competent professional shop, these sorts of issues in my experience crop up occasionally but there are so many mechanisms available to (a) mitigate them from appearing in the first place and (b) have rough plans or strategies in place ahead of time if they do crop up and (c) continuously address them when they do appear, which prevents it from ever blowing up into a true dependency hell. It seems like being judicious about when you decide to use dependencies and how you manage them is sufficient, as opposed to actively avoiding them altogether... Anyways to me it definitely is the casethat Python has loads of footguns/risks in re: dependencies when someone is developing without thinking about dependency management/long term planning, which is usually the case with academic projects, but I don't think these dangers are so bad when the user is actively aware of them and plans for them - or not bad enough to make the risks outweigh the rewards of adopting the dependencies... it definitely means that the ecosystem is elevating risk levels, but I would also think that there are organizational/management issues which contribute significantly to (and compound with) the issues you've described in an academic context. > My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. That seems reasonable; it's problematic from the perspective of the replication crisis in the sciences, but from a different perspective - if the code/project has no users or active developers for 3 years, at what point does it just become cruft? Is it okay to let code die? I don't love this view, but my PI strongly feels that no matter how excellently or poorly engineered a GitHub repository is, it doesn't matter from the perspective of sciences - the only thing that matters is if (a) the methodology is explained thoroughly enough in the literature you produce (in which case the repo can be trash and unrunnable in a few years - if the method is useful enough it will be copied and expanded on etc by others naturally or just completely independently, and 3-5 year old methods are out of date anyways) or (b) it goes on to be actively used by you, your peers, or people in industry, in which case the literature does not matter (but by definition the repo is not trash as it is in continuous use/active development/evolution). I think their are some flaws in this view but I also see merit in it. Anyways, having said all of this, I hate it when one of my peers at the startup I work with wants to pull in some dependency for a relatively straightforward search algorithm they don't understand when I think it would be easier to just implement it ourselves, so I'm contradicting myself :)

crabbone 2y ago
> if the code/project has no users or active developers for 3 years, at what point does it just become cruft? At the time the hardware cannot possibly support it. So, we are talking about something like 50 years. For more high-level languages the deadline should extend further (i.e. languages like C should, in principle, have shorter shelf life because they have stronger dependency on the hardware they run on). This is how, ideally it should work. I.e. there's no reason any Python code should be dead by now: Python simply hadn't been around long enough for that to become a real, impossible to solve issue. Ironically, C programs have far better shelf life than Python. About an order of magnitude better. Second deadline is when businesses will stop using project's code if it doesn't have enough shelf life. This is somewhere between 10 and 5 years. Government / military will want the longer shelf life (and usually have enough bargaining power to compel the developers), while the rest of the industry will suck it up and accept the inevitable death of their software. But, at some point, it no longer makes sense to sacrifice the profits just to keep up with the trends. Eg. you can still find businesses that haven't upgraded from CentOS 7 (released in 2014), and are doing OK. A lot of military software uses 90s or even 80s tech. Banking is also a special case here, as well as some embedded systems that can be many decades old eg. railway networks. I wish that anything below that threshold wasn't commercially viable... but, obviously, a lot of businesses find enough resources to upgrade more frequently. I feel like these businesses make a disservice to the broader community of users depending on the same technology. In some ways, their ability to upgrade more often is also a way to beat the competition. Unfortunately, more often, it's not just the competition that suffers from this, but also businesses in completely unrelated fields. ---- Now, if talking about Python specifically, there's quite a bit of difference based on particular area the code is written for. Web and data science are the areas with the worst shelf life, and require the most effort to simply stand still. These are the areas where projects typically have more dependencies, attract a lot of attention (and so there's more drive to iterate faster on those dependencies). But, on Python's side, there are plenty of bad things happening regardless of what individual teams can do. For example, when talking about support of software like operating systems, this includes support of a large number of individual (optional) components that need to work together. I.e. the LTS Linux distribution will have a set of packages that are (allegedly) tested to work together, and if you want to have a reliable baseline for your own software, you just have to draw from a particular distribution and you should be fine. PyPI, or more broadly, Python community doesn't have that kind of concept. There's no set of dependencies that can be guaranteed for any length of time at any time. Anaconda tried to be more like that, but they were overpowered by the users' desire to get new things out of bound. Another problem with how Python versions packages is that it imposes virtually no restrictions on publishing. If a package author wants to upgrade ten major package versions in a day, there's nobody to tell it to stop. Want to publish an older version than the last one published? -- sure, go ahead. And, of course, on top of it, interface breakage in versions where that's not supposed to happen etc. The poor stewardship of PyPI is a self-fulfilling prophecy: package maintainers realize that the versioning is, essentially, broken in Python, and in order to prevent their packages from breaking, they narrow down the range of acceptable dependency versions as much as possible (and often only accept the single version of the dependency they tested with). This, in turn, results in a situation where upgrades are impossible without massive breakage: incremental upgrades become impossible, because no part of the dependency tree can be upgraded independently. On the other hand, if a single dependency in this tree goes stale, the whole tree is stale. Another factor to this is that dependencies which accept a wider range of transitive dependencies are harder to install using currently available package managers / installers. I.e. because pip / conda will spend a lot more time trying to find a solution for dependency graph with more potential candidates (due to more loose version requirements), developers are prone to try and expedite the process by narrowing down version constrains. This is especially painful in conda world, where the solver can take many hours (or days in pathological cases) to solve such installations. And so developers have to choose between being more friendly to the user (and supporting multiple transitive dependency versions) or being more hostile to the user (by narrowing down supported versions) while speeding up their deployment. Yet another factor is the design of Python module system that is unaware of module versions. I.e. if only it was possible to specify the version of the imported module at import time, a lot of problems would've gone away. There would've been no need for virtual environments. Subsequently, there'd be more incentive to ensure compatibility of various independent libraries and programs, which would put pressure on package maintainers to dedicate more time to integration testing and slow down the release cycle.

by szvsw 2y ago

crabbone 2y ago

> if the code/project has no users or active developers for 3 years, at what point does it just become cruft? At the time the hardware cannot possibly support it. So, we are talking about something like 50 years. For more high-level languages the deadline should extend further (i.e. languages like C should, in principle, have shorter shelf life because they have stronger dependency on the hardware they run on). This is how, ideally it should work. I.e. there's no reason any Python code should be dead by now: Python simply hadn't been around long enough for that to become a real, impossible to solve issue. Ironically, C programs have far better shelf life than Python. About an order of magnitude better. Second deadline is when businesses will stop using project's code if it doesn't have enough shelf life. This is somewhere between 10 and 5 years. Government / military will want the longer shelf life (and usually have enough bargaining power to compel the developers), while the rest of the industry will suck it up and accept the inevitable death of their software. But, at some point, it no longer makes sense to sacrifice the profits just to keep up with the trends. Eg. you can still find businesses that haven't upgraded from CentOS 7 (released in 2014), and are doing OK. A lot of military software uses 90s or even 80s tech. Banking is also a special case here, as well as some embedded systems that can be many decades old eg. railway networks. I wish that anything below that threshold wasn't commercially viable... but, obviously, a lot of businesses find enough resources to upgrade more frequently. I feel like these businesses make a disservice to the broader community of users depending on the same technology. In some ways, their ability to upgrade more often is also a way to beat the competition. Unfortunately, more often, it's not just the competition that suffers from this, but also businesses in completely unrelated fields. ---- Now, if talking about Python specifically, there's quite a bit of difference based on particular area the code is written for. Web and data science are the areas with the worst shelf life, and require the most effort to simply stand still. These are the areas where projects typically have more dependencies, attract a lot of attention (and so there's more drive to iterate faster on those dependencies). But, on Python's side, there are plenty of bad things happening regardless of what individual teams can do. For example, when talking about support of software like operating systems, this includes support of a large number of individual (optional) components that need to work together. I.e. the LTS Linux distribution will have a set of packages that are (allegedly) tested to work together, and if you want to have a reliable baseline for your own software, you just have to draw from a particular distribution and you should be fine. PyPI, or more broadly, Python community doesn't have that kind of concept. There's no set of dependencies that can be guaranteed for any length of time at any time. Anaconda tried to be more like that, but they were overpowered by the users' desire to get new things out of bound. Another problem with how Python versions packages is that it imposes virtually no restrictions on publishing. If a package author wants to upgrade ten major package versions in a day, there's nobody to tell it to stop. Want to publish an older version than the last one published? -- sure, go ahead. And, of course, on top of it, interface breakage in versions where that's not supposed to happen etc. The poor stewardship of PyPI is a self-fulfilling prophecy: package maintainers realize that the versioning is, essentially, broken in Python, and in order to prevent their packages from breaking, they narrow down the range of acceptable dependency versions as much as possible (and often only accept the single version of the dependency they tested with). This, in turn, results in a situation where upgrades are impossible without massive breakage: incremental upgrades become impossible, because no part of the dependency tree can be upgraded independently. On the other hand, if a single dependency in this tree goes stale, the whole tree is stale. Another factor to this is that dependencies which accept a wider range of transitive dependencies are harder to install using currently available package managers / installers. I.e. because pip / conda will spend a lot more time trying to find a solution for dependency graph with more potential candidates (due to more loose version requirements), developers are prone to try and expedite the process by narrowing down version constrains. This is especially painful in conda world, where the solver can take many hours (or days in pathological cases) to solve such installations. And so developers have to choose between being more friendly to the user (and supporting multiple transitive dependency versions) or being more hostile to the user (by narrowing down supported versions) while speeding up their deployment. Yet another factor is the design of Python module system that is unaware of module versions. I.e. if only it was possible to specify the version of the imported module at import time, a lot of problems would've gone away. There would've been no need for virtual environments. Subsequently, there'd be more incentive to ensure compatibility of various independent libraries and programs, which would put pressure on package maintainers to dedicate more time to integration testing and slow down the release cycle.