3 ms·

> I also find it difficult to rationalize third-party dependencies these days, they almost all become disappointments over the long-term. The benefits rarely ju

by szvsw 2y ago

> I also find it difficult to rationalize third-party dependencies these days, they almost all become disappointments over the long-term. The benefits rarely justify the risks. What kind of software development do you do? As someone who is in ML/data science and works primarily in python, occasionally TS/React, at a small company (about 5 FTE, one round of funding), that sentence is so completely foreign to me. Again, this might just be because I am working primarily on ML systems in Python on relatively small teams (also in academia at the same time), where the most basic building blocks like Pandas (or Polars), PyTorch, XGBoost, SKLearn, etc are so essential and so it’s pretty hard to imagine ever being at a scale where they would be feasible or worth it to replace with our own code… and anyways if support for libraries like those collapses, the only explanation is that the entire landscape has shifted so much that it’s hard to imagine the underlying tech of the company not also having naturally shifted with it… Maybe I am misunderstanding how you are using “3rd party,” or maybe you are just operating in a completely different staffing and financial context which I don’t have experience with. Just curious for more details about what kind of world you operate in where that philosophy is viable!

all2 2y ago
I do SW in a hardware world that restricts a lot of dependencies. We vendor some, but mostly roll or own libraries. For time intensive things peripheral to our work we will rely on 3rd party modules (ie, network stack or UI).
crabbone 2y ago
You chose the field with one of the worst rates of churn and brokeness of third-party libraries, and by some kind of magic managed to avoid it? You should be buying lottery tickets by a truckload! All the projects you listed (Pandas, PyTorch, SKLearn) have a huge list of compatibility issues, on top of compatibility issues of Python itself. And I know this because I have to support various research projects, which typically use this stuff. My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. My typical situation with these projects is that I'm called in because some former PhD student, who graduated a year or two ago left a project that now another student is picking up. And nothing works. Results cannot be reproduced. The dependencies cannot be installed. And so on. And then, depending of the amount of effort I'm allowed to put into fixing the project, I end up either crafting some sort of a spec that tries to find a set of versions of used libraries that seem to make the project work again. Or, more commonly, have to vendor some code and add fixes on top to make the packages fit together at least somehow. Sometimes I even end up repackaging someone else's code (a dependency), where the version used before cannot be found anymore, or I cannot figure out the combination of version that could possibly make the program work. I hear that it's worse in JavaScript world. And, my experience with LaTeX packages has also been marred by similar problems (but not anywhere near the extent of how bad it is in Python). On the other hand, I had to build absolutely atrocious (math operations on a mix of signed and unsigned ints, modifying const char etc.) C code (pre-K&R, the kind where they write function argument types after parenthesis), and it was much easier to deal with than any "modern" Python (i.e. 3-4 years old). If anything, Python is the poster child of the problem of unreliability of third-party dependencies.
- szvsw 2y ago
  Oh I have 100% run into dependency hell many times in the Python ecosystem! I am well aware. But the overall force multiplication of that ecosystem has far outweighed the cumulative day or two per year spent in dependency hell. As someone who works in both academia (currently a PhD student!) and industry (at the aforementioned startup), it just seems obvious to me that the situation you are describing is less of an innate issue with incorporating dependencies and more an issue with the lack of training on environment management / containerization and lack of experience with software development in general that you see in academic projects. In a relatively competent professional shop, these sorts of issues in my experience crop up occasionally but there are so many mechanisms available to (a) mitigate them from appearing in the first place and (b) have rough plans or strategies in place ahead of time if they do crop up and (c) continuously address them when they do appear, which prevents it from ever blowing up into a true dependency hell. It seems like being judicious about when you decide to use dependencies and how you manage them is sufficient, as opposed to actively avoiding them altogether... Anyways to me it definitely is the casethat Python has loads of footguns/risks in re: dependencies when someone is developing without thinking about dependency management/long term planning, which is usually the case with academic projects, but I don't think these dangers are so bad when the user is actively aware of them and plans for them - or not bad enough to make the risks outweigh the rewards of adopting the dependencies... it definitely means that the ecosystem is elevating risk levels, but I would also think that there are organizational/management issues which contribute significantly to (and compound with) the issues you've described in an academic context. > My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. That seems reasonable; it's problematic from the perspective of the replication crisis in the sciences, but from a different perspective - if the code/project has no users or active developers for 3 years, at what point does it just become cruft? Is it okay to let code die? I don't love this view, but my PI strongly feels that no matter how excellently or poorly engineered a GitHub repository is, it doesn't matter from the perspective of sciences - the only thing that matters is if (a) the methodology is explained thoroughly enough in the literature you produce (in which case the repo can be trash and unrunnable in a few years - if the method is useful enough it will be copied and expanded on etc by others naturally or just completely independently, and 3-5 year old methods are out of date anyways) or (b) it goes on to be actively used by you, your peers, or people in industry, in which case the literature does not matter (but by definition the repo is not trash as it is in continuous use/active development/evolution). I think their are some flaws in this view but I also see merit in it. Anyways, having said all of this, I hate it when one of my peers at the startup I work with wants to pull in some dependency for a relatively straightforward search algorithm they don't understand when I think it would be easier to just implement it ourselves, so I'm contradicting myself :)
jvans 2y ago
It would obviously be crazy to roll your own numpy or lightgbm. But there's plenty of bloat in the library space where the cost of integration and upkeep is far more expensive than a simpler implementation of your own
jandrewrogers 2y ago
I primarily work on high-performance analytics data infrastructure, mostly written in C++ with a bit of Python and Rust at the periphery. Classic performance-engineered systems software. The only real dependencies are the compiler tool chain and Linux. The biggest issue with most popular libraries is some combination of poor performance, poor scalability on large hardware, and/or a software architecture that is not well-suited to performance-engineered systems. While there is a complaint that external libraries tend to significantly increase build times, that is not the motivating factor here. I've lost count of the number of times runtime behavior and performance issues have been traced back to (for us) far-from-optimal design choices in third-party libraries that probably seemed like reasonable choices to the library authors. Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance. In an age when hardware can deliver >100 GB/s of throughput, many popular software libraries deliver <1 GB/s per core even though far better can be achieved with a modicum of diligence. Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations. There is also a real issue with supply chain risk in terms of both quality and security. The vast majority of open source libraries are implemented to a lower standard of quality and security than the rest of the code base. And those libraries are often not really designed to be testable to the extent that would give similar confidence in the implementation quality. Combine this with the reality that the software needs to be deployable in environments that require higher than average assurance makes this a business risk. As for the manpower cost, it has been low in practice. An engineer-month here, a few engineer-months there, with the benefit of better performance, more optionality, and much easier maintainability. In most cases, we would only use a tiny subset of library functionality regardless, so it adds a large surface area for a narrow benefit. Much of this code is also reusable. The existing open source libraries are very useful here because they provide something to measure against. I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers. I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.

by szvsw 2y ago

all2 2y ago

I do SW in a hardware world that restricts a lot of dependencies. We vendor some, but mostly roll or own libraries. For time intensive things peripheral to our work we will rely on 3rd party modules (ie, network stack or UI).

crabbone 2y ago

You chose the field with one of the worst rates of churn and brokeness of third-party libraries, and by some kind of magic managed to avoid it? You should be buying lottery tickets by a truckload! All the projects you listed (Pandas, PyTorch, SKLearn) have a huge list of compatibility issues, on top of compatibility issues of Python itself. And I know this because I have to support various research projects, which typically use this stuff. My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. My typical situation with these projects is that I'm called in because some former PhD student, who graduated a year or two ago left a project that now another student is picking up. And nothing works. Results cannot be reproduced. The dependencies cannot be installed. And so on. And then, depending of the amount of effort I'm allowed to put into fixing the project, I end up either crafting some sort of a spec that tries to find a set of versions of used libraries that seem to make the project work again. Or, more commonly, have to vendor some code and add fixes on top to make the packages fit together at least somehow. Sometimes I even end up repackaging someone else's code (a dependency), where the version used before cannot be found anymore, or I cannot figure out the combination of version that could possibly make the program work. I hear that it's worse in JavaScript world. And, my experience with LaTeX packages has also been marred by similar problems (but not anywhere near the extent of how bad it is in Python). On the other hand, I had to build absolutely atrocious (math operations on a mix of signed and unsigned ints, modifying const char etc.) C code (pre-K&R, the kind where they write function argument types after parenthesis), and it was much easier to deal with than any "modern" Python (i.e. 3-4 years old). If anything, Python is the poster child of the problem of unreliability of third-party dependencies.

szvsw 2y ago
Oh I have 100% run into dependency hell many times in the Python ecosystem! I am well aware. But the overall force multiplication of that ecosystem has far outweighed the cumulative day or two per year spent in dependency hell. As someone who works in both academia (currently a PhD student!) and industry (at the aforementioned startup), it just seems obvious to me that the situation you are describing is less of an innate issue with incorporating dependencies and more an issue with the lack of training on environment management / containerization and lack of experience with software development in general that you see in academic projects. In a relatively competent professional shop, these sorts of issues in my experience crop up occasionally but there are so many mechanisms available to (a) mitigate them from appearing in the first place and (b) have rough plans or strategies in place ahead of time if they do crop up and (c) continuously address them when they do appear, which prevents it from ever blowing up into a true dependency hell. It seems like being judicious about when you decide to use dependencies and how you manage them is sufficient, as opposed to actively avoiding them altogether... Anyways to me it definitely is the casethat Python has loads of footguns/risks in re: dependencies when someone is developing without thinking about dependency management/long term planning, which is usually the case with academic projects, but I don't think these dangers are so bad when the user is actively aware of them and plans for them - or not bad enough to make the risks outweigh the rewards of adopting the dependencies... it definitely means that the ecosystem is elevating risk levels, but I would also think that there are organizational/management issues which contribute significantly to (and compound with) the issues you've described in an academic context. > My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. That seems reasonable; it's problematic from the perspective of the replication crisis in the sciences, but from a different perspective - if the code/project has no users or active developers for 3 years, at what point does it just become cruft? Is it okay to let code die? I don't love this view, but my PI strongly feels that no matter how excellently or poorly engineered a GitHub repository is, it doesn't matter from the perspective of sciences - the only thing that matters is if (a) the methodology is explained thoroughly enough in the literature you produce (in which case the repo can be trash and unrunnable in a few years - if the method is useful enough it will be copied and expanded on etc by others naturally or just completely independently, and 3-5 year old methods are out of date anyways) or (b) it goes on to be actively used by you, your peers, or people in industry, in which case the literature does not matter (but by definition the repo is not trash as it is in continuous use/active development/evolution). I think their are some flaws in this view but I also see merit in it. Anyways, having said all of this, I hate it when one of my peers at the startup I work with wants to pull in some dependency for a relatively straightforward search algorithm they don't understand when I think it would be easier to just implement it ourselves, so I'm contradicting myself :)

jvans 2y ago

It would obviously be crazy to roll your own numpy or lightgbm. But there's plenty of bloat in the library space where the cost of integration and upkeep is far more expensive than a simpler implementation of your own

jandrewrogers 2y ago

I primarily work on high-performance analytics data infrastructure, mostly written in C++ with a bit of Python and Rust at the periphery. Classic performance-engineered systems software. The only real dependencies are the compiler tool chain and Linux. The biggest issue with most popular libraries is some combination of poor performance, poor scalability on large hardware, and/or a software architecture that is not well-suited to performance-engineered systems. While there is a complaint that external libraries tend to significantly increase build times, that is not the motivating factor here. I've lost count of the number of times runtime behavior and performance issues have been traced back to (for us) far-from-optimal design choices in third-party libraries that probably seemed like reasonable choices to the library authors. Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance. In an age when hardware can deliver >100 GB/s of throughput, many popular software libraries deliver <1 GB/s per core even though far better can be achieved with a modicum of diligence. Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations. There is also a real issue with supply chain risk in terms of both quality and security. The vast majority of open source libraries are implemented to a lower standard of quality and security than the rest of the code base. And those libraries are often not really designed to be testable to the extent that would give similar confidence in the implementation quality. Combine this with the reality that the software needs to be deployable in environments that require higher than average assurance makes this a business risk. As for the manpower cost, it has been low in practice. An engineer-month here, a few engineer-months there, with the benefit of better performance, more optionality, and much easier maintainability. In most cases, we would only use a tiny subset of library functionality regardless, so it adds a large surface area for a narrow benefit. Much of this code is also reusable. The existing open source libraries are very useful here because they provide something to measure against. I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers. I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.