3 ms·

I wonder how much dependencies could be reduced by systematically searching low-hanging fruit and addressing it ad hoc. For example, if commonly-used library A

by xamuel 2y ago

I wonder how much dependencies could be reduced by systematically searching low-hanging fruit and addressing it ad hoc. For example, if commonly-used library A uses one minor thing from (and thus imports all of) library B, which in turn imports hundreds of other libraries, then someone should add the minor thing in question to A and remove the dependency on B there. It's interesting to think of how this sort of "neighborhood watch" could be incentivized, since it's probably way too big of a task for purely volunteer work. It's tricky though because any incentive to remove dependencies would automatically be a perverse incentive to ADD dependencies (so that you can later remove them and get the credit for it).

remram 2y ago
Then the code for library B still exists, still potentially has bugs, the only difference is that the same bug has to be fixed by project A1 then again by project A2 and project A3 etc. There is a cost there too, outlined in the recent article 'Tech Debt: My Rust Library Is Now a CDO' https://news.ycombinator.com/item?id=39827645 https://news.ycombinator.com/item?id=39827645
- abathur 2y ago
  I guess there's a hybrid model where you're able to select exactly what you're depending on and pull it in dynamically at build/package time. I've thought a little about, for example, building something that could slice just the needed utility functions out of a Shell utility library. (Not really for minimizing the dependency graph--just for reducing the source-time overhead of parsing a large utility library that you only want a few functions from.) Would obviously need a lot of toolchain work to really operationalize broadly. I can at least imagine the first few steps of how I might be able to build a Nix expression that, say, depends on the source of some other library and runs a few tools to find and extract a specific function and the other (manually-identified) bits of source necessary to build a ~library with just the one function, and then let the primary project depend on that. It smells like a fair bit of work, but not so much that I wouldn't try doing it if the complexity/stability of the dependency graph was causing me trouble?
  - remram 2y ago
    Some package managers have "features" (e.g. Rust's cargo) or "extras" (e.g. Python) which might be what you are talking about. Of course another solution is to make smaller libraries in the first place, so your users don't feel like they need to break it up.
    - abathur 2y ago
      > Some package managers have "features" (e.g. Rust's cargo) or "extras" (e.g. Python) which might be what you are talking about. I don't think so (though I agree that mechanisms like this are one way to approach the problem). AFAIK both of these examples are mostly used to provide ~optional behavior (usually to exclude dependencies if you don't need the behavior). This can minimize the set of dependencies, but it's resting on the maintainers' sense of what the core of their library is, and what's ancillary. Said the other way around, both require the software's maintainers to anticipate your use case and feel like it was a good use of their time to split things up very granularly. In xamuel's hypothetical of library A using one minor thing from library B, this almost certainly means reusing less of library B than its maintainers anticipate. I can imagine this working in cases where the package is a true bundle of discrete utilities that almost no one will need all of (the package itself is an incredibly small core/stub and each utility is a feature/extra), and the maintainers want to intentionally design it for modular consumption. But it's hard to imagine many maintainers going through the work of dicing a cohesive library up into granular units when they think most users will be consuming it whole?
  - WorldMaker 2y ago
    Isn't that already just the role of tree-shaking optimizers? At that point the problem seems to be languages that don't have good tree-shakers, don't/can't tree-shake library dependencies, or maybe that tree-shaking should happen earlier and more often than it often does? Observably, it seems like the "granularity pendulum" in the JS ecosystem very directly related to the module system. CommonJS was tough to tree-shake so you had sometimes wild levels of granularity where even individual functions might be their own package in the dependency graph. ESM is a lot easier to tree-shake and you start to see more of the libraries that once published dozens or hundreds of sub-packages start to repackage back to just one top-level package alongside ESM adoption.

by xamuel 2y ago

remram 2y ago

Then the code for library B still exists, still potentially has bugs, the only difference is that the same bug has to be fixed by project A1 then again by project A2 and project A3 etc. There is a cost there too, outlined in the recent article 'Tech Debt: My Rust Library Is Now a CDO' https://news.ycombinator.com/item?id=39827645 https://news.ycombinator.com/item?id=39827645

abathur 2y ago
I guess there's a hybrid model where you're able to select exactly what you're depending on and pull it in dynamically at build/package time. I've thought a little about, for example, building something that could slice just the needed utility functions out of a Shell utility library. (Not really for minimizing the dependency graph--just for reducing the source-time overhead of parsing a large utility library that you only want a few functions from.) Would obviously need a lot of toolchain work to really operationalize broadly. I can at least imagine the first few steps of how I might be able to build a Nix expression that, say, depends on the source of some other library and runs a few tools to find and extract a specific function and the other (manually-identified) bits of source necessary to build a ~library with just the one function, and then let the primary project depend on that. It smells like a fair bit of work, but not so much that I wouldn't try doing it if the complexity/stability of the dependency graph was causing me trouble?
- remram 2y ago
  Some package managers have "features" (e.g. Rust's cargo) or "extras" (e.g. Python) which might be what you are talking about. Of course another solution is to make smaller libraries in the first place, so your users don't feel like they need to break it up.
  - abathur 2y ago
    > Some package managers have "features" (e.g. Rust's cargo) or "extras" (e.g. Python) which might be what you are talking about. I don't think so (though I agree that mechanisms like this are one way to approach the problem). AFAIK both of these examples are mostly used to provide ~optional behavior (usually to exclude dependencies if you don't need the behavior). This can minimize the set of dependencies, but it's resting on the maintainers' sense of what the core of their library is, and what's ancillary. Said the other way around, both require the software's maintainers to anticipate your use case and feel like it was a good use of their time to split things up very granularly. In xamuel's hypothetical of library A using one minor thing from library B, this almost certainly means reusing less of library B than its maintainers anticipate. I can imagine this working in cases where the package is a true bundle of discrete utilities that almost no one will need all of (the package itself is an incredibly small core/stub and each utility is a feature/extra), and the maintainers want to intentionally design it for modular consumption. But it's hard to imagine many maintainers going through the work of dicing a cohesive library up into granular units when they think most users will be consuming it whole?
- WorldMaker 2y ago
  Isn't that already just the role of tree-shaking optimizers? At that point the problem seems to be languages that don't have good tree-shakers, don't/can't tree-shake library dependencies, or maybe that tree-shaking should happen earlier and more often than it often does? Observably, it seems like the "granularity pendulum" in the JS ecosystem very directly related to the module system. CommonJS was tough to tree-shake so you had sometimes wild levels of granularity where even individual functions might be their own package in the dependency graph. ESM is a lot easier to tree-shake and you start to see more of the libraries that once published dozens or hundreds of sub-packages start to repackage back to just one top-level package alongside ESM adoption.