5 ms·

Could you expand on why you think that solves dependency hell? It sounds like you're describing tree-shaking, which is already commonplace, particularly in the

Could you expand on why you think that solves dependency hell? It sounds like you're describing tree-shaking, which is already commonplace, particularly in the JS ecosystem. Or LTO dead-code elimination, in compiled languages - neither of which solve dependency hell.

0xbadcafebee 4y ago
Dependency hell happens when code is subject to an external dependency. Within an application, you can determine which of your functions calls some other version of a function/dependency, can remove code, add code, etc. But you can't do that to other applications; you only control your own application's code and what it depends on/is built against/tested against. So every application is effectively locked into what it's view of the world is, and it just has to hope that never deviates, and that every system it runs on is set up exactly the same. You can play fast and loose with your code by trying to keep the same API/ABI as your underlying code changes, but subtle differences will eventually cause bugs/regressions when expected behaviors change. The way to solve the problem of dependency hell is to version every function, and only call functions based on their versions, and ship every old version of every function. Then the application itself, or a dynamic linker, must find and execute the correct version of the function it needs to call. In this way, dependencies can change at any time, but every line of code will only ever call functions that they were originally built and tested against, because those versions are still knocking around somewhere in the dependency. You can upgrade a dependency 50 times but your application will still just keep calling the old version of the dependency's function, and so the behavior of your application remains the same while the dependency receives upgraded functions. You can upgrade any application at any time and it will just pull in the latest dependency, and continue to call the versions of that dependency that they need. The basic concept for this already exists in glibc as you can ship version-specific functions and link/call a version-specific function. But literally no one uses it. To get widespread adoption and make the paradigm implicit/automatic, we would need to modify the programming language, the compiler, the way programs are executed, require lots more storage, probably new paradigms of caching and layering and shipping code. Package deps would become metadata in a build process, package management would become "scan all executables for dependent versions & download them all". Nix OS is an attempt to work around all of this by pretending that the problem is just with the linker or the file tree or PATH or something. But it's simply impossible to resolve dependency hell entirely without versioned functions, due to rare but intractable intra-package dependency conflicts. And it doesn't address data model versions or network service interface version changes. Containers are an attempt to work around all of this by literally having a different copy of the world for every containerized application. It works well enough, except again the boundary between containers is subject to application interface versions and data model versions changing. (The data model and code are basically the same thing since you need code to do anything with data) The only way to have dependency-hell-free containerized applications is if you version their interfaces/data models and have them call the correct version for other containerized apps [or network services], so again you're back to versions of functions.
- Retr0id 4y ago
  I think you've just added a 10th circle. More specifically, it seems like you're solving a subset of dependency issues (mainly version conflicts), while making the other issues worse (e.g. keeping sub-dependencies up-to-date, ballooning disk space usage, less easily cachable requests to package repos).
  - 0xbadcafebee 4y ago
    Thank you! ;-) Well that's what dependency hell is, a subset of issues: https://en.wikipedia.org/wiki/Dependency_hell https://en.wikipedia.org/wiki/Dependency_hell The other issues are of course important but require their own solutions. Keeping dependencies up to date [or, in the case of the above method, recompiling applications to use new function versions with security fixes] requires CI/CD, feature flags, auto-upgrading, avoiding drift, etc. Disk space and caching issues require storage with built-in delta layers/CoW/compression (which can also be applied to dependencies, e.g download the dependent versions of functions rather than keeping all of them on disk at once). As we get more advanced, the hell will get deeper for sure.
- asah 4y ago
  "The way to solve the problem of dependency hell is to version every function... always ship every old version of every function..." This seems like hell to maintain and transitively speaking, a mountain of code. For this much work, I'd prefer to invest in obscenely detailed unittests that allow the team to retire old function-versions and everybody stay on HEAD. That said, I can imagine cases where old-versions might be helpful for some period of time, and you could call them via their function name and commit hash... then a dependency detector only keeps old versions as needed, a warning tool detects ancient versions to consider retiring, etc. You'd need editor/debugger support so developers can see and interact with old versions, and I'm not sure how this works with raw text editors - perhaps the dependency detector copies the (transitive) code into a new subdirectory?
  - 0xbadcafebee 4y ago
    Yeah, I updated my comment to reflect that, you definitely would only need versions of functions on disk where some code depends on it. That's basically how container layers work. I think CI/CD practices are really important to get rid of the old code. Automated build systems should be constantly downloading new versions of deps and running tests so devs can fix bugs quickly and release new versions that depend on new deps. The quicker that cycle happens, the quicker old code can disappear, new functionality can be implemented, bugs can be fixed, etc. Because everything would still use pinned versions of deps, it would still work exactly as it was tested, so you wouldn't be sacrificing reliability for up-to-date-ness. Speed and automation of that process are critical.

0xbadcafebee 4y ago

Dependency hell happens when code is subject to an external dependency. Within an application, you can determine which of your functions calls some other version of a function/dependency, can remove code, add code, etc. But you can't do that to other applications; you only control your own application's code and what it depends on/is built against/tested against. So every application is effectively locked into what it's view of the world is, and it just has to hope that never deviates, and that every system it runs on is set up exactly the same. You can play fast and loose with your code by trying to keep the same API/ABI as your underlying code changes, but subtle differences will eventually cause bugs/regressions when expected behaviors change. The way to solve the problem of dependency hell is to version every function, and only call functions based on their versions, and ship every old version of every function. Then the application itself, or a dynamic linker, must find and execute the correct version of the function it needs to call. In this way, dependencies can change at any time, but every line of code will only ever call functions that they were originally built and tested against, because those versions are still knocking around somewhere in the dependency. You can upgrade a dependency 50 times but your application will still just keep calling the old version of the dependency's function, and so the behavior of your application remains the same while the dependency receives upgraded functions. You can upgrade any application at any time and it will just pull in the latest dependency, and continue to call the versions of that dependency that they need. The basic concept for this already exists in glibc as you can ship version-specific functions and link/call a version-specific function. But literally no one uses it. To get widespread adoption and make the paradigm implicit/automatic, we would need to modify the programming language, the compiler, the way programs are executed, require lots more storage, probably new paradigms of caching and layering and shipping code. Package deps would become metadata in a build process, package management would become "scan all executables for dependent versions & download them all". Nix OS is an attempt to work around all of this by pretending that the problem is just with the linker or the file tree or PATH or something. But it's simply impossible to resolve dependency hell entirely without versioned functions, due to rare but intractable intra-package dependency conflicts. And it doesn't address data model versions or network service interface version changes. Containers are an attempt to work around all of this by literally having a different copy of the world for every containerized application. It works well enough, except again the boundary between containers is subject to application interface versions and data model versions changing. (The data model and code are basically the same thing since you need code to do anything with data) The only way to have dependency-hell-free containerized applications is if you version their interfaces/data models and have them call the correct version for other containerized apps [or network services], so again you're back to versions of functions.

Retr0id 4y ago
I think you've just added a 10th circle. More specifically, it seems like you're solving a subset of dependency issues (mainly version conflicts), while making the other issues worse (e.g. keeping sub-dependencies up-to-date, ballooning disk space usage, less easily cachable requests to package repos).
- 0xbadcafebee 4y ago
  Thank you! ;-) Well that's what dependency hell is, a subset of issues: https://en.wikipedia.org/wiki/Dependency_hell https://en.wikipedia.org/wiki/Dependency_hell The other issues are of course important but require their own solutions. Keeping dependencies up to date [or, in the case of the above method, recompiling applications to use new function versions with security fixes] requires CI/CD, feature flags, auto-upgrading, avoiding drift, etc. Disk space and caching issues require storage with built-in delta layers/CoW/compression (which can also be applied to dependencies, e.g download the dependent versions of functions rather than keeping all of them on disk at once). As we get more advanced, the hell will get deeper for sure.
asah 4y ago
"The way to solve the problem of dependency hell is to version every function... always ship every old version of every function..." This seems like hell to maintain and transitively speaking, a mountain of code. For this much work, I'd prefer to invest in obscenely detailed unittests that allow the team to retire old function-versions and everybody stay on HEAD. That said, I can imagine cases where old-versions might be helpful for some period of time, and you could call them via their function name and commit hash... then a dependency detector only keeps old versions as needed, a warning tool detects ancient versions to consider retiring, etc. You'd need editor/debugger support so developers can see and interact with old versions, and I'm not sure how this works with raw text editors - perhaps the dependency detector copies the (transitive) code into a new subdirectory?
- 0xbadcafebee 4y ago
  Yeah, I updated my comment to reflect that, you definitely would only need versions of functions on disk where some code depends on it. That's basically how container layers work. I think CI/CD practices are really important to get rid of the old code. Automated build systems should be constantly downloading new versions of deps and running tests so devs can fix bugs quickly and release new versions that depend on new deps. The quicker that cycle happens, the quicker old code can disappear, new functionality can be implemented, bugs can be fixed, etc. Because everything would still use pinned versions of deps, it would still work exactly as it was tested, so you wouldn't be sacrificing reliability for up-to-date-ness. Speed and automation of that process are critical.