8 ms·

The "we can't figure out how to change Linux to use only primary storage" is real. I realized not too long ago that the biggest thing holding back the advanceme

The "we can't figure out how to change Linux to use only primary storage" is real. I realized not too long ago that the biggest thing holding back the advancement of computing is software developers. They can't imagine novel ideas. Try to explain to a developer that dependency hell only exists at all because their own software/compilers/programming languages aren't tracking dependencies within their own application's functions, and they look at you like a dog looking at a human in a dog costume. (Most of them also think YAML is a configuration format and don't know what SDLC stands for, so I guess the dependency thing is way out there)

Retr0id 4y ago
Could you expand on why you think that solves dependency hell? It sounds like you're describing tree-shaking, which is already commonplace, particularly in the JS ecosystem. Or LTO dead-code elimination, in compiled languages - neither of which solve dependency hell.
- 0xbadcafebee 4y ago
  Dependency hell happens when code is subject to an external dependency. Within an application, you can determine which of your functions calls some other version of a function/dependency, can remove code, add code, etc. But you can't do that to other applications; you only control your own application's code and what it depends on/is built against/tested against. So every application is effectively locked into what it's view of the world is, and it just has to hope that never deviates, and that every system it runs on is set up exactly the same. You can play fast and loose with your code by trying to keep the same API/ABI as your underlying code changes, but subtle differences will eventually cause bugs/regressions when expected behaviors change. The way to solve the problem of dependency hell is to version every function, and only call functions based on their versions, and ship every old version of every function. Then the application itself, or a dynamic linker, must find and execute the correct version of the function it needs to call. In this way, dependencies can change at any time, but every line of code will only ever call functions that they were originally built and tested against, because those versions are still knocking around somewhere in the dependency. You can upgrade a dependency 50 times but your application will still just keep calling the old version of the dependency's function, and so the behavior of your application remains the same while the dependency receives upgraded functions. You can upgrade any application at any time and it will just pull in the latest dependency, and continue to call the versions of that dependency that they need. The basic concept for this already exists in glibc as you can ship version-specific functions and link/call a version-specific function. But literally no one uses it. To get widespread adoption and make the paradigm implicit/automatic, we would need to modify the programming language, the compiler, the way programs are executed, require lots more storage, probably new paradigms of caching and layering and shipping code. Package deps would become metadata in a build process, package management would become "scan all executables for dependent versions & download them all". Nix OS is an attempt to work around all of this by pretending that the problem is just with the linker or the file tree or PATH or something. But it's simply impossible to resolve dependency hell entirely without versioned functions, due to rare but intractable intra-package dependency conflicts. And it doesn't address data model versions or network service interface version changes. Containers are an attempt to work around all of this by literally having a different copy of the world for every containerized application. It works well enough, except again the boundary between containers is subject to application interface versions and data model versions changing. (The data model and code are basically the same thing since you need code to do anything with data) The only way to have dependency-hell-free containerized applications is if you version their interfaces/data models and have them call the correct version for other containerized apps [or network services], so again you're back to versions of functions.
  - Retr0id 4y ago
    I think you've just added a 10th circle. More specifically, it seems like you're solving a subset of dependency issues (mainly version conflicts), while making the other issues worse (e.g. keeping sub-dependencies up-to-date, ballooning disk space usage, less easily cachable requests to package repos).
    - 0xbadcafebee 4y ago
      Thank you! ;-) Well that's what dependency hell is, a subset of issues: https://en.wikipedia.org/wiki/Dependency_hell https://en.wikipedia.org/wiki/Dependency_hell The other issues are of course important but require their own solutions. Keeping dependencies up to date [or, in the case of the above method, recompiling applications to use new function versions with security fixes] requires CI/CD, feature flags, auto-upgrading, avoiding drift, etc. Disk space and caching issues require storage with built-in delta layers/CoW/compression (which can also be applied to dependencies, e.g download the dependent versions of functions rather than keeping all of them on disk at once). As we get more advanced, the hell will get deeper for sure.
  - asah 4y ago
    "The way to solve the problem of dependency hell is to version every function... always ship every old version of every function..." This seems like hell to maintain and transitively speaking, a mountain of code. For this much work, I'd prefer to invest in obscenely detailed unittests that allow the team to retire old function-versions and everybody stay on HEAD. That said, I can imagine cases where old-versions might be helpful for some period of time, and you could call them via their function name and commit hash... then a dependency detector only keeps old versions as needed, a warning tool detects ancient versions to consider retiring, etc. You'd need editor/debugger support so developers can see and interact with old versions, and I'm not sure how this works with raw text editors - perhaps the dependency detector copies the (transitive) code into a new subdirectory?
    - 0xbadcafebee 4y ago
      Yeah, I updated my comment to reflect that, you definitely would only need versions of functions on disk where some code depends on it. That's basically how container layers work. I think CI/CD practices are really important to get rid of the old code. Automated build systems should be constantly downloading new versions of deps and running tests so devs can fix bugs quickly and release new versions that depend on new deps. The quicker that cycle happens, the quicker old code can disappear, new functionality can be implemented, bugs can be fixed, etc. Because everything would still use pinned versions of deps, it would still work exactly as it was tested, so you wouldn't be sacrificing reliability for up-to-date-ness. Speed and automation of that process are critical.
dboreham 4y ago
Oh, an easy one : https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control
btilly 4y ago
And when you track fine-grained dependencies that way the result is that upgrading an insecure version of a library still leaves a dozen other copies in memory that leave you still vulnerable. Dependency hell isn't just dealing with incompatible dependencies. It has another side of forcing upgrades when they are needed for very, very good reason.
- 6510 4y ago
  The old version of the function could get security patches. There could be a dialog that warns the user and offers to disable functionality. We will have to eventually do an accurate permission system anyway.
  - Retr0id 4y ago
    Who's going to write the patches? Out-of-support branches aren't suddenly going to start being supported again, just because you've changed how dependencies are managed. Lets say there are 10 versions of my function. A critical vulnerability is discovered, and I publish a fix in version 11. I hire someone to backport the fix to all 10 historic versions of the function. Now I have 21 distinct versions of the same function. Oh no, another critical vulnerability has been discovered! I fix it in version 22, and hire a team to backport the fix to all 21 other versions. Now I have 43 versions of the same function! Oh no... Sure, you could be a bit more pragmatic about which history nodes you backport to, but it's still fundamentally O(n^2). It's hard enough to manage backports at a whole-project level, let alone per-function.
  - dagw 4y ago
    The old version of the function could get security patches. But then it wouldn't be the old version of the function any more, but a brand new function. If you allow arbitrarily changing 'old' function then you just lost all the guarantees of stability and reproducability that this approach was supposed to offer.
AdamH12113 4y ago
Dependency hell strikes me as one of those things where people keep trying to apply technological solutions to a sociological problem and failing over and over again. If we can’t produce stable APIs or recognize that Hyrum’s Law is a sign of fatally-flawed engineering, I don’t think that any amount of versioning is ultimately going to help.
Shorel 4y ago
Dependency hell exists because people still think static linking was not a great idea. All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. If you can update a library, you can also update all other software, and usually from the same source.
- dehrmann 4y ago
  > Dependency hell exists because people still think static linking was not a great idea. Then someone decided to call it Docker.
  - Shorel 4y ago
    Exactly, containers are static linking with many extra steps. And more memory and storage requirements.
    - 0xbadcafebee 4y ago
      If your app calls the tool yq, is it written for yq v3, or v4? v4 has breaking changes in its interface. Just statically linking your app won't deal with that, but a container with both your app and yq v3 will. What containers don't deal with is the interfaces between containers.
- saagarjha 4y ago
  > All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. No, it's more that they were designed by entities that distributed their software using static linking.
  - pcwalton 4y ago
    In Rust's case it was popular demand. Old Rust used dynamic linking a lot more. But our users demanded static linking (Go's popularity was a big driver of this), so the decision was made to do static linking by default.
josephg 4y ago
> I realized not too long ago that the biggest thing holding back the advancement of computing is software developers. They can't imagine novel ideas. Hey - Software engineers can imagine novel ideas. The problem is that every year, there's more code in the OS, and there's more code in web browsers, and there's more applications which depend on all that stuff. Chrome has 25 million lines. At that size, its really hard to innovate on how the browser works. And that matters because some solutions require changing the operating environment. For example, I'd love to replace the filesystem with an append-only log, and replace files with CRDT based datasets. That would solve all sorts of problems. But doing so would require changing linux, and changing most existing software. Every year that passes, that gets harder and harder to do. I don't know what the answer is here. Lots of people "solve this" by building further and further up the stack. Linux binaries not feature rich enough for you? We could fix them, but lets use Docker instead. Oh, you want to make a desktop app? You could use the native APIs for that; but who can be bothered learning the native UI APIs? Its much easier to write web app and ship a copy of Chrome on every platform. We have to find a better answer than going "up the stack" every few years. But so long as people find it easier to write software than read software, its be hard to imagine the system getting simpler over time.
- mort96 4y ago
  > I'd love to replace the filesystem with an append-only log How would you solve the problem of, "I have a finite amount of storage and my drive is full and I need to delete something to make space for this new thing I want"?
  - josephg 4y ago
    I'd make it be an append-only log + some sort of kafka style compaction. So for example you can tell it to compact all the parts of the log involving that file you've deleted. Create + Delete compacts to a no-op.
  - solardev 4y ago
    Some sort of smart FIFO system that combines old entries and then culls them? Time Machine (apple's automated backup thing) does something similar, at a simpler level
formerly_proven 4y ago
It's a pretty consistent pattern that hardware can do things and software fails to utilize them.
mort96 4y ago
Why do we _want_ to change Linux to use only primary storage? Having a hard distinction between persistent and non-persistent memory is insanely useful. What would we do when something goes wrong if rebooting stops being a thing?
- rbanffy 4y ago
  There are a couple aspects that are interesting. With Optane, there is still a distinction between persistent and volatile primary storage - you don't want things that change quickly to be on persistent memory because it'll be slower to update and wear down the device. What you gain is fast memory addressable persistent storage. As for rebooting not being a thing, the IBM AS/400 and its offspring (currently named IBM i) have been doing it for decades now. They are very alien when compared to Unix, but still quite cool. And extraordinarily mainframe-like reliable. There's a reason they cost a small fortune.
Brian_K_White 4y ago
You forgot to include the link to your github where you showed us all how it's done.
imtringued 4y ago
We already do the function tracking with left-pad and isOdd. I don't like it. Frankly, what we actually need is a unified package format like we have for containers rather than specific micro optimizations that nobody really asked for. If gradle, npm, pip, cargo, pacman, apt all speak the same package format you only need one registry for all languages. After that you can think about building a unified package manager. After that you can think about a unified build system.

Retr0id 4y ago

Could you expand on why you think that solves dependency hell? It sounds like you're describing tree-shaking, which is already commonplace, particularly in the JS ecosystem. Or LTO dead-code elimination, in compiled languages - neither of which solve dependency hell.

0xbadcafebee 4y ago
Dependency hell happens when code is subject to an external dependency. Within an application, you can determine which of your functions calls some other version of a function/dependency, can remove code, add code, etc. But you can't do that to other applications; you only control your own application's code and what it depends on/is built against/tested against. So every application is effectively locked into what it's view of the world is, and it just has to hope that never deviates, and that every system it runs on is set up exactly the same. You can play fast and loose with your code by trying to keep the same API/ABI as your underlying code changes, but subtle differences will eventually cause bugs/regressions when expected behaviors change. The way to solve the problem of dependency hell is to version every function, and only call functions based on their versions, and ship every old version of every function. Then the application itself, or a dynamic linker, must find and execute the correct version of the function it needs to call. In this way, dependencies can change at any time, but every line of code will only ever call functions that they were originally built and tested against, because those versions are still knocking around somewhere in the dependency. You can upgrade a dependency 50 times but your application will still just keep calling the old version of the dependency's function, and so the behavior of your application remains the same while the dependency receives upgraded functions. You can upgrade any application at any time and it will just pull in the latest dependency, and continue to call the versions of that dependency that they need. The basic concept for this already exists in glibc as you can ship version-specific functions and link/call a version-specific function. But literally no one uses it. To get widespread adoption and make the paradigm implicit/automatic, we would need to modify the programming language, the compiler, the way programs are executed, require lots more storage, probably new paradigms of caching and layering and shipping code. Package deps would become metadata in a build process, package management would become "scan all executables for dependent versions & download them all". Nix OS is an attempt to work around all of this by pretending that the problem is just with the linker or the file tree or PATH or something. But it's simply impossible to resolve dependency hell entirely without versioned functions, due to rare but intractable intra-package dependency conflicts. And it doesn't address data model versions or network service interface version changes. Containers are an attempt to work around all of this by literally having a different copy of the world for every containerized application. It works well enough, except again the boundary between containers is subject to application interface versions and data model versions changing. (The data model and code are basically the same thing since you need code to do anything with data) The only way to have dependency-hell-free containerized applications is if you version their interfaces/data models and have them call the correct version for other containerized apps [or network services], so again you're back to versions of functions.
- Retr0id 4y ago
  I think you've just added a 10th circle. More specifically, it seems like you're solving a subset of dependency issues (mainly version conflicts), while making the other issues worse (e.g. keeping sub-dependencies up-to-date, ballooning disk space usage, less easily cachable requests to package repos).
  - 0xbadcafebee 4y ago
    Thank you! ;-) Well that's what dependency hell is, a subset of issues: https://en.wikipedia.org/wiki/Dependency_hell https://en.wikipedia.org/wiki/Dependency_hell The other issues are of course important but require their own solutions. Keeping dependencies up to date [or, in the case of the above method, recompiling applications to use new function versions with security fixes] requires CI/CD, feature flags, auto-upgrading, avoiding drift, etc. Disk space and caching issues require storage with built-in delta layers/CoW/compression (which can also be applied to dependencies, e.g download the dependent versions of functions rather than keeping all of them on disk at once). As we get more advanced, the hell will get deeper for sure.
- asah 4y ago
  "The way to solve the problem of dependency hell is to version every function... always ship every old version of every function..." This seems like hell to maintain and transitively speaking, a mountain of code. For this much work, I'd prefer to invest in obscenely detailed unittests that allow the team to retire old function-versions and everybody stay on HEAD. That said, I can imagine cases where old-versions might be helpful for some period of time, and you could call them via their function name and commit hash... then a dependency detector only keeps old versions as needed, a warning tool detects ancient versions to consider retiring, etc. You'd need editor/debugger support so developers can see and interact with old versions, and I'm not sure how this works with raw text editors - perhaps the dependency detector copies the (transitive) code into a new subdirectory?
  - 0xbadcafebee 4y ago
    Yeah, I updated my comment to reflect that, you definitely would only need versions of functions on disk where some code depends on it. That's basically how container layers work. I think CI/CD practices are really important to get rid of the old code. Automated build systems should be constantly downloading new versions of deps and running tests so devs can fix bugs quickly and release new versions that depend on new deps. The quicker that cycle happens, the quicker old code can disappear, new functionality can be implemented, bugs can be fixed, etc. Because everything would still use pinned versions of deps, it would still work exactly as it was tested, so you wouldn't be sacrificing reliability for up-to-date-ness. Speed and automation of that process are critical.

dboreham 4y ago

Oh, an easy one : https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control https://en.wikipedia.org/wiki/Synchronous_Data_Link_Control

btilly 4y ago

And when you track fine-grained dependencies that way the result is that upgrading an insecure version of a library still leaves a dozen other copies in memory that leave you still vulnerable. Dependency hell isn't just dealing with incompatible dependencies. It has another side of forcing upgrades when they are needed for very, very good reason.

6510 4y ago
The old version of the function could get security patches. There could be a dialog that warns the user and offers to disable functionality. We will have to eventually do an accurate permission system anyway.
- Retr0id 4y ago
  Who's going to write the patches? Out-of-support branches aren't suddenly going to start being supported again, just because you've changed how dependencies are managed. Lets say there are 10 versions of my function. A critical vulnerability is discovered, and I publish a fix in version 11. I hire someone to backport the fix to all 10 historic versions of the function. Now I have 21 distinct versions of the same function. Oh no, another critical vulnerability has been discovered! I fix it in version 22, and hire a team to backport the fix to all 21 other versions. Now I have 43 versions of the same function! Oh no... Sure, you could be a bit more pragmatic about which history nodes you backport to, but it's still fundamentally O(n^2). It's hard enough to manage backports at a whole-project level, let alone per-function.
- dagw 4y ago
  The old version of the function could get security patches. But then it wouldn't be the old version of the function any more, but a brand new function. If you allow arbitrarily changing 'old' function then you just lost all the guarantees of stability and reproducability that this approach was supposed to offer.

AdamH12113 4y ago

Dependency hell strikes me as one of those things where people keep trying to apply technological solutions to a sociological problem and failing over and over again. If we can’t produce stable APIs or recognize that Hyrum’s Law is a sign of fatally-flawed engineering, I don’t think that any amount of versioning is ultimately going to help.

Shorel 4y ago

Dependency hell exists because people still think static linking was not a great idea. All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. If you can update a library, you can also update all other software, and usually from the same source.

dehrmann 4y ago
> Dependency hell exists because people still think static linking was not a great idea. Then someone decided to call it Docker.
- Shorel 4y ago
  Exactly, containers are static linking with many extra steps. And more memory and storage requirements.
  - 0xbadcafebee 4y ago
    If your app calls the tool yq, is it written for yq v3, or v4? v4 has breaking changes in its interface. Just statically linking your app won't deal with that, but a container with both your app and yq v3 will. What containers don't deal with is the interfaces between containers.
saagarjha 4y ago
> All this while Rust, go, and all the new kids on the block use only static linking. Because static linking is a great idea. No, it's more that they were designed by entities that distributed their software using static linking.
- pcwalton 4y ago
  In Rust's case it was popular demand. Old Rust used dynamic linking a lot more. But our users demanded static linking (Go's popularity was a big driver of this), so the decision was made to do static linking by default.

josephg 4y ago

> I realized not too long ago that the biggest thing holding back the advancement of computing is software developers. They can't imagine novel ideas. Hey - Software engineers can imagine novel ideas. The problem is that every year, there's more code in the OS, and there's more code in web browsers, and there's more applications which depend on all that stuff. Chrome has 25 million lines. At that size, its really hard to innovate on how the browser works. And that matters because some solutions require changing the operating environment. For example, I'd love to replace the filesystem with an append-only log, and replace files with CRDT based datasets. That would solve all sorts of problems. But doing so would require changing linux, and changing most existing software. Every year that passes, that gets harder and harder to do. I don't know what the answer is here. Lots of people "solve this" by building further and further up the stack. Linux binaries not feature rich enough for you? We could fix them, but lets use Docker instead. Oh, you want to make a desktop app? You could use the native APIs for that; but who can be bothered learning the native UI APIs? Its much easier to write web app and ship a copy of Chrome on every platform. We have to find a better answer than going "up the stack" every few years. But so long as people find it easier to write software than read software, its be hard to imagine the system getting simpler over time.

mort96 4y ago
> I'd love to replace the filesystem with an append-only log How would you solve the problem of, "I have a finite amount of storage and my drive is full and I need to delete something to make space for this new thing I want"?
- josephg 4y ago
  I'd make it be an append-only log + some sort of kafka style compaction. So for example you can tell it to compact all the parts of the log involving that file you've deleted. Create + Delete compacts to a no-op.
- solardev 4y ago
  Some sort of smart FIFO system that combines old entries and then culls them? Time Machine (apple's automated backup thing) does something similar, at a simpler level

formerly_proven 4y ago

It's a pretty consistent pattern that hardware can do things and software fails to utilize them.

mort96 4y ago

Why do we _want_ to change Linux to use only primary storage? Having a hard distinction between persistent and non-persistent memory is insanely useful. What would we do when something goes wrong if rebooting stops being a thing?

rbanffy 4y ago
There are a couple aspects that are interesting. With Optane, there is still a distinction between persistent and volatile primary storage - you don't want things that change quickly to be on persistent memory because it'll be slower to update and wear down the device. What you gain is fast memory addressable persistent storage. As for rebooting not being a thing, the IBM AS/400 and its offspring (currently named IBM i) have been doing it for decades now. They are very alien when compared to Unix, but still quite cool. And extraordinarily mainframe-like reliable. There's a reason they cost a small fortune.

Brian_K_White 4y ago

You forgot to include the link to your github where you showed us all how it's done.

imtringued 4y ago

We already do the function tracking with left-pad and isOdd. I don't like it. Frankly, what we actually need is a unified package format like we have for containers rather than specific micro optimizations that nobody really asked for. If gradle, npm, pip, cargo, pacman, apt all speak the same package format you only need one registry for all languages. After that you can think about building a unified package manager. After that you can think about a unified build system.