4 ms·

Around 2002 or so, I had an idea to tag every part of a project with a unique hash code. With a hash code, one could download the corresponding file. A hash cod

Around 2002 or so, I had an idea to tag every part of a project with a unique hash code. With a hash code, one could download the corresponding file. A hash code for the whole project would be a file containing a list of hash codes for the files that make up the project. Hash codes could represent the compiler that builds it, along with the library(s) it links with. I showed it to a couple software entrepreneuers (Wild Tangent and Chromium), but they had no interest in it. I never did anything else with it, and so it goes.

DiggyJohnson 2y ago
Hey Walter, what would you improve with Git?
- WalterBright 2y ago
  Git hasn't quite taken the step of making the hash the URL you use to download a file, any file, and be assured it is exactly what you thought it was, as the hash of the file must match its URL. This is currently done in a haphazard way, not particularly organized.
  - exe34 2y ago
    git over ipfs then?
    - mdaniel 2y ago
      I believe that's approximately what this is trying to do https://radicle.xyz/#:~:text=radicle%20is%20an%20open%20source%2C%20peer-to-peer%20code%20collaboration%20stack%20built%20on%20git https://radicle.xyz/#:~:text=radicle%20is%20an%20open%20sour... although evidently using a custom protocol not ipfs itself
swsieber 2y ago
While I get how that's like git, it sounds even closer to unison: https://softwaremill.com/trying-out-unison-part-1-code-as-hashes/ https://softwaremill.com/trying-out-unison-part-1-code-as-ha...
- WalterBright 2y ago
  20 years later :-)
WalterBright 2y ago
I had actually done a writeup on it, and thought I had lost it. I found it, dated 2/15/2002: --- Consider that any D app is completely specified by a list of .module files and the tools necessary to compile them. Assign a unique GUID to each unique .module file. Then, an app is specified by a list of .module GUIDs. Each app is also assigned a GUID. On the client's machine is stored a pool of already downloaded .module files. When a new app is downloaded, what is actually downloaded is just a GUID. The client sees if that GUID is an already built app in the pool, then he's done. If not, the client requests the manifest for the GUID, a manifest being a list of .module GUIDs. Each GUID in the manifest is checked against the client pool, any that are not found are downloaded and added to the pool. Once the client has all the .module files for the GUIDs that make up an app, they can all be compiled, linked, and the result cached in the pool. Thus, if an app is updated, only the changed .module files ever need to get downloaded. This can be taken a step further and a changed .module file can be represented as a diff from a previous .module. Since .module files are tokenized source, two source files that differ only in comments and whitespace will have identical .module files. There will be a master pool of .module files on WT's server. When an app is ready to release, it is "checked in" to the master pool by assigning GUIDs to its .module files. This master pool is what is consulted by the client when requesting .module files by GUID. The D "VM" compiler, linker, engine, etc., can also be identified by GUIDs. This way, if an app is developed with a particular combination of tools, it can specify the GUIDs for them in the manifest. Hence the client will automatically download "VM" updates to get the exact tools needed to duplicate the app exactly.
- endgame 2y ago
  Sounds like it's also halfway to a version of Nix designed specifically for D toolchains, too, using GUIDs instead of hashing inputs.
  - WalterBright 2y ago
    It wasn't designed specifically for D toolchains, that was just an example of what it could do.
- pmarreck 2y ago
exe34 2y ago
So you invented nix :-D
frutiger 2y ago
Your description (including the detailed description in the reply) seems to be missing the crucial difference that git uses - the hash code of the object is not some GUID, it is literally the hash of the content of the object. This makes a big difference as you don't need some central registry that maps the GUID to the object.
- fragmede 2y ago
  Every git repo has a copy of that mapping instead of there being a central registry though, and because the commit author's name and email, and the date of the commit and a commit message (among other things) go into the hash that represents a commit, it's not that big a difference, is it? Given a collection of files, but not the git repo they're from, and libgit, I can't say if those files match a git tag hash if I don't also have the metadata that makes up the commit to make the git hash, and not just the files inside of it.
  - ajross 2y ago
    Yes, but the commit object (which includes metadata) references a tree object by its hash. The tree object is a text representation of a directory tree, basically, referencing file blobs by hash. So yes, you can recognize identical files between commits. It's true there's no fast indexing: if you want to ask the question "which commits contain exactly this file?" you have to search every commit. But you don't need to delta the file contents itself.
    - fragmede 2y ago
      but people don't use the file hash, that's internal to git. I go to the centralized repository of repositories at github.com and look up tagged version 1.0.0 of whatever software, which refers to a git tag which references a commit hash (which yes it references a tree object as you said).
      - XorNot 2y ago
        That's for human consumption though, which is what frustrates so many "hashing will solve everything!" schemes - it breaks as soon as you need a bug fix. At the end of the day none of us want "exactly this hash" we want "latest". Exact hashes and other reproducibility are things which are useful when debugging or providing traceability - valuable but also not the human side of the equation.
pmarreck 2y ago
Isn't this basically... a Merkle Tree, the underlying storage architecture of things like git and Nix? https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree Except that instead of a GUID, it's just a hash of the binary data itself, which ends up being more useful because it is a natural key and doesn't require storing a separate mapping
- WalterBright 2y ago
  I'd never heard of a Merkle Tree before, thanks for the reference.
- davidgl 2y ago
  Yep, invented in 1979 and also the core data structure of crypto block chains
emmelaich 2y ago
Similarly but I also had rsync or rdiff as a central character in my mental model of a VCS.

DiggyJohnson 2y ago

Hey Walter, what would you improve with Git?

WalterBright 2y ago
Git hasn't quite taken the step of making the hash the URL you use to download a file, any file, and be assured it is exactly what you thought it was, as the hash of the file must match its URL. This is currently done in a haphazard way, not particularly organized.
- exe34 2y ago
  git over ipfs then?
  - mdaniel 2y ago
    I believe that's approximately what this is trying to do https://radicle.xyz/#:~:text=radicle%20is%20an%20open%20source%2C%20peer-to-peer%20code%20collaboration%20stack%20built%20on%20git https://radicle.xyz/#:~:text=radicle%20is%20an%20open%20sour... although evidently using a custom protocol not ipfs itself

swsieber 2y ago

While I get how that's like git, it sounds even closer to unison: https://softwaremill.com/trying-out-unison-part-1-code-as-hashes/ https://softwaremill.com/trying-out-unison-part-1-code-as-ha...

WalterBright 2y ago
20 years later :-)

WalterBright 2y ago

I had actually done a writeup on it, and thought I had lost it. I found it, dated 2/15/2002: --- Consider that any D app is completely specified by a list of .module files and the tools necessary to compile them. Assign a unique GUID to each unique .module file. Then, an app is specified by a list of .module GUIDs. Each app is also assigned a GUID. On the client's machine is stored a pool of already downloaded .module files. When a new app is downloaded, what is actually downloaded is just a GUID. The client sees if that GUID is an already built app in the pool, then he's done. If not, the client requests the manifest for the GUID, a manifest being a list of .module GUIDs. Each GUID in the manifest is checked against the client pool, any that are not found are downloaded and added to the pool. Once the client has all the .module files for the GUIDs that make up an app, they can all be compiled, linked, and the result cached in the pool. Thus, if an app is updated, only the changed .module files ever need to get downloaded. This can be taken a step further and a changed .module file can be represented as a diff from a previous .module. Since .module files are tokenized source, two source files that differ only in comments and whitespace will have identical .module files. There will be a master pool of .module files on WT's server. When an app is ready to release, it is "checked in" to the master pool by assigning GUIDs to its .module files. This master pool is what is consulted by the client when requesting .module files by GUID. The D "VM" compiler, linker, engine, etc., can also be identified by GUIDs. This way, if an app is developed with a particular combination of tools, it can specify the GUIDs for them in the manifest. Hence the client will automatically download "VM" updates to get the exact tools needed to duplicate the app exactly.

endgame 2y ago
Sounds like it's also halfway to a version of Nix designed specifically for D toolchains, too, using GUIDs instead of hashing inputs.
- WalterBright 2y ago
  It wasn't designed specifically for D toolchains, that was just an example of what it could do.
pmarreck 2y ago

exe34 2y ago

So you invented nix :-D

frutiger 2y ago

Your description (including the detailed description in the reply) seems to be missing the crucial difference that git uses - the hash code of the object is not some GUID, it is literally the hash of the content of the object. This makes a big difference as you don't need some central registry that maps the GUID to the object.

fragmede 2y ago
Every git repo has a copy of that mapping instead of there being a central registry though, and because the commit author's name and email, and the date of the commit and a commit message (among other things) go into the hash that represents a commit, it's not that big a difference, is it? Given a collection of files, but not the git repo they're from, and libgit, I can't say if those files match a git tag hash if I don't also have the metadata that makes up the commit to make the git hash, and not just the files inside of it.
- ajross 2y ago
  Yes, but the commit object (which includes metadata) references a tree object by its hash. The tree object is a text representation of a directory tree, basically, referencing file blobs by hash. So yes, you can recognize identical files between commits. It's true there's no fast indexing: if you want to ask the question "which commits contain exactly this file?" you have to search every commit. But you don't need to delta the file contents itself.
  - fragmede 2y ago
    but people don't use the file hash, that's internal to git. I go to the centralized repository of repositories at github.com and look up tagged version 1.0.0 of whatever software, which refers to a git tag which references a commit hash (which yes it references a tree object as you said).
    - XorNot 2y ago
      That's for human consumption though, which is what frustrates so many "hashing will solve everything!" schemes - it breaks as soon as you need a bug fix. At the end of the day none of us want "exactly this hash" we want "latest". Exact hashes and other reproducibility are things which are useful when debugging or providing traceability - valuable but also not the human side of the equation.

pmarreck 2y ago

Isn't this basically... a Merkle Tree, the underlying storage architecture of things like git and Nix? https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree Except that instead of a GUID, it's just a hash of the binary data itself, which ends up being more useful because it is a natural key and doesn't require storing a separate mapping

WalterBright 2y ago
I'd never heard of a Merkle Tree before, thanks for the reference.
davidgl 2y ago
Yep, invented in 1979 and also the core data structure of crypto block chains

emmelaich 2y ago

Similarly but I also had rsync or rdiff as a central character in my mental model of a VCS.