10 ms·

Distributing NixOS with IPFS

241 points by robto 10y ago

citrusui 10y ago
I'm really excited to see what the future holds for IPFS! However, hosting websites with custom domains is not quite feasible yet. Using IPFS' DNS (IPNS) means you have to keep the IPFS daemon running constantly, or else the files will be purged within an hour.
- diggan 10y ago
  > However, hosting websites with custom domains is not quite feasible yet Sure, point your domain to your IPFS hash and use dnslink, it's quite reliable already actually. That's how ipfs.io is hosted for example, and we haven't hit any issues so far. > Using IPFS' DNS (IPNS) means you have to keep the IPFS daemon running constantly, or else the files will be purged within an hour So, the files won't be purged, but the record you push out with IPNS won't be valid after 24 hours. You can solve this easily by using /ipfs/:hash instead of /ipns/:id and it wont disappear.
  - citrusui 10y ago
    Yet again I failed to include important details... :p Yeah, it certainly is possible to host static sites on IPFS -- I have been testing it on my site[0] just for kicks. However, since I really enjoy using my domain name, rather than "ipfs.io/ip{f|n}s/$hash", I'm reluctant to try adapting my site to IPFS. I am aware that this is an alpha product, though, and I can't stress enough how cool it is for files containing the same data to be given the same ID (hash). That way you don't have to run shasum ever again :D [0]: https://ipfs.citrusui.me https://ipfs.citrusui.me (on a random note: what's up with /blog returning the IPFS blog, instead of my own content? did i misconfigure something?)
    - lgierth 10y ago
      The HTTP gateway included in go-ipfs will try to use a Host header on requests for constructing an /ipns path. So e.g. http://ipfs.io/docs http://ipfs.io/docs gets turned into /ipns/ipfs.io/docs -- this is actually how we host all webpages within the ipfs project. edit: And the part which turns /ipns/ipfs.io into an /ipfs path is called dnslink: https://github.com/ipfs/go-dnslink https://github.com/ipfs/go-dnslink -- it resolves to what's in TXT _dnslink.ipfs.io. > (on a random note: what's up with /blog returning the IPFS blog, instead of my own content? did i misconfigure something?) I'm so sorry, that's a bug I put into the nginx configuration -- will fix it!
vog 10y ago
Very interesting development. It would be great to see NixOS as an early adopter for IPFS. BTW, there is a small typo: IPFS is aims to create the distributed net. It should be: IPFS aims to create the distributed net.
- sly010 10y ago
  This is a great idea. A lot of businesses heavily rely on old versions of open source packages always being available. The one time someone deprecated an npm package, half of the nodejs stacks went with it. edit: Didn't mean to hit reply. Sorry.
  - vog 10y ago
    Just delete your comment and put it on toplevel again.
    - sp332 10y ago
      You can't delete comments that have been replied to.
      - vog 10y ago
        You can, and the reply would remain. The tree then looks like this: comment1 [deleted] comment3 However, you can delete comments only until a certain amount of time. If you wait too long, you can't delete anymore (and this is what happened here).
        sp332 10y ago
        That's how it used to work, but I'm pretty sure it doesn't work like that anymore.
        vog 10y ago
        I wonder if I'm the only one who is annoyed by the continuing lack of transparency at HN, wasting my time and other people's time with guesswork like this. I just took the time to start an "Ask HN" entry on that topic: "Ask HN: Where can I follow the changes of HN itself?" https://news.ycombinator.com/item?id=13460588 https://news.ycombinator.com/item?id=13460588
drdre2001 10y ago
This is a really great idea! Reminds me of other projects that are working on integrating IPFS with the Operating System: https://github.com/vtomole/IPOS https://github.com/vtomole/IPOS
chriswarbo 10y ago
I've been following these github issues for a while; fetching sources from IPFS seems like a great step forward for resiliency in general, and quite a natural one for Nix considering things are already immutable. Using IPFS as a binary cache is nice, as it would lower the maintainers' burden and make out-of-tree experimentation easier, i.e. without damaging the integrity of nixpkgs and cache.nixos.org. I hadn't even thought about using the FUSE integration of IPFS, but it makes a lot of sense. Nix is a lazy language, and the nixpkgs repository basically defines one big value: a set of name/value pairs for every package it contains (as well various libraries for e.g. working with Python packages, Haskell packages, etc.). The only difference between installed/uninstalled packages is whether anything's forced the contents to be evaluated yet. Likewise, an IPFS FUSE mount conceptually contains the whole of IPFS. The only difference between downloaded/undownloaded files is whether anything's forced the contents to be evaluated yet.
- ris 10y ago
  This article doesn't mention the most significant fallout of the IPFS idea (imo), which is that of .nar deduplication, as detailed in the issue https://github.com/NixOS/nix/issues/859 https://github.com/NixOS/nix/issues/859 (point 4). Perhaps a nail in the coffin of one of Nix's biggest absurdities.
cjbprime 10y ago
Very cool. One benefit of schemes like this that people don't talk about much is that, by no longer downloading from an expected place, you're removing the possibility for a compromised developer or server operator to selectively serve up malware to a targeted user. Instead you're getting the file over bittorrent and checking its hash, and you could gossip with other bittorrent clients to confirm that everyone's trying to get the same hash. Compare with the state of the art in most software updates, which is that you connect to some download server and it could serve signed malware to people on its target list and probably no-one would notice. (Schemes that use some of these techniques to take out the single point of malware-insertion have been called "Binary Transparency" schemes, as an analog to Certificate Transparency.)
- nextos 10y ago
  Guix has a very good complementary approach to this problem, guix challenge. Perhaps it's also implemented in Nix too already: https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-challenge.html https://www.gnu.org/software/guix/manual/html_node/Invoking-... Basically, since builds are reproducible, you can automatically build from source and see if the hash of the binary you built matches the one you are downloading. Obviously, source can be still compromised. But that's probably something IPFS won't fix unless wherever you get sources from is also on IPFS.
  - grhmc 10y ago
    `nix-build` can take `--check` to do a similar thing. However, not all packages are reproducible. We've been doing a bit of work on this: https://garbas.si/2016/reproducible-builds-summit-in-berlin.html https://garbas.si/2016/reproducible-builds-summit-in-berlin.... and have begun checking reproducibility in our CI system: http://hydra.nixos.org/jobset/nixos/reproducibility http://hydra.nixos.org/jobset/nixos/reproducibility
  - davexunit 10y ago
    Just a tiny nitpick, if I may: most builds are reproducible, but not all. Out of ~5000 packages, ~600 are not yet reproducible.
civodul 10y ago
Nice project! Guix had a GSoC student working on binary distribution using GNUnet's file sharing component a while back: https://gnu.org/s/guix/news/gsoc-update.html https://gnu.org/s/guix/news/gsoc-update.html . That has not led (yet?) to production code, but there might be ideas worth sharing.
k__ 10y ago
It's almost ridiculous how good the two fit together. I had the feeling NixOS has a bit of a hard time get users and prove that it's a superior solution to ansible/docker/chef/etc. probably because of it's mediocre UX, haha. But this would add another killer feature to it.
rkeene2 10y ago
Good to see other people are inventing AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) :-)
- HurrdurrHodor 10y ago
  Please don't use SHA-1. It's almost broken.
  - lgierth 10y ago
    Use multihashes for hash algorithm agility :) https://github.com/multiformats/multihash https://github.com/multiformats/multihash
    - rkeene2 10y ago
      Multiple hashing algorithms is already built-in and mandatory, everywhere a hashing operation is used the hashing algorithm must also be specified.
  - rkeene2 10y ago
    The protocol is actually extensible, and the hashing algorithm MUST always be specified by the server (which the client could then choose to not accept, just as it can reject the certificate because of the signature algorithm). Also, it would require a preimage attack against one of the hashed items to be useful which SHA-1 will likely be resistant to a long time (though decreasing with the number of items hashed) and SHA-1 is unlikely to be vulnerable to a preimage attack in the near future based on what we know so far. The signature and certificates that are used to validate the top-level index can be based on a far better hashing algorithm independently of the content-based hashing.
- _prometheus 10y ago
  > Good to see other people are inventing AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) :-) I'll take the construction "other people are inventing <the thing i invented some time ago>" to mean that you think you came up with this first, or at least prior to Nix or IPFS. And thus ":-)" to be a bit sarcastic and unhappy, instead of genuinely happy. Similar times: * http://appfs.rkeene.org/web/timeline?c=78c60b0c9e7da1c9&unhide http://appfs.rkeene.org/web/timeline?c=78c60b0c9e7da1c9&unhi... * https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c19b7 https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1... * https://github.com/ipfs/ipfs/commit/8004db75262fcd29399d6c7fa979b70f20546884#diff-47ff1e80cf6b0d77a3e258a1e6288416R1102 https://github.com/ipfs/ipfs/commit/8004db75262fcd29399d6c7f... * https://github.com/jbenet/random-ideas/issues/19 https://github.com/jbenet/random-ideas/issues/19 * i think the Nix people probably came up with this way before either of us * and i am willing to bet everything that at least 100 other people (maybe thousands) have came up with this exact same idea over a decade before any of us... In fact, most of the best "original ideas" in the IPFS body of work were probably first discovered decades prior. I repeatedly see people succumbing to sadness over multiple discovery. It shouldn't be sad, it should be a happy event, as it confirms our own thoughts and presents an opportunity for collaborations. :) Further thoughts here: https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c19b7 https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1...
twoodfin 10y ago
I've felt for a while that a standard, widely-implemented, distributed content-addressable store is one of the biggest missing pieces of the modern internet. Glad to see any steps in that direction. I'll know real progress has been made when my browser can resolve something like: cas://sha256/2d66257e9d2cda5c08e850a947caacbc0177411f379f986dd9a8abc653ce5a8e
- cjbprime 10y ago
  See https://github.com/beakerbrowser/beaker https://github.com/beakerbrowser/beaker :)
- deleted 10y ago
  [deleted]
- problems 10y ago
  Bittorrent's DHT is used that way now, it's probably the biggest public DHT deployment in existence, all a magnet link is at it's core is magnet:?xt=urn:btih:<infohash>, so just having an infohash of a torrent is enough for you to get its content. Of course, there's also IPFS, Zeronet and Freenet which all address this exact issue in slightly different ways, all more web-targetted.
  - djsumdog 10y ago
    I thought Zeronet used IPFS underneath?
    - problems 10y ago
      I believe it's an entirely separate network, not DHT based like IPFS though, but they've done a lot of cool work on the client side of things to allow user accounts and stuff.
- jstanley 10y ago
  Morphis is supposed to provide something like this: https://morph.is/ https://morph.is/
matthewbauer 10y ago
Stage 2 seems problematic at least the way I see it. Most users have at least a thousand derivations- is it possible to fuse mount each one? Also: I think some people are unaware that Nix hashes are not content addressable. The best solution (which OP is proposing) is probably to use the .nar hashes in IPFS which is content addressable.
Ericson2314 10y ago
I want to ditch the Nar format as soon as possible. IPFS's unixfs format is too rich however. When will the IPFS people finish up https://github.com/ipld/cid https://github.com/ipld/cid so we can link whatever content addressable data we want? I'd use git tree objects, despite SHA-1, because it's widely supported. Or do a format identical tree objects but with the IPFS's multihash and SHA-1 banned. Point is, underlying protocol should be agnostic to hashing scheme, we should have a trait/type class like /// Node in try trait Payload { type Hash: HashingTrait; fn unpack(Payload) -> (Vec<u8>, Set<Hash>); fn pack(Vec<u8>, Set<Hash>) -> Payload; // Implement either and get the other for free! fn hash_packed(p: Payload) -> Hash { hash_unpacked(packed(p)) fn hash_unpacked(p: (Vec<u8>, Set<Hash>)) -> Hash { hash_packed(packed(p)) } } any `(Hash, Payload)` than can define a `(binary blob, Set<Hash>) -> Hash` and Payload function should work.
- whyrusleeping 10y ago
  Hey! IPFS Dev here. The cid stuff has been implemented and initial support for it is being landed in our 0.4.5 release (which will be soon, hopefully release candidate within a week). With that and IPLD, you can craft arbitrary objects in JSON or CBOR (theres a 1 to 1 mapping, objects are stored as cbor) and work with them in ipfs. For example, i could make an object that looks like: { "Contents": {"/":"QmHashOfPkgContents"}, "Compression": "bzip2", "NarSize": 12345, "References": { "foo": {"/": "QmHashOfFoo"}, "bar": {"/": "QmHashOfBar"} }, "Signature": "signature info, or a link to the signature", } (please excuse my attempt at recreating a nar file in rough json). This object could then be put into ipfs with: cat thing.json | ipfs dag put And you would get an ipld object that you can move around in ipfs, and do fun things like: ipfs get <thatobjhash>/Contents to download the package contents, or: ipfs get <thatobjhash>/References/foo to get the referenced package (or open that hash/path in an ipfs gateway to browse the package graph for free in your browser :) )
  - Ericson2314 10y ago
    IPLD does allow storing tons of data, but custom schemas allow restricting the data referenced in arbitrary ways. IPLD, last I checked, supports relative paths (which can make certain cycles), and not every node child gets its own hash. This is too much flexibility for my purposes (Nix or otherwise). Also, when interfacing with legacy systems like git repos, one needs to dereference a legacy hash without knowing what it points to, which is easiest done with custom schemas. Now, granted, customs schemas aren't a super fine-grained solution as every node in the network that cares about the data needs to implement the schema, but they are useful tool for these reasons (and that downside doesn't apply to private networks).
anonbanker 10y ago
Someone should do something similar with Gentoo's portage, because the potential of IPFS could lead to amazing things, like verified pre-compiled -march=native builds for every architechture Gentoo supports.
- taktoa 10y ago
  For a while, I have interested in the idea of modifying the NixOS stdenv (standard build environment) to use a compiler that emits LLVM bitcode, and then having a function that takes any derivation to an equivalent derivation containing the result of running the LLVM IR through the specializer for your architecture. This would mean that you can share a binary cache with others, but still get `-march=native` performance. There's also some pretty interesting ideas along these lines wrt. randomly permuting instructions to prevent ROP attacks (you could even implement that as yet another package -> package function, so that you don't have to do the full set of LLVM optimizations for every package at boot time).

241 points by robto 10y ago

I'm really excited to see what the future holds for IPFS! However, hosting websites with custom domains is not quite feasible yet. Using IPFS' DNS (IPNS) means you have to keep the IPFS daemon running constantly, or else the files will be purged within an hour.

diggan 10y ago
> However, hosting websites with custom domains is not quite feasible yet Sure, point your domain to your IPFS hash and use dnslink, it's quite reliable already actually. That's how ipfs.io is hosted for example, and we haven't hit any issues so far. > Using IPFS' DNS (IPNS) means you have to keep the IPFS daemon running constantly, or else the files will be purged within an hour So, the files won't be purged, but the record you push out with IPNS won't be valid after 24 hours. You can solve this easily by using /ipfs/:hash instead of /ipns/:id and it wont disappear.
- citrusui 10y ago
  Yet again I failed to include important details... :p Yeah, it certainly is possible to host static sites on IPFS -- I have been testing it on my site[0] just for kicks. However, since I really enjoy using my domain name, rather than "ipfs.io/ip{f|n}s/$hash", I'm reluctant to try adapting my site to IPFS. I am aware that this is an alpha product, though, and I can't stress enough how cool it is for files containing the same data to be given the same ID (hash). That way you don't have to run shasum ever again :D [0]: https://ipfs.citrusui.me https://ipfs.citrusui.me (on a random note: what's up with /blog returning the IPFS blog, instead of my own content? did i misconfigure something?)
  - lgierth 10y ago
    The HTTP gateway included in go-ipfs will try to use a Host header on requests for constructing an /ipns path. So e.g. http://ipfs.io/docs http://ipfs.io/docs gets turned into /ipns/ipfs.io/docs -- this is actually how we host all webpages within the ipfs project. edit: And the part which turns /ipns/ipfs.io into an /ipfs path is called dnslink: https://github.com/ipfs/go-dnslink https://github.com/ipfs/go-dnslink -- it resolves to what's in TXT _dnslink.ipfs.io. > (on a random note: what's up with /blog returning the IPFS blog, instead of my own content? did i misconfigure something?) I'm so sorry, that's a bug I put into the nginx configuration -- will fix it!

vog 10y ago

Very interesting development. It would be great to see NixOS as an early adopter for IPFS. BTW, there is a small typo: IPFS is aims to create the distributed net. It should be: IPFS aims to create the distributed net.

sly010 10y ago
This is a great idea. A lot of businesses heavily rely on old versions of open source packages always being available. The one time someone deprecated an npm package, half of the nodejs stacks went with it. edit: Didn't mean to hit reply. Sorry.
- vog 10y ago
  Just delete your comment and put it on toplevel again.
  - sp332 10y ago
    You can't delete comments that have been replied to.
    - vog 10y ago
      You can, and the reply would remain. The tree then looks like this: comment1 [deleted] comment3 However, you can delete comments only until a certain amount of time. If you wait too long, you can't delete anymore (and this is what happened here).
      - sp332 10y ago
        That's how it used to work, but I'm pretty sure it doesn't work like that anymore.
        vog 10y ago
        I wonder if I'm the only one who is annoyed by the continuing lack of transparency at HN, wasting my time and other people's time with guesswork like this. I just took the time to start an "Ask HN" entry on that topic: "Ask HN: Where can I follow the changes of HN itself?" https://news.ycombinator.com/item?id=13460588 https://news.ycombinator.com/item?id=13460588

drdre2001 10y ago

This is a really great idea! Reminds me of other projects that are working on integrating IPFS with the Operating System: https://github.com/vtomole/IPOS https://github.com/vtomole/IPOS

chriswarbo 10y ago

I've been following these github issues for a while; fetching sources from IPFS seems like a great step forward for resiliency in general, and quite a natural one for Nix considering things are already immutable. Using IPFS as a binary cache is nice, as it would lower the maintainers' burden and make out-of-tree experimentation easier, i.e. without damaging the integrity of nixpkgs and cache.nixos.org. I hadn't even thought about using the FUSE integration of IPFS, but it makes a lot of sense. Nix is a lazy language, and the nixpkgs repository basically defines one big value: a set of name/value pairs for every package it contains (as well various libraries for e.g. working with Python packages, Haskell packages, etc.). The only difference between installed/uninstalled packages is whether anything's forced the contents to be evaluated yet. Likewise, an IPFS FUSE mount conceptually contains the whole of IPFS. The only difference between downloaded/undownloaded files is whether anything's forced the contents to be evaluated yet.

ris 10y ago
This article doesn't mention the most significant fallout of the IPFS idea (imo), which is that of .nar deduplication, as detailed in the issue https://github.com/NixOS/nix/issues/859 https://github.com/NixOS/nix/issues/859 (point 4). Perhaps a nail in the coffin of one of Nix's biggest absurdities.

cjbprime 10y ago

Very cool. One benefit of schemes like this that people don't talk about much is that, by no longer downloading from an expected place, you're removing the possibility for a compromised developer or server operator to selectively serve up malware to a targeted user. Instead you're getting the file over bittorrent and checking its hash, and you could gossip with other bittorrent clients to confirm that everyone's trying to get the same hash. Compare with the state of the art in most software updates, which is that you connect to some download server and it could serve signed malware to people on its target list and probably no-one would notice. (Schemes that use some of these techniques to take out the single point of malware-insertion have been called "Binary Transparency" schemes, as an analog to Certificate Transparency.)

nextos 10y ago
Guix has a very good complementary approach to this problem, guix challenge. Perhaps it's also implemented in Nix too already: https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-challenge.html https://www.gnu.org/software/guix/manual/html_node/Invoking-... Basically, since builds are reproducible, you can automatically build from source and see if the hash of the binary you built matches the one you are downloading. Obviously, source can be still compromised. But that's probably something IPFS won't fix unless wherever you get sources from is also on IPFS.
- grhmc 10y ago
  `nix-build` can take `--check` to do a similar thing. However, not all packages are reproducible. We've been doing a bit of work on this: https://garbas.si/2016/reproducible-builds-summit-in-berlin.html https://garbas.si/2016/reproducible-builds-summit-in-berlin.... and have begun checking reproducibility in our CI system: http://hydra.nixos.org/jobset/nixos/reproducibility http://hydra.nixos.org/jobset/nixos/reproducibility
- davexunit 10y ago
  Just a tiny nitpick, if I may: most builds are reproducible, but not all. Out of ~5000 packages, ~600 are not yet reproducible.

civodul 10y ago

Nice project! Guix had a GSoC student working on binary distribution using GNUnet's file sharing component a while back: https://gnu.org/s/guix/news/gsoc-update.html https://gnu.org/s/guix/news/gsoc-update.html . That has not led (yet?) to production code, but there might be ideas worth sharing.

k__ 10y ago

It's almost ridiculous how good the two fit together. I had the feeling NixOS has a bit of a hard time get users and prove that it's a superior solution to ansible/docker/chef/etc. probably because of it's mediocre UX, haha. But this would add another killer feature to it.

rkeene2 10y ago

Good to see other people are inventing AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) :-)

HurrdurrHodor 10y ago
Please don't use SHA-1. It's almost broken.
- lgierth 10y ago
  Use multihashes for hash algorithm agility :) https://github.com/multiformats/multihash https://github.com/multiformats/multihash
  - rkeene2 10y ago
    Multiple hashing algorithms is already built-in and mandatory, everywhere a hashing operation is used the hashing algorithm must also be specified.
- rkeene2 10y ago
  The protocol is actually extensible, and the hashing algorithm MUST always be specified by the server (which the client could then choose to not accept, just as it can reject the certificate because of the signature algorithm). Also, it would require a preimage attack against one of the hashed items to be useful which SHA-1 will likely be resistant to a long time (though decreasing with the number of items hashed) and SHA-1 is unlikely to be vulnerable to a preimage attack in the near future based on what we know so far. The signature and certificates that are used to validate the top-level index can be based on a far better hashing algorithm independently of the content-based hashing.
_prometheus 10y ago
> Good to see other people are inventing AppFS ( http://appfs.rkeene.org/ http://appfs.rkeene.org/ ) :-) I'll take the construction "other people are inventing <the thing i invented some time ago>" to mean that you think you came up with this first, or at least prior to Nix or IPFS. And thus ":-)" to be a bit sarcastic and unhappy, instead of genuinely happy. Similar times: * http://appfs.rkeene.org/web/timeline?c=78c60b0c9e7da1c9&unhide http://appfs.rkeene.org/web/timeline?c=78c60b0c9e7da1c9&unhi... * https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c19b7 https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1... * https://github.com/ipfs/ipfs/commit/8004db75262fcd29399d6c7fa979b70f20546884#diff-47ff1e80cf6b0d77a3e258a1e6288416R1102 https://github.com/ipfs/ipfs/commit/8004db75262fcd29399d6c7f... * https://github.com/jbenet/random-ideas/issues/19 https://github.com/jbenet/random-ideas/issues/19 * i think the Nix people probably came up with this way before either of us * and i am willing to bet everything that at least 100 other people (maybe thousands) have came up with this exact same idea over a decade before any of us... In fact, most of the best "original ideas" in the IPFS body of work were probably first discovered decades prior. I repeatedly see people succumbing to sadness over multiple discovery. It shouldn't be sad, it should be a happy event, as it confirms our own thoughts and presents an opportunity for collaborations. :) Further thoughts here: https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c19b7 https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1...

twoodfin 10y ago

I've felt for a while that a standard, widely-implemented, distributed content-addressable store is one of the biggest missing pieces of the modern internet. Glad to see any steps in that direction. I'll know real progress has been made when my browser can resolve something like: cas://sha256/2d66257e9d2cda5c08e850a947caacbc0177411f379f986dd9a8abc653ce5a8e

cjbprime 10y ago
See https://github.com/beakerbrowser/beaker https://github.com/beakerbrowser/beaker :)
deleted 10y ago
[deleted]
problems 10y ago
Bittorrent's DHT is used that way now, it's probably the biggest public DHT deployment in existence, all a magnet link is at it's core is magnet:?xt=urn:btih:<infohash>, so just having an infohash of a torrent is enough for you to get its content. Of course, there's also IPFS, Zeronet and Freenet which all address this exact issue in slightly different ways, all more web-targetted.
- djsumdog 10y ago
  I thought Zeronet used IPFS underneath?
  - problems 10y ago
    I believe it's an entirely separate network, not DHT based like IPFS though, but they've done a lot of cool work on the client side of things to allow user accounts and stuff.
jstanley 10y ago
Morphis is supposed to provide something like this: https://morph.is/ https://morph.is/

matthewbauer 10y ago

Stage 2 seems problematic at least the way I see it. Most users have at least a thousand derivations- is it possible to fuse mount each one? Also: I think some people are unaware that Nix hashes are not content addressable. The best solution (which OP is proposing) is probably to use the .nar hashes in IPFS which is content addressable.

Ericson2314 10y ago

I want to ditch the Nar format as soon as possible. IPFS's unixfs format is too rich however. When will the IPFS people finish up https://github.com/ipld/cid https://github.com/ipld/cid so we can link whatever content addressable data we want? I'd use git tree objects, despite SHA-1, because it's widely supported. Or do a format identical tree objects but with the IPFS's multihash and SHA-1 banned. Point is, underlying protocol should be agnostic to hashing scheme, we should have a trait/type class like /// Node in try trait Payload { type Hash: HashingTrait; fn unpack(Payload) -> (Vec<u8>, Set<Hash>); fn pack(Vec<u8>, Set<Hash>) -> Payload; // Implement either and get the other for free! fn hash_packed(p: Payload) -> Hash { hash_unpacked(packed(p)) fn hash_unpacked(p: (Vec<u8>, Set<Hash>)) -> Hash { hash_packed(packed(p)) } } any `(Hash, Payload)` than can define a `(binary blob, Set<Hash>) -> Hash` and Payload function should work.

whyrusleeping 10y ago
Hey! IPFS Dev here. The cid stuff has been implemented and initial support for it is being landed in our 0.4.5 release (which will be soon, hopefully release candidate within a week). With that and IPLD, you can craft arbitrary objects in JSON or CBOR (theres a 1 to 1 mapping, objects are stored as cbor) and work with them in ipfs. For example, i could make an object that looks like: { "Contents": {"/":"QmHashOfPkgContents"}, "Compression": "bzip2", "NarSize": 12345, "References": { "foo": {"/": "QmHashOfFoo"}, "bar": {"/": "QmHashOfBar"} }, "Signature": "signature info, or a link to the signature", } (please excuse my attempt at recreating a nar file in rough json). This object could then be put into ipfs with: cat thing.json | ipfs dag put And you would get an ipld object that you can move around in ipfs, and do fun things like: ipfs get <thatobjhash>/Contents to download the package contents, or: ipfs get <thatobjhash>/References/foo to get the referenced package (or open that hash/path in an ipfs gateway to browse the package graph for free in your browser :) )
- Ericson2314 10y ago
  IPLD does allow storing tons of data, but custom schemas allow restricting the data referenced in arbitrary ways. IPLD, last I checked, supports relative paths (which can make certain cycles), and not every node child gets its own hash. This is too much flexibility for my purposes (Nix or otherwise). Also, when interfacing with legacy systems like git repos, one needs to dereference a legacy hash without knowing what it points to, which is easiest done with custom schemas. Now, granted, customs schemas aren't a super fine-grained solution as every node in the network that cares about the data needs to implement the schema, but they are useful tool for these reasons (and that downside doesn't apply to private networks).

anonbanker 10y ago

Someone should do something similar with Gentoo's portage, because the potential of IPFS could lead to amazing things, like verified pre-compiled -march=native builds for every architechture Gentoo supports.

taktoa 10y ago
For a while, I have interested in the idea of modifying the NixOS stdenv (standard build environment) to use a compiler that emits LLVM bitcode, and then having a function that takes any derivation to an equivalent derivation containing the result of running the LLVM IR through the specializer for your architecture. This would mean that you can share a binary cache with others, but still get `-march=native` performance. There's also some pretty interesting ideas along these lines wrt. randomly permuting instructions to prevent ROP attacks (you could even implement that as yet another package -> package function, so that you don't have to do the full set of LLVM optimizations for every package at boot time).