3 ms·

What IPFS lacks that I think would be ideal is a distributed file system. Let's say that Wikipedia shares some data, and I only have 100GB of space but want to

What IPFS lacks that I think would be ideal is a distributed file system. Let's say that Wikipedia shares some data, and I only have 100GB of space but want to help them out, then being able to dedicate 100GB of storage to help Wikipedia or whoever else would be awesome. Also, if I have a complete file or backup on multiple computers being able to immediately start sharing that across multiple servers without having to use like double the storage to also share it with IPFS.

mdaniel 4y ago
> ideal is a distributed file system Well, some of that is anti the way IPFS CID hashing works: every edit would result in a new CID for (e.g. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System) so your local pin of /ipfs/68106339799a6170efed62807e5c245d becomes orphaned when the next edit makes /ipfs/1ba72a5504f558ade3e784d9d349543c the most recent revision. I'm aware of /ipns but unless there was going to be an /ipns entry for every single page then it doesn't become as much of a distributed filesystem as you might think. Or, I guess put another way: it's really great for a distributed filesystem that doesn't change very often. I would expect archive.org would get more benefit from having IPFS seeds than anything on the Internet that is dynamically generated content I was actually sad that they don't already generate torrents for all their collections (or at least it didn't appear that they do)
- stoplying1 4y ago
  Er, it's not that hard. Think about git repos, they have refs that point to a tip that you traverse. This would work the same way. And in fact CID and chunking is what makes this possible... Ipfs is in fact ripe and perfect for some of these sorts of things and I highly suspect we're going to see more innovation in this space, in applications that are actually pushing the envelope, now that there's a sold rust impl
  - mdaniel 4y ago
    > Er, it's not that hard. You must be my boss; he loves to say that expression on zoom meetings where he's not responsible for the implementation > refs that point to a tip that you traverse Oh, ok, I guess you'll go to /ipns/wikipedia.org and just keep clicking links until you arrive at /wiki/InterPlanetary_File_System then. I think someone tried to prove the "6 degrees of Wikipedia," so yeah, how hard can it be? I'm ashamed I got baited into even replying to this
    - stoplying1 4y ago
      The discussion was a way of replicating or distributing a filesystem structures, there's absolutely no requirement anywhere that individual files in that FS are directly addressable via IPNS. And yes, it's simple because there are countless implementations of chunking deduping CID systems. Basically none of which have that artificially created requirement. MANY of which ARE ALREADY BUILT ON IPFS AND IPNS and work exactly the way I describe. Also, again, its like I mentioned it for a reason, see Git, where in fact, there isn't a nameable identity per-file, per-commit, and yet, there's a revisioned filesystem abstraction built on top! That works perfectly fine with pointers to CID content that points to other CID content. It's almost exactly what is being discussed at hand. Shame away buddy. Btw I've contributed code to more than one Content Addressable chunking deduping filesystem projects, because they're the future of file sync, and frankly now that Iroh is here, we're likely to start realizing these things in a "final" ish from. Oh and the other commenter saying the exact same thing as me ;).
- rklaehn 4y ago
  In theory, you would create an IPNS entry for the root of wikipedia, then update that on each change. Wikipedia is a relatively static data set, so on a day probably much less than 1% gets changed. So the new wikipedia tree reuses the vast majority of the previous version.
rklaehn 4y ago
iroh contributor here You can kind of handle this use case today. You can download a part of wikipedia, and then people should retrieve content from your node in addition to the official mirrors. The problem is that the performance of content discovery and content sync in ipfs leaves something to be desired. And also, probably due to this, there is also a lack of tooling for such use cases. The "shared wikipedia" use case is one I specifically would like to solve. Compared to some other things out there, it is a relatively small data set. And lots of people would be willing to help share wikipedia. So it makes for a great test case.

mdaniel 4y ago

> ideal is a distributed file system Well, some of that is anti the way IPFS CID hashing works: every edit would result in a new CID for (e.g. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System) so your local pin of /ipfs/68106339799a6170efed62807e5c245d becomes orphaned when the next edit makes /ipfs/1ba72a5504f558ade3e784d9d349543c the most recent revision. I'm aware of /ipns but unless there was going to be an /ipns entry for every single page then it doesn't become as much of a distributed filesystem as you might think. Or, I guess put another way: it's really great for a distributed filesystem that doesn't change very often. I would expect archive.org would get more benefit from having IPFS seeds than anything on the Internet that is dynamically generated content I was actually sad that they don't already generate torrents for all their collections (or at least it didn't appear that they do)

stoplying1 4y ago
Er, it's not that hard. Think about git repos, they have refs that point to a tip that you traverse. This would work the same way. And in fact CID and chunking is what makes this possible... Ipfs is in fact ripe and perfect for some of these sorts of things and I highly suspect we're going to see more innovation in this space, in applications that are actually pushing the envelope, now that there's a sold rust impl
- mdaniel 4y ago
  > Er, it's not that hard. You must be my boss; he loves to say that expression on zoom meetings where he's not responsible for the implementation > refs that point to a tip that you traverse Oh, ok, I guess you'll go to /ipns/wikipedia.org and just keep clicking links until you arrive at /wiki/InterPlanetary_File_System then. I think someone tried to prove the "6 degrees of Wikipedia," so yeah, how hard can it be? I'm ashamed I got baited into even replying to this
  - stoplying1 4y ago
    The discussion was a way of replicating or distributing a filesystem structures, there's absolutely no requirement anywhere that individual files in that FS are directly addressable via IPNS. And yes, it's simple because there are countless implementations of chunking deduping CID systems. Basically none of which have that artificially created requirement. MANY of which ARE ALREADY BUILT ON IPFS AND IPNS and work exactly the way I describe. Also, again, its like I mentioned it for a reason, see Git, where in fact, there isn't a nameable identity per-file, per-commit, and yet, there's a revisioned filesystem abstraction built on top! That works perfectly fine with pointers to CID content that points to other CID content. It's almost exactly what is being discussed at hand. Shame away buddy. Btw I've contributed code to more than one Content Addressable chunking deduping filesystem projects, because they're the future of file sync, and frankly now that Iroh is here, we're likely to start realizing these things in a "final" ish from. Oh and the other commenter saying the exact same thing as me ;).
rklaehn 4y ago
In theory, you would create an IPNS entry for the root of wikipedia, then update that on each change. Wikipedia is a relatively static data set, so on a day probably much less than 1% gets changed. So the new wikipedia tree reuses the vast majority of the previous version.

rklaehn 4y ago

iroh contributor here You can kind of handle this use case today. You can download a part of wikipedia, and then people should retrieve content from your node in addition to the official mirrors. The problem is that the performance of content discovery and content sync in ipfs leaves something to be desired. And also, probably due to this, there is also a lack of tooling for such use cases. The "shared wikipedia" use case is one I specifically would like to solve. Compared to some other things out there, it is a relatively small data set. And lots of people would be willing to help share wikipedia. So it makes for a great test case.