3 ms·

We'd be better off by putting wikipedia on IPFS and IPFS into p2p cache.

by pmlnr 10y ago

cabalamat 10y ago
This could be a useful application to run on IPFS, which IMO needs a killer app to become widespread.
goodplay 10y ago
If it's execution is similar to this project where hosting a page on your system is effortless (or at most a one-click affair), then it's definitely something I can get behind. You'll get the benefit of insuring off-line availability as well as only hosting pages you care about. Win-win.
hobofan 10y ago
I agree, but sadly IPFS still needs some optimization for that to be feasible. I published the whole Wikidata dataset as separately accessible entities to IPFS and the initial publishing took ~2 weeks. Theoretically updating the dataset with weekly changes should be pretty fast after that, but if there is a change that impacts the JSON structure of every entity you have to start all over (which recently happened). Right now Wikipedia/Wikidata is only really useful as a stress test for IPFS, but I'm optimistic for the future. For anyone interested in this there is some (slow) progress on this at https://github.com/ipfs-wikidata https://github.com/ipfs-wikidata , though it's admittedly a low priority for me right now, as I wait for the technical state of IPFS to improve.
- lgierth 10y ago
  Performance of adding to IPFS has improved in the past months, I recently added 3.7 TB of https://cdn.media.ccc.de https://cdn.media.ccc.de and it took ~24h, admittedly these files are each pretty big, while Wikidata sounds like tons of tiny files. One more major improvement is on the way, which is directory sharding using a HAMT data structure: https://github.com/ipfs/go-ipfs/pull/3042 https://github.com/ipfs/go-ipfs/pull/3042
  - hobofan 10y ago
    Yeah, it was most certainly the millions of tiny files that were killing the performance, since every publish with IPFS has quite some overhead. I'm quite excited for directory sharding since it would make the project simpler, but the local filesystem on my server doesn't even seem to be able to handle all the files in a single directory.
tscs37 10y ago
The problem is that it requires installing third-party software to function (the IPFS client) or putting load on some IPFS gateway.
- diggan 10y ago
  Not necessary. Either Wikipedia can help out with hosting their own gateways in addition to the ones running on ipfs.io, OR they can include a js-ipfs node in the browser.
  - tscs37 10y ago
    >Wikipedia can help out with hosting their own gateways Which kinda defeats the point of saving bandwidth on their side. >js-ipfs node That could work, though I was under the impressino that the js-ipfs implementation was WIP/Experimental (last I checked)
    - diggan 10y ago
      > Which kinda defeats the point of saving bandwidth on their side. Traffic doesn't only need to be served by Wikipedia but could also include other volunteers (think archive.org and others) and the setup for helping out would be easy. > That could work, though I was under the impressino that the js-ipfs implementation was WIP/Experimental (last I checked) That is absolutely true, which WikipediaP2P is as well, and IPFS for that matter. I was simply pointing out some ways you can make the whole "Wikipedia on IPFS" thing to work in the future.
- pmlnr 10y ago
  The original idea requires a 3rd party addon and Chrome. How is this different?
  - tscs37 10y ago
    Wikipedia on IPFS as an addon would also require operating a node to fulfill the goal of the original project, namely to save bandwidth for Wikipedia. Simply pointing it at gateway.ipfs.io would achieve nothing but shift the problem around.

We'd be better off by putting wikipedia on IPFS and IPFS into p2p cache.

by pmlnr 10y ago

We'd be better off by putting wikipedia on IPFS and IPFS into p2p cache.

cabalamat 10y ago

This could be a useful application to run on IPFS, which IMO needs a killer app to become widespread.

goodplay 10y ago

If it's execution is similar to this project where hosting a page on your system is effortless (or at most a one-click affair), then it's definitely something I can get behind. You'll get the benefit of insuring off-line availability as well as only hosting pages you care about. Win-win.

hobofan 10y ago

I agree, but sadly IPFS still needs some optimization for that to be feasible. I published the whole Wikidata dataset as separately accessible entities to IPFS and the initial publishing took ~2 weeks. Theoretically updating the dataset with weekly changes should be pretty fast after that, but if there is a change that impacts the JSON structure of every entity you have to start all over (which recently happened). Right now Wikipedia/Wikidata is only really useful as a stress test for IPFS, but I'm optimistic for the future. For anyone interested in this there is some (slow) progress on this at https://github.com/ipfs-wikidata https://github.com/ipfs-wikidata , though it's admittedly a low priority for me right now, as I wait for the technical state of IPFS to improve.

lgierth 10y ago
Performance of adding to IPFS has improved in the past months, I recently added 3.7 TB of https://cdn.media.ccc.de https://cdn.media.ccc.de and it took ~24h, admittedly these files are each pretty big, while Wikidata sounds like tons of tiny files. One more major improvement is on the way, which is directory sharding using a HAMT data structure: https://github.com/ipfs/go-ipfs/pull/3042 https://github.com/ipfs/go-ipfs/pull/3042
- hobofan 10y ago
  Yeah, it was most certainly the millions of tiny files that were killing the performance, since every publish with IPFS has quite some overhead. I'm quite excited for directory sharding since it would make the project simpler, but the local filesystem on my server doesn't even seem to be able to handle all the files in a single directory.

tscs37 10y ago

The problem is that it requires installing third-party software to function (the IPFS client) or putting load on some IPFS gateway.

diggan 10y ago
Not necessary. Either Wikipedia can help out with hosting their own gateways in addition to the ones running on ipfs.io, OR they can include a js-ipfs node in the browser.
- tscs37 10y ago
  >Wikipedia can help out with hosting their own gateways Which kinda defeats the point of saving bandwidth on their side. >js-ipfs node That could work, though I was under the impressino that the js-ipfs implementation was WIP/Experimental (last I checked)
  - diggan 10y ago
    > Which kinda defeats the point of saving bandwidth on their side. Traffic doesn't only need to be served by Wikipedia but could also include other volunteers (think archive.org and others) and the setup for helping out would be easy. > That could work, though I was under the impressino that the js-ipfs implementation was WIP/Experimental (last I checked) That is absolutely true, which WikipediaP2P is as well, and IPFS for that matter. I was simply pointing out some ways you can make the whole "Wikipedia on IPFS" thing to work in the future.
pmlnr 10y ago
The original idea requires a 3rd party addon and Chrome. How is this different?
- tscs37 10y ago
  Wikipedia on IPFS as an addon would also require operating a node to fulfill the goal of the original project, namely to save bandwidth for Wikipedia. Simply pointing it at gateway.ipfs.io would achieve nothing but shift the problem around.