2 ms·
I agree, but sadly IPFS still needs some optimization for that to be feasible. I published the whole Wikidata dataset as separately accessible entities to IPFS
by hobofan 10y ago
I agree, but sadly IPFS still needs some optimization for that to be feasible.
I published the whole Wikidata dataset as separately accessible entities to IPFS and the initial publishing took ~2 weeks. Theoretically updating the dataset with weekly changes should be pretty fast after that, but if there is a change that impacts the JSON structure of every entity you have to start all over (which recently happened).
Right now Wikipedia/Wikidata is only really useful as a stress test for IPFS, but I'm optimistic for the future.
For anyone interested in this there is some (slow) progress on this at https://github.com/ipfs-wikidata https://github.com/ipfs-wikidata , though it's admittedly a low priority for me right now, as I wait for the technical state of IPFS to improve.
- lgierth 10y agoPerformance of adding to IPFS has improved in the past months, I recently added 3.7 TB of https://cdn.media.ccc.de https://cdn.media.ccc.de and it took ~24h, admittedly these files are each pretty big, while Wikidata sounds like tons of tiny files. One more major improvement is on the way, which is directory sharding using a HAMT data structure: https://github.com/ipfs/go-ipfs/pull/3042 https://github.com/ipfs/go-ipfs/pull/3042
- hobofan 10y agoYeah, it was most certainly the millions of tiny files that were killing the performance, since every publish with IPFS has quite some overhead. I'm quite excited for directory sharding since it would make the project simpler, but the local filesystem on my server doesn't even seem to be able to handle all the files in a single directory.