4 ms·
Performance of adding to IPFS has improved in the past months, I recently added 3.7 TB of https://cdn.media.ccc.de https://cdn.media.ccc.de and it took ~24h, ad
by lgierth 10y ago
Performance of adding to IPFS has improved in the past months, I recently added 3.7 TB of https://cdn.media.ccc.de https://cdn.media.ccc.de and it took ~24h, admittedly these files are each pretty big, while Wikidata sounds like tons of tiny files.
One more major improvement is on the way, which is directory sharding using a HAMT data structure: https://github.com/ipfs/go-ipfs/pull/3042 https://github.com/ipfs/go-ipfs/pull/3042
- hobofan 10y agoYeah, it was most certainly the millions of tiny files that were killing the performance, since every publish with IPFS has quite some overhead. I'm quite excited for directory sharding since it would make the project simpler, but the local filesystem on my server doesn't even seem to be able to handle all the files in a single directory.