4 ms·
I think this highlights the insight that IPFS is more of a "decentralized transfer" tool rather than "decentralized storage" tool necessarily. Storage is part o
by imrehg 9y ago
I think this highlights the insight that IPFS is more of a "decentralized transfer" tool rather than "decentralized storage" tool necessarily. Storage is part of the caching, but the discovery of the required data pieces is the main advantage here too, as they mention that "With IPFS installed on each node, the traffic between the DataNodes
(slaves) and the NameNode (master) is reduced."
To me this understanding and shift in perception came after a few days of intense experimentation with IPFS, trying different use cases, etc. There are plenty of surprises if you just want to use IPFS as decentralized storage, as most introduction seems to approach it.
And it's definitely very cool & inspiring to find different use cases for IPFS :) For example the IPFS-backed docker registry is similarly cool https://github.com/jvassev/image2ipfs/ https://github.com/jvassev/image2ipfs/ (though unfortunately seems abandoned or just in deep hibernation...)
- XR0CSWV3h3kZWg 9y agoAre you interpreting the speed up as due to less contention for the NameNode's resources?
- imrehg 9y agoNot a hadoop expert, but that seems to be the case, more decentralized caching due to the way IPFS works.
- XR0CSWV3h3kZWg 9y agoIf that's the case you could also boost the replication # of a given folder/file in HDFS and see a similar speed up. There are circumstances where IPFS is nicer by default, but it might not be something that is a clear winner.
- imrehg 9y agoI think this is definitely a worthy experiment if you'd like to dig more into it. The poster (I believe) is very thin on details, so more eyes are better.
- XR0CSWV3h3kZWg 9y agoUnfortunately I don't see the connector code, or the experiment set up.