3 ms·

Is the performance a critical design flaw or just an implementation issue?

rklaehn 3y ago
Difficult to answer. IPNS uses the IPFS kademlia DHT, which has some performance problems that you can argue are fundamental. For solving a similar problem with iroh, we would use the bittorrent mainline DHT, which is the largest DHT in existence and has stood the test of time - it still exists despite lots of powerful entities wanting it to go away. It also generally has very good performance and a very minimalist design. There is a rust crate to interact with the mainline DHT, https://crates.io/crates/mainline https://crates.io/crates/mainline , and a more high level idea to use DHTs as a kind of p2p DNS, https://github.com/nuhvi/pkarr https://github.com/nuhvi/pkarr
anacrolix 3y ago
It's a design flaw.
wscott 3y ago
Design flaw. In IPFS every piece of data (even every chuck of large files) is globally indexable on the same namespace. You need namespaces and/or a path to restrict yourself to just the subset of peers that might actually have the data. It would be possible to add a layer on top of IPFS to include some context with every hash lookup so the search can be more focused, but the original design just assumed it was ok to do a log2 search for every chunk.
- rklaehn 3y ago
  That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish). Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic. If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed. In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot. You still get verified range access.
tadfisher 3y ago
Imagine if DNS supplied every URL and not just domain names. You need some mechanism to propagate resource changes. IPNS has two practical mechanisms: a global DHT that takes time to propagate, and a pub/sub that requires peers to actively subscribe to your changes.
- anacrolix 3y ago
  btlink does DNS per domain name, which you could argue is a sweet spot between too many queries and being too broad. at least in the case of the web, it works nicely.

Difficult to answer. IPNS uses the IPFS kademlia DHT, which has some performance problems that you can argue are fundamental. For solving a similar problem with iroh, we would use the bittorrent mainline DHT, which is the largest DHT in existence and has stood the test of time - it still exists despite lots of powerful entities wanting it to go away. It also generally has very good performance and a very minimalist design. There is a rust crate to interact with the mainline DHT, https://crates.io/crates/mainline https://crates.io/crates/mainline , and a more high level idea to use DHTs as a kind of p2p DNS, https://github.com/nuhvi/pkarr https://github.com/nuhvi/pkarr

anacrolix 3y ago

It's a design flaw.

wscott 3y ago

Design flaw. In IPFS every piece of data (even every chuck of large files) is globally indexable on the same namespace. You need namespaces and/or a path to restrict yourself to just the subset of peers that might actually have the data. It would be possible to add a layer on top of IPFS to include some context with every hash lookup so the search can be more focused, but the original design just assumed it was ok to do a log2 search for every chunk.

rklaehn 3y ago
That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish). Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic. If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed. In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot. You still get verified range access.

tadfisher 3y ago

Imagine if DNS supplied every URL and not just domain names. You need some mechanism to propagate resource changes. IPNS has two practical mechanisms: a global DHT that takes time to propagate, and a pub/sub that requires peers to actively subscribe to your changes.

anacrolix 3y ago
btlink does DNS per domain name, which you could argue is a sweet spot between too many queries and being too broad. at least in the case of the web, it works nicely.