3 ms·

It's a cool idea but falls down when you start thinking about how to discover what Instagram ID has what content ID in the P2P store. Torrents have trackers, IP

It's a cool idea but falls down when you start thinking about how to discover what Instagram ID has what content ID in the P2P store. Torrents have trackers, IPFS have 3rd party indexers, DC++ have hubs.

graderjs 5y ago
Not so. What if lookup for content is implemented as a distributed skiplist. Each node has a limited vertical slice of the skiplist. The skiplist info distributed among the nodes doesn't have to be consistent for it to work. When a new node enters the network it reaches out to peers, and randomly selects some peers to be 'in peers' and some to be 'out peers', gets their skiplist slices, and generates a slice that's between the others. If a node has content (ie, is a seeder), then it inserts those contents into the skiplist. The insertion changes propagate by peers exchanging info. It's a rough idea, but I think it would work or could be made to. Then to lookup a piece of content, you do a normal run on the skiplist, but each node your local slice is connected to is actually a peer. And sort of like DNS, you're connected to multiple inconsistent versions of the lookup paths, and you can run a lookup across more than such path. When data is found, it is sent back through the path.
- capableweb 5y ago
  You've basically reinvented DHTs but with the extra feature of supporting range queries, which this use case doesn't need anyway :) DHTs still have bunch of limitations that doesn't make it suitable for a really global and humongous hash table. I think your skiplist would have the different problem of making queries and lookups inconsistent based on node uptimes and such.
  - graderjs 5y ago
    Hehehe. I'm not an expert in this stuff, but thanks for your feedback! :p :) xx Inconsistent definitely. We could probably engineer our way out of this. ;p ;) xx If you have any recommendations for how we solve this p2p lookup problem I'd love to hear it! ;p xx

graderjs 5y ago

Not so. What if lookup for content is implemented as a distributed skiplist. Each node has a limited vertical slice of the skiplist. The skiplist info distributed among the nodes doesn't have to be consistent for it to work. When a new node enters the network it reaches out to peers, and randomly selects some peers to be 'in peers' and some to be 'out peers', gets their skiplist slices, and generates a slice that's between the others. If a node has content (ie, is a seeder), then it inserts those contents into the skiplist. The insertion changes propagate by peers exchanging info. It's a rough idea, but I think it would work or could be made to. Then to lookup a piece of content, you do a normal run on the skiplist, but each node your local slice is connected to is actually a peer. And sort of like DNS, you're connected to multiple inconsistent versions of the lookup paths, and you can run a lookup across more than such path. When data is found, it is sent back through the path.

capableweb 5y ago
You've basically reinvented DHTs but with the extra feature of supporting range queries, which this use case doesn't need anyway :) DHTs still have bunch of limitations that doesn't make it suitable for a really global and humongous hash table. I think your skiplist would have the different problem of making queries and lookups inconsistent based on node uptimes and such.
- graderjs 5y ago
  Hehehe. I'm not an expert in this stuff, but thanks for your feedback! :p :) xx Inconsistent definitely. We could probably engineer our way out of this. ;p ;) xx If you have any recommendations for how we solve this p2p lookup problem I'd love to hear it! ;p xx