5 ms·

Ask HN: What are some examples of “decentralized discoverability?”

35 points by mijustin 4y ago

While the web itself is built on a series of decentralized protocols (TCP/IP, HTTP, DNS), discoverability is still centralized on platforms like Google Search, Twitter, YouTube, etc... Even web3 stuff relies on centralized indexes/marketplaces like OpenSea. The idea of mirroring an index (Pirate Bay), or having a distributed database (DNS), is fascinating to me. These are the examples I can think of: DNS UseNet FidoNet PirateBay Mastadon Any others? What could "decentralized discoverability" look like for content?

deleted 4y ago
[deleted]
dceddia 4y ago
I've heard about CommonCrawl [0] a few times here and it seems interesting to think of building a search engine around it. Or, maybe, a boilerplate search engine that people could build upon. Maybe someone will even reply with an example :) I see they have a big list of projects [1] and honestly this might already be in there. It's fun to imagine a whole bunch of specialized search engines, like one that's great at finding product reviews, one for software development, one for recipes, and so on. I think the powerful aspect of a distributed/mirrored index is that the data is public, so anyone can build a UI or a product around it to offer their unique spin. From a more cynical angle, I also think it's likely that people still tend to gather around a few "popular" ones of anything like this. The web3 stuff seemed to do this. It happened with torrent trackers too. Eventually there's an option that is "base + X" where X is enticing enough to attract a crowd, and then once that crowd is big enough, that popular thing can pivot into a centralized thing, and the cycle repeats... 0: https://commoncrawl.org/ https://commoncrawl.org/ 1: https://commoncrawl.org/the-data/examples/ https://commoncrawl.org/the-data/examples/
- mijustin 4y ago
  The difference with torrent trackers is their index is public and can be mirrored. Torrent sites like 1337x were born out of previously popular indexes (Kickass Torrents, h33t). [0] It's not dissimilar to when a GitHub fork becomes more popular than the original project. [1] To me, the biggest disappointment with web3 is they never cracked "distributed discoverability." It still relied on the old model, of proprietary indexes/marketplaces. I'm way more inspired by the P2P efforts, especially Gnutella's initial attempts at distributed search. [2] Feels like this tech could still be iterated on and improved. 0: https://en.wikipedia.org/wiki/1337x https://en.wikipedia.org/wiki/1337x 1: http://gitpop2.herokuapp.com/tobi/delayed_job http://gitpop2.herokuapp.com/tobi/delayed_job 2: http://rfc-gnutella.sourceforge.net/developer/share/intro.html http://rfc-gnutella.sourceforge.net/developer/share/intro.ht...
robcohen 4y ago
OpenSea was popularized because the NFT minting functionality cost nothing for the artist. So all costs for the blockchain part was passed to the consumer. I believe this was the largest reason it became popular. Blockchain still doesn’t scale well, layer 2 is getting pretty mature though and with Eth Merge happening hopefully things will get significantly better in the next few years.
- mijustin 4y ago
  The problem with web3 is that the projects still rely on centralized indexes/distribution (OpenSea, Rarible, etc) for discoverability. "We have had decentralised ownership and distribution of our own content since the early 90s. I own my own domain, but that doesn’t get me traffic. The question is in the networks that sort, filter and organise that, and those aren’t our tokens that we can move." – Benedict Evans [0] web3 can't deliver on its promise of "freedom from the big platforms" as long as we're beholden to big platforms (Google, OpenSea, Twitter, YouTube, etc) for distribution/discoverability. The web3 world still hasn't figured out how to decentralize discoverability. 0: https://twitter.com/benedictevans/status/1476285185557774337 https://twitter.com/benedictevans/status/1476285185557774337
msencenb 4y ago
This reminds me a little of the cuban offline network (https://www.fastcompany.com/3048163/in-cuba-an-underground-network-armed-with-usb-drives-does-the-work-of-google-and-youtube https://www.fastcompany.com/3048163/in-cuba-an-underground-n...)
- mijustin 4y ago
  This is definitely in the same spirit. What’s missing is a truly decentralized portal for discovering that content. ;)
rakoo 4y ago
Using bittorrent in a really decentralized way: https://github.com/boramalper/magnetico/ https://github.com/boramalper/magnetico/
ratorx 4y ago
BitTorrent had a protocol extension for DHTs, as a replacement for the trackers that are required by default: https://www.bittorrent.org/beps/bep_0005.html https://www.bittorrent.org/beps/bep_0005.html So you still need a central web server to serve the .torrent file, but peers are found without centralisation. But I’m sure with effort that could have been generalised further. There’s also IPFS (https://ipfs.tech https://ipfs.tech), which uses content addressability to try to be even more decentralised than the normal web (without any cryptocurrency shenanigans, at least last I checked). I don’t think either took off massively though.
- anacrolix 4y ago
  Most BitTorrent clients support the ability to transfer the torrent info over the wire too: central trackers are not necessary at all.
- anacrolix 4y ago
  I think IPFS development is funded significantly by Filecoin, a cryptocurrency.
  - aaaaaaaaata 4y ago
    Heaven forbid.
beecafe 4y ago
https://en.m.wikipedia.org/wiki/Kademlia https://en.m.wikipedia.org/wiki/Kademlia
h0p3 4y ago
You can use links, webrings, and word of mouth. You can crawl (for non-web this may require quite specialized tooling), scrape, and compile a database for search (which can be done locally using whatever tools you prefer or even a full-blown webserver like gigablast or YaCy); and, to some degree, these practices themselves can be further decentralized and shared. Scaling is hard, especially if users expect immediate results or there are adversarial nodes to contend with. Ideally, we could programmatically share feeds of curated and tagged information with each other that can be nested, mixed, or further shaped. Leveraging our trust in others (and their trust in others by proxy), which can be similar to some problems in distributed governance, is a p2p holy grail, imho.
- openfuture 4y ago
  In order to start doing this we need a standard to build on. The simplest thing should be the easiest thing to agree on and that has been the case so far in how math has moved forward (afaict) but sometimes local optimas get in the way of progress.
  - h0p3 4y ago
    `/nod`. I hear ya. RSS and mutable torrents seem like a fine starting place (if only to learn from), and I think we could argue ActivityPub's community has attempted to work on this problem. I worry that if we can't achieve the underlying distributed network (and effective DHTs [or the overlays on them] have a very long way to go, imho), we will lack the proper testing grounds to even know what the fitting standard(s) should be at some level. I hope I'm wrong about that. I could see federated networks perfecting some aspects of self-sovereign, decentralizable identities or portable logical accounts, as well as fitting standards for sharing the data we find most salient among that federation. It's a start. I can't say something like Mastadon has succeeded here, but these amazing projects simply haven't received the funding they deserve. I could also imagine a centralized, for-profit institution creating something similar, but it would be a trade secret. I hope we find the answer before they do. I'm not going to work on that problem until I have an underlying p2p network meant to actually replace at least the web (and, I am, of course, a huge fan of Sir pfrazee's astounding work in this sphere). I consider that an extremely tall task as it is, and, so far (and I say this with profound [[gratitude]] for those who have worked to solve the problem), nothing comes even close. It's a pleasure to meet you, btw.

35 points by mijustin 4y ago

deleted 4y ago

[deleted]

dceddia 4y ago

I've heard about CommonCrawl [0] a few times here and it seems interesting to think of building a search engine around it. Or, maybe, a boilerplate search engine that people could build upon. Maybe someone will even reply with an example :) I see they have a big list of projects [1] and honestly this might already be in there. It's fun to imagine a whole bunch of specialized search engines, like one that's great at finding product reviews, one for software development, one for recipes, and so on. I think the powerful aspect of a distributed/mirrored index is that the data is public, so anyone can build a UI or a product around it to offer their unique spin. From a more cynical angle, I also think it's likely that people still tend to gather around a few "popular" ones of anything like this. The web3 stuff seemed to do this. It happened with torrent trackers too. Eventually there's an option that is "base + X" where X is enticing enough to attract a crowd, and then once that crowd is big enough, that popular thing can pivot into a centralized thing, and the cycle repeats... 0: https://commoncrawl.org/ https://commoncrawl.org/ 1: https://commoncrawl.org/the-data/examples/ https://commoncrawl.org/the-data/examples/

mijustin 4y ago
The difference with torrent trackers is their index is public and can be mirrored. Torrent sites like 1337x were born out of previously popular indexes (Kickass Torrents, h33t). [0] It's not dissimilar to when a GitHub fork becomes more popular than the original project. [1] To me, the biggest disappointment with web3 is they never cracked "distributed discoverability." It still relied on the old model, of proprietary indexes/marketplaces. I'm way more inspired by the P2P efforts, especially Gnutella's initial attempts at distributed search. [2] Feels like this tech could still be iterated on and improved. 0: https://en.wikipedia.org/wiki/1337x https://en.wikipedia.org/wiki/1337x 1: http://gitpop2.herokuapp.com/tobi/delayed_job http://gitpop2.herokuapp.com/tobi/delayed_job 2: http://rfc-gnutella.sourceforge.net/developer/share/intro.html http://rfc-gnutella.sourceforge.net/developer/share/intro.ht...

robcohen 4y ago

OpenSea was popularized because the NFT minting functionality cost nothing for the artist. So all costs for the blockchain part was passed to the consumer. I believe this was the largest reason it became popular. Blockchain still doesn’t scale well, layer 2 is getting pretty mature though and with Eth Merge happening hopefully things will get significantly better in the next few years.

mijustin 4y ago
The problem with web3 is that the projects still rely on centralized indexes/distribution (OpenSea, Rarible, etc) for discoverability. "We have had decentralised ownership and distribution of our own content since the early 90s. I own my own domain, but that doesn’t get me traffic. The question is in the networks that sort, filter and organise that, and those aren’t our tokens that we can move." – Benedict Evans [0] web3 can't deliver on its promise of "freedom from the big platforms" as long as we're beholden to big platforms (Google, OpenSea, Twitter, YouTube, etc) for distribution/discoverability. The web3 world still hasn't figured out how to decentralize discoverability. 0: https://twitter.com/benedictevans/status/1476285185557774337 https://twitter.com/benedictevans/status/1476285185557774337

msencenb 4y ago

This reminds me a little of the cuban offline network (https://www.fastcompany.com/3048163/in-cuba-an-underground-network-armed-with-usb-drives-does-the-work-of-google-and-youtube https://www.fastcompany.com/3048163/in-cuba-an-underground-n...)

mijustin 4y ago
This is definitely in the same spirit. What’s missing is a truly decentralized portal for discovering that content. ;)

rakoo 4y ago

Using bittorrent in a really decentralized way: https://github.com/boramalper/magnetico/ https://github.com/boramalper/magnetico/

ratorx 4y ago

BitTorrent had a protocol extension for DHTs, as a replacement for the trackers that are required by default: https://www.bittorrent.org/beps/bep_0005.html https://www.bittorrent.org/beps/bep_0005.html So you still need a central web server to serve the .torrent file, but peers are found without centralisation. But I’m sure with effort that could have been generalised further. There’s also IPFS (https://ipfs.tech https://ipfs.tech), which uses content addressability to try to be even more decentralised than the normal web (without any cryptocurrency shenanigans, at least last I checked). I don’t think either took off massively though.

anacrolix 4y ago
Most BitTorrent clients support the ability to transfer the torrent info over the wire too: central trackers are not necessary at all.
anacrolix 4y ago
I think IPFS development is funded significantly by Filecoin, a cryptocurrency.
- aaaaaaaaata 4y ago
  Heaven forbid.

beecafe 4y ago

https://en.m.wikipedia.org/wiki/Kademlia https://en.m.wikipedia.org/wiki/Kademlia

h0p3 4y ago

You can use links, webrings, and word of mouth. You can crawl (for non-web this may require quite specialized tooling), scrape, and compile a database for search (which can be done locally using whatever tools you prefer or even a full-blown webserver like gigablast or YaCy); and, to some degree, these practices themselves can be further decentralized and shared. Scaling is hard, especially if users expect immediate results or there are adversarial nodes to contend with. Ideally, we could programmatically share feeds of curated and tagged information with each other that can be nested, mixed, or further shaped. Leveraging our trust in others (and their trust in others by proxy), which can be similar to some problems in distributed governance, is a p2p holy grail, imho.

openfuture 4y ago
In order to start doing this we need a standard to build on. The simplest thing should be the easiest thing to agree on and that has been the case so far in how math has moved forward (afaict) but sometimes local optimas get in the way of progress.
- h0p3 4y ago
  `/nod`. I hear ya. RSS and mutable torrents seem like a fine starting place (if only to learn from), and I think we could argue ActivityPub's community has attempted to work on this problem. I worry that if we can't achieve the underlying distributed network (and effective DHTs [or the overlays on them] have a very long way to go, imho), we will lack the proper testing grounds to even know what the fitting standard(s) should be at some level. I hope I'm wrong about that. I could see federated networks perfecting some aspects of self-sovereign, decentralizable identities or portable logical accounts, as well as fitting standards for sharing the data we find most salient among that federation. It's a start. I can't say something like Mastadon has succeeded here, but these amazing projects simply haven't received the funding they deserve. I could also imagine a centralized, for-profit institution creating something similar, but it would be a trade secret. I hope we find the answer before they do. I'm not going to work on that problem until I have an underlying p2p network meant to actually replace at least the web (and, I am, of course, a huge fan of Sir pfrazee's astounding work in this sphere). I consider that an extremely tall task as it is, and, so far (and I say this with profound [[gratitude]] for those who have worked to solve the problem), nothing comes even close. It's a pleasure to meet you, btw.