I've heard about CommonCrawl [0] a few times here and it seems interesting to think of building a search engine around it. Or, maybe, a boilerplate search engin

I've heard about CommonCrawl [0] a few times here and it seems interesting to think of building a search engine around it. Or, maybe, a boilerplate search engine that people could build upon. Maybe someone will even reply with an example :) I see they have a big list of projects [1] and honestly this might already be in there. It's fun to imagine a whole bunch of specialized search engines, like one that's great at finding product reviews, one for software development, one for recipes, and so on. I think the powerful aspect of a distributed/mirrored index is that the data is public, so anyone can build a UI or a product around it to offer their unique spin. From a more cynical angle, I also think it's likely that people still tend to gather around a few "popular" ones of anything like this. The web3 stuff seemed to do this. It happened with torrent trackers too. Eventually there's an option that is "base + X" where X is enticing enough to attract a crowd, and then once that crowd is big enough, that popular thing can pivot into a centralized thing, and the cycle repeats... 0: https://commoncrawl.org/ https://commoncrawl.org/ 1: https://commoncrawl.org/the-data/examples/ https://commoncrawl.org/the-data/examples/