4 ms·
Has anyone experimented with creating a search engine that only indexes the landing page of domains? I’m less interested in another Google, and more interested
by heywire 4y ago
Has anyone experimented with creating a search engine that only indexes the landing page of domains? I’m less interested in another Google, and more interested in a way to find new and interesting sites/blogs/etc. Stumbleupon was great for this back in the day.
Seems like it would be an interesting experiment to see what the results would be, indexing only the content / meta tags of “index.html”.
- kordlessagain 4y agoI built a solution at https://mitta.us/ https://mitta.us/ that lets you submit the sites you want crawled, and puts them in a self-managed index (which isn't shared globally). I don't do link extraction, but instead let GPT-3 generate URLs based off keywords. !url <keyterms> |synthesize I also wrote a screenshot extension for Chrome that lets you save a page when you find it interesting. The site is definitely not "done" but it's usable if you want to try it. Some info in help and in commands is inaccurate/broken, so it is what it is for now. It does the !google <search term> and !ddg <search term> thing to find pages to save to the index. There are a bunch of other commands I added, and there's an ability for others to write commands and submit them to a Github repo: https://github.com/kordless/mitta-community https://github.com/kordless/mitta-community !xkcd was fun to write. It shows comics. The rest of the commands can be viewed from !help or just !<tab> I've been working on pivoting the site to do prompt management for GPT-3 developers and have been kicking around Open Sourcing the other version for use as a personal search engine for bookmarked pages.