4 ms·
"a human-curated whitelist-only search engine" This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful ind
by dchuk 4y ago
"a human-curated whitelist-only search engine"
This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great.
Said another way: moderate and protect the index, instead of trying to clean up the results of queries.
You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit.
You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic.
You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.
- ZephyrBlu 4y ago> combined with the powerful indexing tools we have now What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
- smt88 4y agoI'm not sure, but it sounds like they mean that building and quickly searching an index has been democratized. You can now use any number of FOSS projects or SaaS offerings to do that, depending on how big you want the index to be and how much you want to spend.
- dchuk 4y agoThat’s right. Back when google was coming to be, building a search index was basically computer science/grad school research territory. Now there’s tons of good starting points that you can throw money and hardware to scale quite a bit to at least prove out a product concept.
- ZephyrBlu 4y agoWhich starting points are those? I'm familiar with some open source search engines, but not open source solutions for crawling and updating a search index of the web.
- smt88 4y ago> You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. I actually think a browser plugin is a core component, especially if people could opt in to contributing their pages to the index (with a method to make sure they're not logged in to anything). The plugin would also allow people to flag websites as spam/scams/low-value, regardless of whether they're in the index or not.