4 ms·

For a long time, I've been considering a different solution to the problem, which is to create a human-curated whitelist-only search engine. I do think your id

by smt88 4y ago

For a long time, I've been considering a different solution to the problem, which is to create a human-curated whitelist-only search engine. I do think your idea is great and something with a lot of value for many use cases. The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.

kbyatnal 4y ago
yup totally agree, but I think it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc). They have too much data and too much of a head start. The best way to compete with them IMO is to choose a slice of queries that they perform very poorly at (like non-SEO spam results) and build something 10x better. But that brings up the challenge of getting users to remember to come back to your search engine. It's tricky for sure, with no clear answer. But for the first time in a long time, I sense that we're approaching a tipping point for Google's dominance over the web.
- smt88 4y ago
  > it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc) Some of these things can be outsourced to other services. An easy Google fallback is also a common solution for new search engines. > But that brings up the challenge of getting users to remember to come back to your search engine. This wouldn't really be something I'd build as a business. I'd create a nonprofit, fund it with subscriptions or donations, and cover the losses myself. I really just want it to exist for myself, and if other people used it, that would be icing on the cake.
dchuk 4y ago
"a human-curated whitelist-only search engine" This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great. Said another way: moderate and protect the index, instead of trying to clean up the results of queries. You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit. You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic. You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.
- ZephyrBlu 4y ago
  > combined with the powerful indexing tools we have now What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
  - smt88 4y ago
    I'm not sure, but it sounds like they mean that building and quickly searching an index has been democratized. You can now use any number of FOSS projects or SaaS offerings to do that, depending on how big you want the index to be and how much you want to spend.
    - dchuk 4y ago
      That’s right. Back when google was coming to be, building a search index was basically computer science/grad school research territory. Now there’s tons of good starting points that you can throw money and hardware to scale quite a bit to at least prove out a product concept.
      - ZephyrBlu 4y ago
        Which starting points are those? I'm familiar with some open source search engines, but not open source solutions for crawling and updating a search index of the web.
- smt88 4y ago
  > You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. I actually think a browser plugin is a core component, especially if people could opt in to contributing their pages to the index (with a method to make sure they're not logged in to anything). The plugin would also allow people to flag websites as spam/scams/low-value, regardless of whether they're in the index or not.
nagonago 4y ago
I like this idea in theory, but I think it would be difficult to scale up enough to be useful. Who would be the curators? If you open up curation to volunteers, it could be gamed by bad actors. However if you have too small a team, the results will be limited and biased. For example, favoring the English-speaking or tech-sphere web while ignoring large sections of the web with which the curators are unfamiliar. Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. However that could end up getting gamed too.
- smt88 4y ago
  > Who would be the curators? That's a key question. You don't necessarily need that many of them, though, if you're whitelisting at the domain level. A few dozen people could work giving a rating to many thousands of domains every year. An ideal number might be in the thousands. I'd also say they probably shouldn't be in industry, because they'd have an incentive to game the index. > Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. While this is true, I don't think search engines other than Google and Bing need to worry that much about being gamed. It's just not worth the effort.
- larksimian 4y ago
  I think curation by paying users could work. At least it would make manipulation quite expensive. Let people downvote/upvote search results as an extra strong signal on top of monitoring what they choose to click on in the result list. Also let people block domains from searches and provide some discoverability of commonly blocked domains.
dylan604 4y ago
>which is to create a human-curated whitelist-only search engine. You might be interested in watching the series "Halt and Catch Fire".
- smt88 4y ago
  I'm not interested. Why would I be? I also don't need fiction to know what a human-curated search engine would be like, because I was using the web in the 90s when we had things like Dmoz.
  - dylan604 4y ago
    Well, you can be a curmudgeon and not enjoy someone's recommendation for a decent bit of entertainment, or you can say just politely carry on and be a grouch in the living room of your trashcan, or you could maybe try something new and actually consider watching something you haven't seen yet. edit: softened the tone oh so slightly
    - smt88 4y ago
      Sorry, I misinterpreted your suggestion as, "This is not a novel idea, and you should watch how badly it goes when someone on this TV show tries the same thing." I didn't insult you or your suggestion, though. I said I wasn't interested in the show and that it doesn't inform my understanding of the concept. Why did that hurt your feelings enough to make insulting accusations about my personality and character?
      - dylan604 4y ago
        "I'm not interested. Why would I be?" That just reads as a rude way to respond to me. "Why would I be?" Because you just described a plot line of a show. It might be fun to see how someone used that plot to make a show that someone reading HN might actually enjoy. Both sentences combined also reads as "I am perfectly content in my ways and will not listen to anything anyone might kindly suggest because I know all". Maybe because it's election season, but this unwilling to listen and stonewall is quite tiresome. So maybe just rubbed me the wrong way because of everything else and might not been taken the same way at a different time?

by smt88 4y ago

kbyatnal 4y ago

yup totally agree, but I think it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc). They have too much data and too much of a head start. The best way to compete with them IMO is to choose a slice of queries that they perform very poorly at (like non-SEO spam results) and build something 10x better. But that brings up the challenge of getting users to remember to come back to your search engine. It's tricky for sure, with no clear answer. But for the first time in a long time, I sense that we're approaching a tipping point for Google's dominance over the web.

smt88 4y ago
> it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc) Some of these things can be outsourced to other services. An easy Google fallback is also a common solution for new search engines. > But that brings up the challenge of getting users to remember to come back to your search engine. This wouldn't really be something I'd build as a business. I'd create a nonprofit, fund it with subscriptions or donations, and cover the losses myself. I really just want it to exist for myself, and if other people used it, that would be icing on the cake.

dchuk 4y ago

"a human-curated whitelist-only search engine" This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great. Said another way: moderate and protect the index, instead of trying to clean up the results of queries. You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit. You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic. You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.

ZephyrBlu 4y ago
> combined with the powerful indexing tools we have now What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
- smt88 4y ago
  I'm not sure, but it sounds like they mean that building and quickly searching an index has been democratized. You can now use any number of FOSS projects or SaaS offerings to do that, depending on how big you want the index to be and how much you want to spend.
  - dchuk 4y ago
    That’s right. Back when google was coming to be, building a search index was basically computer science/grad school research territory. Now there’s tons of good starting points that you can throw money and hardware to scale quite a bit to at least prove out a product concept.
    - ZephyrBlu 4y ago
      Which starting points are those? I'm familiar with some open source search engines, but not open source solutions for crawling and updating a search index of the web.
smt88 4y ago
> You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. I actually think a browser plugin is a core component, especially if people could opt in to contributing their pages to the index (with a method to make sure they're not logged in to anything). The plugin would also allow people to flag websites as spam/scams/low-value, regardless of whether they're in the index or not.

nagonago 4y ago

I like this idea in theory, but I think it would be difficult to scale up enough to be useful. Who would be the curators? If you open up curation to volunteers, it could be gamed by bad actors. However if you have too small a team, the results will be limited and biased. For example, favoring the English-speaking or tech-sphere web while ignoring large sections of the web with which the curators are unfamiliar. Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. However that could end up getting gamed too.

smt88 4y ago
> Who would be the curators? That's a key question. You don't necessarily need that many of them, though, if you're whitelisting at the domain level. A few dozen people could work giving a rating to many thousands of domains every year. An ideal number might be in the thousands. I'd also say they probably shouldn't be in industry, because they'd have an incentive to game the index. > Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. While this is true, I don't think search engines other than Google and Bing need to worry that much about being gamed. It's just not worth the effort.
larksimian 4y ago
I think curation by paying users could work. At least it would make manipulation quite expensive. Let people downvote/upvote search results as an extra strong signal on top of monitoring what they choose to click on in the result list. Also let people block domains from searches and provide some discoverability of commonly blocked domains.

dylan604 4y ago

>which is to create a human-curated whitelist-only search engine. You might be interested in watching the series "Halt and Catch Fire".

smt88 4y ago
I'm not interested. Why would I be? I also don't need fiction to know what a human-curated search engine would be like, because I was using the web in the 90s when we had things like Dmoz.
- dylan604 4y ago
  Well, you can be a curmudgeon and not enjoy someone's recommendation for a decent bit of entertainment, or you can say just politely carry on and be a grouch in the living room of your trashcan, or you could maybe try something new and actually consider watching something you haven't seen yet. edit: softened the tone oh so slightly
  - smt88 4y ago
    Sorry, I misinterpreted your suggestion as, "This is not a novel idea, and you should watch how badly it goes when someone on this TV show tries the same thing." I didn't insult you or your suggestion, though. I said I wasn't interested in the show and that it doesn't inform my understanding of the concept. Why did that hurt your feelings enough to make insulting accusations about my personality and character?
    - dylan604 4y ago
      "I'm not interested. Why would I be?" That just reads as a rude way to respond to me. "Why would I be?" Because you just described a plot line of a show. It might be fun to see how someone used that plot to make a show that someone reading HN might actually enjoy. Both sentences combined also reads as "I am perfectly content in my ways and will not listen to anything anyone might kindly suggest because I know all". Maybe because it's election season, but this unwilling to listen and stonewall is quite tiresome. So maybe just rubbed me the wrong way because of everything else and might not been taken the same way at a different time?