7 ms·

IMO this article misses the biggest issue with search engines today. It's less about any ranking algorithm (like PageRank), but rather the root cause is with in

IMO this article misses the biggest issue with search engines today. It's less about any ranking algorithm (like PageRank), but rather the root cause is with indexing. No matter how much you fine tune your ranking system, if your index is filled with SEO junk and blogspam, you're fighting a losing battle from the start. For some reason, a lot of these search engines like to brag about the number of documents in their index, which never made sense to me. Maybe it was true in the past, but on the modern web, larger index !== better results. In fact, I'd argue the opposite since you're much more likely to serve SEO spam. That was my motivation to start hacking around on CrowdView (https://crew-rho.vercel.app https://crew-rho.vercel.app), a search engine specifically for forums and discussion content (e.g. forums, discords, twitter, reddit, etc). It has a curated index (today, curated by yours truly) to remove SEO junk and help you figure out "what does a real, genuine human think about this think?"

_aavaa_ 4y ago
On Kagi you as an individual can up rank and down rank websites (even hiding them entirely). They can conceivably use this information to do the same thing, curare the results and change their ordering
smt88 4y ago
For a long time, I've been considering a different solution to the problem, which is to create a human-curated whitelist-only search engine. I do think your idea is great and something with a lot of value for many use cases. The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.
- kbyatnal 4y ago
  yup totally agree, but I think it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc). They have too much data and too much of a head start. The best way to compete with them IMO is to choose a slice of queries that they perform very poorly at (like non-SEO spam results) and build something 10x better. But that brings up the challenge of getting users to remember to come back to your search engine. It's tricky for sure, with no clear answer. But for the first time in a long time, I sense that we're approaching a tipping point for Google's dominance over the web.
  - smt88 4y ago
    > it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc) Some of these things can be outsourced to other services. An easy Google fallback is also a common solution for new search engines. > But that brings up the challenge of getting users to remember to come back to your search engine. This wouldn't really be something I'd build as a business. I'd create a nonprofit, fund it with subscriptions or donations, and cover the losses myself. I really just want it to exist for myself, and if other people used it, that would be icing on the cake.
- dchuk 4y ago
  "a human-curated whitelist-only search engine" This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great. Said another way: moderate and protect the index, instead of trying to clean up the results of queries. You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit. You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic. You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.
  - ZephyrBlu 4y ago
    > combined with the powerful indexing tools we have now What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
    - smt88 4y ago
      I'm not sure, but it sounds like they mean that building and quickly searching an index has been democratized. You can now use any number of FOSS projects or SaaS offerings to do that, depending on how big you want the index to be and how much you want to spend.
      - dchuk 4y ago
        That’s right. Back when google was coming to be, building a search index was basically computer science/grad school research territory. Now there’s tons of good starting points that you can throw money and hardware to scale quite a bit to at least prove out a product concept.
- nagonago 4y ago
  I like this idea in theory, but I think it would be difficult to scale up enough to be useful. Who would be the curators? If you open up curation to volunteers, it could be gamed by bad actors. However if you have too small a team, the results will be limited and biased. For example, favoring the English-speaking or tech-sphere web while ignoring large sections of the web with which the curators are unfamiliar. Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. However that could end up getting gamed too.
  - smt88 4y ago
    > Who would be the curators? That's a key question. You don't necessarily need that many of them, though, if you're whitelisting at the domain level. A few dozen people could work giving a rating to many thousands of domains every year. An ideal number might be in the thousands. I'd also say they probably shouldn't be in industry, because they'd have an incentive to game the index. > Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. While this is true, I don't think search engines other than Google and Bing need to worry that much about being gamed. It's just not worth the effort.
  - larksimian 4y ago
    I think curation by paying users could work. At least it would make manipulation quite expensive. Let people downvote/upvote search results as an extra strong signal on top of monitoring what they choose to click on in the result list. Also let people block domains from searches and provide some discoverability of commonly blocked domains.
- dylan604 4y ago
  >which is to create a human-curated whitelist-only search engine. You might be interested in watching the series "Halt and Catch Fire".
  - smt88 4y ago
    I'm not interested. Why would I be? I also don't need fiction to know what a human-curated search engine would be like, because I was using the web in the 90s when we had things like Dmoz.
    - dylan604 4y ago
      Well, you can be a curmudgeon and not enjoy someone's recommendation for a decent bit of entertainment, or you can say just politely carry on and be a grouch in the living room of your trashcan, or you could maybe try something new and actually consider watching something you haven't seen yet. edit: softened the tone oh so slightly
      - smt88 4y ago
        Sorry, I misinterpreted your suggestion as, "This is not a novel idea, and you should watch how badly it goes when someone on this TV show tries the same thing." I didn't insult you or your suggestion, though. I said I wasn't interested in the show and that it doesn't inform my understanding of the concept. Why did that hurt your feelings enough to make insulting accusations about my personality and character?
        dylan604 4y ago
        "I'm not interested. Why would I be?" That just reads as a rude way to respond to me. "Why would I be?" Because you just described a plot line of a show. It might be fun to see how someone used that plot to make a show that someone reading HN might actually enjoy. Both sentences combined also reads as "I am perfectly content in my ways and will not listen to anything anyone might kindly suggest because I know all". Maybe because it's election season, but this unwilling to listen and stonewall is quite tiresome. So maybe just rubbed me the wrong way because of everything else and might not been taken the same way at a different time?
albatrosstrophy 4y ago
I like your idea of the search engine. It's what I miss from Google. At one time (2006?) we had the option to google search only forums and discussions, and that helped me get quality answers to my queries.
Semaphor 4y ago
Kagi has a feature, lenses, which includes a discussion lens. It’s pretty much that, results from forums and reddit.
probablypower 4y ago
Not to be rude, as I appreciate the motivation behind your work, but your post ironically followed SEO junk format: 1. Enter the discussion on a human level 2. Hook onto existing context 3. Argue issues with existing context 4. Present your own software as a solution to these issues It is like a date that only wants to offer MLM business opportunities, or a long-form joke that ends with a disappointing "better nate than lever" punchline.
- mjburgess 4y ago
  In general scams work because the form works, and should work, but the content is an illusion. You could equally say everyone's following Aristotle's Rhetoric.

_aavaa_ 4y ago

On Kagi you as an individual can up rank and down rank websites (even hiding them entirely). They can conceivably use this information to do the same thing, curare the results and change their ordering

smt88 4y ago

For a long time, I've been considering a different solution to the problem, which is to create a human-curated whitelist-only search engine. I do think your idea is great and something with a lot of value for many use cases. The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.

kbyatnal 4y ago
yup totally agree, but I think it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc). They have too much data and too much of a head start. The best way to compete with them IMO is to choose a slice of queries that they perform very poorly at (like non-SEO spam results) and build something 10x better. But that brings up the challenge of getting users to remember to come back to your search engine. It's tricky for sure, with no clear answer. But for the first time in a long time, I sense that we're approaching a tipping point for Google's dominance over the web.
- smt88 4y ago
  > it's nearly impossible to compete with Google and make something much better for certain types of queries (e.g. factual info, location queries, etc) Some of these things can be outsourced to other services. An easy Google fallback is also a common solution for new search engines. > But that brings up the challenge of getting users to remember to come back to your search engine. This wouldn't really be something I'd build as a business. I'd create a nonprofit, fund it with subscriptions or donations, and cover the losses myself. I really just want it to exist for myself, and if other people used it, that would be icing on the cake.
dchuk 4y ago
"a human-curated whitelist-only search engine" This is a good idea. Basically harkening back to the old internet directory days, combined with the powerful indexing tools we have now, and then use all the great language models and topic modeling tooling to make the querying great. Said another way: moderate and protect the index, instead of trying to clean up the results of queries. You'd need to monitor the whitelisted sites in various ways to make sure they don't bait and switch and turn to spam later, or get hacked and go to shit. You might even be able to monetize on the publisher side by allowing sites to pay to be indexed faster/more regularly, which could potentially (but not totally) help with spam control too. Every publisher wants traffic. You might be able to jump start the whitelist by getting folks using a plugin so you can know what domains they spend time on in the first place and index those when signal strength warrants it. Gotta be careful about that being gamed too though. Also I think this is actually the core principal of Brave's search engine come to think of it.
- ZephyrBlu 4y ago
  > combined with the powerful indexing tools we have now What powerful indexing tools exist today? I have little context on the search domain but it's very interesting.
  - smt88 4y ago
    I'm not sure, but it sounds like they mean that building and quickly searching an index has been democratized. You can now use any number of FOSS projects or SaaS offerings to do that, depending on how big you want the index to be and how much you want to spend.
    - dchuk 4y ago
      That’s right. Back when google was coming to be, building a search index was basically computer science/grad school research territory. Now there’s tons of good starting points that you can throw money and hardware to scale quite a bit to at least prove out a product concept.
nagonago 4y ago
I like this idea in theory, but I think it would be difficult to scale up enough to be useful. Who would be the curators? If you open up curation to volunteers, it could be gamed by bad actors. However if you have too small a team, the results will be limited and biased. For example, favoring the English-speaking or tech-sphere web while ignoring large sections of the web with which the curators are unfamiliar. Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. However that could end up getting gamed too.
- smt88 4y ago
  > Who would be the curators? That's a key question. You don't necessarily need that many of them, though, if you're whitelisting at the domain level. A few dozen people could work giving a rating to many thousands of domains every year. An ideal number might be in the thousands. I'd also say they probably shouldn't be in industry, because they'd have an incentive to game the index. > Perhaps machine learning could help with scale - start with a human curated dataset, then train a model on it. While this is true, I don't think search engines other than Google and Bing need to worry that much about being gamed. It's just not worth the effort.
- larksimian 4y ago
  I think curation by paying users could work. At least it would make manipulation quite expensive. Let people downvote/upvote search results as an extra strong signal on top of monitoring what they choose to click on in the result list. Also let people block domains from searches and provide some discoverability of commonly blocked domains.
dylan604 4y ago
>which is to create a human-curated whitelist-only search engine. You might be interested in watching the series "Halt and Catch Fire".
- smt88 4y ago
  I'm not interested. Why would I be? I also don't need fiction to know what a human-curated search engine would be like, because I was using the web in the 90s when we had things like Dmoz.
  - dylan604 4y ago
    Well, you can be a curmudgeon and not enjoy someone's recommendation for a decent bit of entertainment, or you can say just politely carry on and be a grouch in the living room of your trashcan, or you could maybe try something new and actually consider watching something you haven't seen yet. edit: softened the tone oh so slightly
    - smt88 4y ago
      Sorry, I misinterpreted your suggestion as, "This is not a novel idea, and you should watch how badly it goes when someone on this TV show tries the same thing." I didn't insult you or your suggestion, though. I said I wasn't interested in the show and that it doesn't inform my understanding of the concept. Why did that hurt your feelings enough to make insulting accusations about my personality and character?
      - dylan604 4y ago
        "I'm not interested. Why would I be?" That just reads as a rude way to respond to me. "Why would I be?" Because you just described a plot line of a show. It might be fun to see how someone used that plot to make a show that someone reading HN might actually enjoy. Both sentences combined also reads as "I am perfectly content in my ways and will not listen to anything anyone might kindly suggest because I know all". Maybe because it's election season, but this unwilling to listen and stonewall is quite tiresome. So maybe just rubbed me the wrong way because of everything else and might not been taken the same way at a different time?

albatrosstrophy 4y ago

I like your idea of the search engine. It's what I miss from Google. At one time (2006?) we had the option to google search only forums and discussions, and that helped me get quality answers to my queries.

Semaphor 4y ago

Kagi has a feature, lenses, which includes a discussion lens. It’s pretty much that, results from forums and reddit.

probablypower 4y ago

Not to be rude, as I appreciate the motivation behind your work, but your post ironically followed SEO junk format: 1. Enter the discussion on a human level 2. Hook onto existing context 3. Argue issues with existing context 4. Present your own software as a solution to these issues It is like a date that only wants to offer MLM business opportunities, or a long-form joke that ends with a disappointing "better nate than lever" punchline.

mjburgess 4y ago
In general scams work because the form works, and should work, but the content is an illusion. You could equally say everyone's following Aristotle's Rhetoric.