5 ms·

>Speaking as a person who spent some years if his career working on full-text search, I doubt that there’s much left in the way of low-hanging fruit. Maybe so,

>Speaking as a person who spent some years if his career working on full-text search, I doubt that there’s much left in the way of low-hanging fruit. Maybe so, but judging by my search results most of the fruit is still on the tree. Natural language understanding is an active field of research. Turning Google into a search utility is a bad idea. Splitting off the crawling part and turning that into a utility would be a lot more interesting. It would allow search engines to compete on search quality while sharing the capex required for crawling.

throwaway3699 6y ago
Crawling the web is the easy part compared to human labour needed for search quality algorithms.
- cblconfederate 6y ago
  thanks partially to google, NLP is now more of a problem of (moderate) scale
  - andrepd 6y ago
    What? This is not remotely true.
    - cblconfederate 6y ago
      Applications built on pretrained bert don't necessarily need huge scale
cblconfederate 6y ago
I think we are at a point where a distributed index becomes an attractive idea. Compute nodes are cheap and every website could devote resources to support it. Considering how exclusionary google has become, web companies might have an incentive there to support a distributed solution. I don't know how developed those solutiosn are, though. Apparently there is some effort in Distributed SE coordinated by blockhains. I think this comes close to the idea of "Google as an utility". https://hackernoon.com/is-the-concept-of-a-distributed-search-engine-potent-enough-to-challenge-googles-dominance-l1s44t2 https://hackernoon.com/is-the-concept-of-a-distributed-searc...
- Spearchucker 6y ago
  Sooo... every time blockchain is mentioned I get cautious. Same for "distributed" search. From the link you posted - "...does away with every form of a single point control in all of the tasks relating to indexing, querying, data mining, and crawling." Yeah mostly that's distribution. How, though, will something that does indexing in 10 different locations ensure that I get results including something that was indexed in Japan 5 minutes ago? And Blockchain. I get that one might like to mitigate fake search entries (as that post notes). How though, using a blockchain, does one manage someone's right to be forgotten, when the blockchain's sole raison d'etre is to make all the things indelible?
  - cblconfederate 6y ago
    unlike bittorrent, there needs to be some proof of work (e.g. that the document was indexed properly) which would require multiple confirmations. Right to be forgotten would be also possible if there was one blockchain, and a mechanism to verify that the request to be forgotten is legit. Having a blockchain also means there will be one development team of "Wise people" who are going to be developing the search algorithm for everyone. Such a blockchain would not be good for current news, obviously. But even google takes days to update the average blog.
    - acdha 6y ago
      That’s the usual level of magical thinking about blockchains but we’re talking technology and economics, not magic. Right to be forgotten requires the ability to edit a blockchain, which is contrary to the entire point. A judge is unlikely to say you’re compliant with a voluntary flag on a distributed network so you’re either editing history or restricting it to trusted parties with legal obligations, at which point you’ve just invented a centralized system with more overhead. Similarly, that hypothetical team of genius developers working to solve all of the hard problems hasn’t shown up on cryptocurrencies in the last decade despite enormous amounts of money being plowed into the field, and this would have orders of magnitude less funding. Basic economics suggests that you’re far more likely to see a few ideologically committed people drowned out by the SEO types (who have far more money) trying to manipulate things to their advantage. The underlying problem is that almost nobody thinks they should pay for search and the minimum quality bar is quite high if you want people to switch from Google.
swiley 6y ago
NLP isn't something I would call low hanging fruit.
BoppreH 6y ago
> most of the fruit is still on the tree Case in point: image search for "shirt without stripes": https://github.com/elsamuko/Shirt-without-Stripes https://github.com/elsamuko/Shirt-without-Stripes HN comments: https://news.ycombinator.com/item?id=22925087 https://news.ycombinator.com/item?id=22925087
- einpoklum 6y ago
  This example is ruined, because now the search for "shirt without stripes" gives you that GitHub repository and the threads of comments about it...
  - BoppreH 6y ago
    General search is ruined, but image and product search contains very few items about the repository and comments.
    - Grimm1 6y ago
      General search being ruined is a consequence of how Google has chosen to rank results and not an indictment against search in general though.
      - zepto 6y ago
        I’d like to believe that, but surely even if Google was a non-profit which gave away its index and showed no adverts anywhere, placement in search results would still be a valuable commodity and SEO would be pretty much the same as it is today.
        Grimm1 6y ago
        I wasn't even attacking their positioning of advertisements among results. I was more saying that I think one of their larger signals (that we know of) PageRank has some consequences in terms of presenting things like a well linked and referenced GitHub result as a top result instead of an actual striped shirt. They do a really good job of filtering out garbage but in terms of contextual relevance it doesn't work out so well. Edit: I'll also point out that it may not even be a modern PageRank fault but a consequence of learning to rank or some other major signal but it's in line with what I would suppose to be the culprit.
blazespin 6y ago
Yeah, famous last words. The lawyers were right not to talk about remedies. Really, just getting the court to agree that google is abusing it's power is enough to stop them. Fool me once, as they say. That said, something has to be done about pushing more advertising rev and therefore power/influence back to alternative publishers (wpost/wsj/nyt are alt publishers now). This monopoly/duopoly on 'thought' is insane and could very well be driving a great deal of ignorance / conspiratorial thought / etc in society. Trump loves twitter! Trump is crazy, but crazy is what you need to get your message out these days. Just look at Obama and his tirade yesterday. I think it was ill-advised, the value prop of biden is dialing down the crazy, but that could just be the world we have now without more long form / substantive dialogue.
julienfr112 6y ago
What about https://commoncrawl.org/ https://commoncrawl.org/ ?
- fauigerzigerk 6y ago
  I don't think a search engine based on Commoncrawl would be able to compete with Google. They're not crawling the web frequently enough. They don't execute JavaScript (as far as I know) and I'm not sure they have access to everything that Google has access to (such as paywalled sites). I could be wrong though as I'm not following the project very closely.
  - anoncake 6y ago
    > and I'm not sure they have access to everything that Google has access to (such as paywalled sites). Which, by the way, is another hindrance for potential competitors.
    - chongli 6y ago
      Depends how you look at it. If you want to build a search engine that avoids annoyance like paywalls and JavaScript-only sites, then this isn’t so much of a hindrance.
- Grimm1 6y ago
  Having used it in some early work on our search engine, CC does not update near often enough or crawl a big enough portion of the web to be competitive with commercial crawling solutions. It's a great thing that exists, don't get me wrong, but it's better for researchers looking to amass a dataset versus competing in search.
- julienfr112 6y ago
  Could Google give it some money à la firefox so that hey, it's an competitive market !
narag 6y ago
Maybe so, but judging by my search results most of the fruit is still on the tree. So is the Stratechery guy right? Is dis-aggregation the only solution?
CivBase 6y ago
My understanding is that the superior quality of Google searches has less to do with indexing and speech analysis and more to do with how they contextualize a user's searches against their treasure trove of data on said user - data which is harvested from the huge variety of products and services they offer across their whole business.
- idank 6y ago
  Google's search results were great even before they knew your every move on the web. Some would argue it got worse.
  - skrtskrt 6y ago
    They are definitely worse, but I have attributed that to the prevalence of SEO spam. Most top results for my searches are quite literally gibberish.
    - pyrale 6y ago
      Frequently enough, you have to scroll down a full page to see the first organic results. While SEO spam is definitely a problem, that problem is very similar to search advertising to me.
- caoilte 6y ago
  or maybe that just makes it easier to target adverts at you.
- anoncake 6y ago
  They would be a lot better if they took the damn search query into account instead.
gravypod 6y ago
Even within Google, BERT was released recently which was a massive improvement to their semantic search capabilities. There's a lot of active research going on.
ocdtrekkie 6y ago
I honestly think discussing search quality with relation to Google these days is almost irrelevant: Nobody is looking at the organic results. Nobody. On a given Google search, you have one to four ads, maybe an AMP carousel, and a knowledge graph box. Somewhere is the end of what fits on your device screen. Past that is the first real search result. Google is making money because everyone thinks the first ad is the first search result (They're nearly indistinguishable, visually, and studies have demonstrated non-technical users can't tell them apart) and then everything below the ads that fits on screen is packed with links to other Google properties. We are at the point where organic search results are nearly the "page three" of Google Search: Somewhere nobody goes anymore.
- ficklepickle 6y ago
  I guess one could argue that is the actual ranking algorithm. Descending order of how much they are willing to pay. If you aren't paying, you are tied for last.
- i_love_music 6y ago
  I think this is an important piece that is often overlooked when discussing search. I wonder if all the ads magically disappeared one day if people would view it as a positive or negative change in what they perceive as relevant?
  - MacsHeadroom 6y ago
    DuckDuckGo's organic results are just as good and the 0-2 native ads above the organic results can be permanently disabled with 3 clicks on their own site: 'Settings > Advertising > Disable Ads' and intentionally made easily identifiable to respect ad blockers. Meanwhile, Google search traps me in a loop of the top ~5 ad spenders in a given industry VERY frequently. Used the phrase "public records" anywhere in your search query? Now the first page is all click funnel blog posts by the same 5 background check companies and ancestry.com. Mentioned the term "VPN" somewhere in your 10 word query? Clearly you wanted a page full of sales funnel blog posts by the same 5 consumer VPN providers with big marketing budgets. Want to disable Google Search ads? You can't. You can only make them less targeted, until the ToS changes in a month, and only if you log into a phone verified Google Account. Want to use an ad blocker on Google Search? Too bad. Google's native ads continuously define the state of the art in evasion of end-user controls through endless evolution and complex adversarial misdirection even within the same browsing session. Just use DuckDuckGo, and disable ads, and donate. It's a much better user experience (even with ads).

throwaway3699 6y ago

Crawling the web is the easy part compared to human labour needed for search quality algorithms.

cblconfederate 6y ago
thanks partially to google, NLP is now more of a problem of (moderate) scale
- andrepd 6y ago
  What? This is not remotely true.
  - cblconfederate 6y ago
    Applications built on pretrained bert don't necessarily need huge scale

cblconfederate 6y ago

I think we are at a point where a distributed index becomes an attractive idea. Compute nodes are cheap and every website could devote resources to support it. Considering how exclusionary google has become, web companies might have an incentive there to support a distributed solution. I don't know how developed those solutiosn are, though. Apparently there is some effort in Distributed SE coordinated by blockhains. I think this comes close to the idea of "Google as an utility". https://hackernoon.com/is-the-concept-of-a-distributed-search-engine-potent-enough-to-challenge-googles-dominance-l1s44t2 https://hackernoon.com/is-the-concept-of-a-distributed-searc...

Spearchucker 6y ago
Sooo... every time blockchain is mentioned I get cautious. Same for "distributed" search. From the link you posted - "...does away with every form of a single point control in all of the tasks relating to indexing, querying, data mining, and crawling." Yeah mostly that's distribution. How, though, will something that does indexing in 10 different locations ensure that I get results including something that was indexed in Japan 5 minutes ago? And Blockchain. I get that one might like to mitigate fake search entries (as that post notes). How though, using a blockchain, does one manage someone's right to be forgotten, when the blockchain's sole raison d'etre is to make all the things indelible?
- cblconfederate 6y ago
  unlike bittorrent, there needs to be some proof of work (e.g. that the document was indexed properly) which would require multiple confirmations. Right to be forgotten would be also possible if there was one blockchain, and a mechanism to verify that the request to be forgotten is legit. Having a blockchain also means there will be one development team of "Wise people" who are going to be developing the search algorithm for everyone. Such a blockchain would not be good for current news, obviously. But even google takes days to update the average blog.
  - acdha 6y ago
    That’s the usual level of magical thinking about blockchains but we’re talking technology and economics, not magic. Right to be forgotten requires the ability to edit a blockchain, which is contrary to the entire point. A judge is unlikely to say you’re compliant with a voluntary flag on a distributed network so you’re either editing history or restricting it to trusted parties with legal obligations, at which point you’ve just invented a centralized system with more overhead. Similarly, that hypothetical team of genius developers working to solve all of the hard problems hasn’t shown up on cryptocurrencies in the last decade despite enormous amounts of money being plowed into the field, and this would have orders of magnitude less funding. Basic economics suggests that you’re far more likely to see a few ideologically committed people drowned out by the SEO types (who have far more money) trying to manipulate things to their advantage. The underlying problem is that almost nobody thinks they should pay for search and the minimum quality bar is quite high if you want people to switch from Google.

swiley 6y ago

NLP isn't something I would call low hanging fruit.

BoppreH 6y ago

> most of the fruit is still on the tree Case in point: image search for "shirt without stripes": https://github.com/elsamuko/Shirt-without-Stripes https://github.com/elsamuko/Shirt-without-Stripes HN comments: https://news.ycombinator.com/item?id=22925087 https://news.ycombinator.com/item?id=22925087

einpoklum 6y ago
This example is ruined, because now the search for "shirt without stripes" gives you that GitHub repository and the threads of comments about it...
- BoppreH 6y ago
  General search is ruined, but image and product search contains very few items about the repository and comments.
  - Grimm1 6y ago
    General search being ruined is a consequence of how Google has chosen to rank results and not an indictment against search in general though.
    - zepto 6y ago
      I’d like to believe that, but surely even if Google was a non-profit which gave away its index and showed no adverts anywhere, placement in search results would still be a valuable commodity and SEO would be pretty much the same as it is today.
      - Grimm1 6y ago
        I wasn't even attacking their positioning of advertisements among results. I was more saying that I think one of their larger signals (that we know of) PageRank has some consequences in terms of presenting things like a well linked and referenced GitHub result as a top result instead of an actual striped shirt. They do a really good job of filtering out garbage but in terms of contextual relevance it doesn't work out so well. Edit: I'll also point out that it may not even be a modern PageRank fault but a consequence of learning to rank or some other major signal but it's in line with what I would suppose to be the culprit.

blazespin 6y ago

Yeah, famous last words. The lawyers were right not to talk about remedies. Really, just getting the court to agree that google is abusing it's power is enough to stop them. Fool me once, as they say. That said, something has to be done about pushing more advertising rev and therefore power/influence back to alternative publishers (wpost/wsj/nyt are alt publishers now). This monopoly/duopoly on 'thought' is insane and could very well be driving a great deal of ignorance / conspiratorial thought / etc in society. Trump loves twitter! Trump is crazy, but crazy is what you need to get your message out these days. Just look at Obama and his tirade yesterday. I think it was ill-advised, the value prop of biden is dialing down the crazy, but that could just be the world we have now without more long form / substantive dialogue.

julienfr112 6y ago

What about https://commoncrawl.org/ https://commoncrawl.org/ ?

fauigerzigerk 6y ago
I don't think a search engine based on Commoncrawl would be able to compete with Google. They're not crawling the web frequently enough. They don't execute JavaScript (as far as I know) and I'm not sure they have access to everything that Google has access to (such as paywalled sites). I could be wrong though as I'm not following the project very closely.
- anoncake 6y ago
  > and I'm not sure they have access to everything that Google has access to (such as paywalled sites). Which, by the way, is another hindrance for potential competitors.
  - chongli 6y ago
    Depends how you look at it. If you want to build a search engine that avoids annoyance like paywalls and JavaScript-only sites, then this isn’t so much of a hindrance.
Grimm1 6y ago
Having used it in some early work on our search engine, CC does not update near often enough or crawl a big enough portion of the web to be competitive with commercial crawling solutions. It's a great thing that exists, don't get me wrong, but it's better for researchers looking to amass a dataset versus competing in search.
julienfr112 6y ago
Could Google give it some money à la firefox so that hey, it's an competitive market !

narag 6y ago

Maybe so, but judging by my search results most of the fruit is still on the tree. So is the Stratechery guy right? Is dis-aggregation the only solution?

CivBase 6y ago

My understanding is that the superior quality of Google searches has less to do with indexing and speech analysis and more to do with how they contextualize a user's searches against their treasure trove of data on said user - data which is harvested from the huge variety of products and services they offer across their whole business.

idank 6y ago
Google's search results were great even before they knew your every move on the web. Some would argue it got worse.
- skrtskrt 6y ago
  They are definitely worse, but I have attributed that to the prevalence of SEO spam. Most top results for my searches are quite literally gibberish.
  - pyrale 6y ago
    Frequently enough, you have to scroll down a full page to see the first organic results. While SEO spam is definitely a problem, that problem is very similar to search advertising to me.
caoilte 6y ago
or maybe that just makes it easier to target adverts at you.
anoncake 6y ago
They would be a lot better if they took the damn search query into account instead.

gravypod 6y ago

Even within Google, BERT was released recently which was a massive improvement to their semantic search capabilities. There's a lot of active research going on.

ocdtrekkie 6y ago

I honestly think discussing search quality with relation to Google these days is almost irrelevant: Nobody is looking at the organic results. Nobody. On a given Google search, you have one to four ads, maybe an AMP carousel, and a knowledge graph box. Somewhere is the end of what fits on your device screen. Past that is the first real search result. Google is making money because everyone thinks the first ad is the first search result (They're nearly indistinguishable, visually, and studies have demonstrated non-technical users can't tell them apart) and then everything below the ads that fits on screen is packed with links to other Google properties. We are at the point where organic search results are nearly the "page three" of Google Search: Somewhere nobody goes anymore.

ficklepickle 6y ago
I guess one could argue that is the actual ranking algorithm. Descending order of how much they are willing to pay. If you aren't paying, you are tied for last.
i_love_music 6y ago
I think this is an important piece that is often overlooked when discussing search. I wonder if all the ads magically disappeared one day if people would view it as a positive or negative change in what they perceive as relevant?
- MacsHeadroom 6y ago
  DuckDuckGo's organic results are just as good and the 0-2 native ads above the organic results can be permanently disabled with 3 clicks on their own site: 'Settings > Advertising > Disable Ads' and intentionally made easily identifiable to respect ad blockers. Meanwhile, Google search traps me in a loop of the top ~5 ad spenders in a given industry VERY frequently. Used the phrase "public records" anywhere in your search query? Now the first page is all click funnel blog posts by the same 5 background check companies and ancestry.com. Mentioned the term "VPN" somewhere in your 10 word query? Clearly you wanted a page full of sales funnel blog posts by the same 5 consumer VPN providers with big marketing budgets. Want to disable Google Search ads? You can't. You can only make them less targeted, until the ToS changes in a month, and only if you log into a phone verified Google Account. Want to use an ad blocker on Google Search? Too bad. Google's native ads continuously define the state of the art in evasion of end-user controls through endless evolution and complex adversarial misdirection even within the same browsing session. Just use DuckDuckGo, and disable ads, and donate. It's a much better user experience (even with ads).