5 ms·

I don’t see how this will ultimately be successful. They’re basically reverse engineering Google by looking at user logs. Google will always have a leg up her

I don’t see how this will ultimately be successful. They’re basically reverse engineering Google by looking at user logs. Google will always have a leg up here because they have all the Google data. And even if it does work for a while, there still needs to be the original signal to copy. Someone will have to crawl the web and index content. I’m super eager to find new approaches to search, but another Google clone is not that.

gojomo 7y ago
Indeed, using the (query -> result) mapping in their 'Human Web' logs, from other search engines, is a roundabout way of copying both the simple word-to-doc-index, and the complicated ranking logic, of other search engines. A sufficiently strict intellectual-property regime might find this a copyright violation. But without any inside info, I strongly suspect Google & other incumbents already do similarly-indirect modeling of their competitors' behavior, via extensive query/click-trail mining, in ways that ultimately feed into improvements of their own systems. So, they might not want to press the issue. Still, this creates a dependency on the competitor you were hoping to displace, where most of your earlier values comes from "drafting" in the easy-path they've already cleared.
- solso 7y ago
  [Disclaimer I work for Cliqz] Copying, learning would be a bit more precise, and I'm not kidding. We do not answer queries 1 to 1, query-logs are used to build a more concise model of the page. It's not just a cache, we started like this, but we quickly learn to answer unseen queries. A sufficient strict IP might find text snipped a copyright violation too. It's a trick area. Personally, I'm at peace as we get the content of web pages, as everyone else. As for the dependency, it would be if we were not able to generate our own synthetic queries, which we are. So even if all other search engines of the world were to disappear, we would still be able to operate. That was not always the case, as you pointed out.
  - lawl 7y ago
    > A sufficient strict IP might find text snipped a copyright violation too. It's a trick area. Personally, I'm at peace as we get the content of web pages, as everyone else. Interesting you bring this up. Wasn't the company funding cliqz in favor of text snippets being copyright violations when the big G does it? (German/EU Leistungsschutzrecht) [0] I mean, I think bootstrapping from google query logs is fine, but following the money it does seem like a double standard. Cliqz "stealing" from Google SERPs is fine, but Google news "stealing" text snippets isn't? [0]: https://leistungsschutzrecht.info/stimmen-zum-lsr/presseartikel/burda-zieht-sich-aus-vg-media-zur-ck https://leistungsschutzrecht.info/stimmen-zum-lsr/pressearti... (Relevant quote: Das Leistungsschutzrecht halte man nach wie vor nicht für falsch. Man setze sogar weiterhin auf ein euop. Leistungsschutzrecht, mit dessen Hilfe man sich erhofft, endlich Geld von Google zu erhalten. sloppy translation: We still consider the [we-want-money-for-google-news-snippets-law] to not be the wrong approach. We in fact contiue hoping it will work out on a european scale)
    - solso 7y ago
      [Disclaimer I work for Cliqz] This is 100% personal opinion, not a spokesman neither for Cliqz nor Burda. As all German media? Yes, it seems that they were lobbying for it, not clear if they still are part of the consortium or not. Why? No idea, really. Perhaps they are diversifying, lobbying on one hand, trying to build a competitor with another. But to make an assessment of the "goodness" of their intentions I prefer to stick to the facts: besides complaining to regulators, or not, they do fund a potential alternative. That's very commendable, cannot name many companies that are crazy/adventurous enough to put a ton of money on something as risky as what Cliqz is trying to do. To sum up, if the lobbying is a minus, the building is a massive plus, a clear positive outcome IMHO.
commoner 7y ago
The article describes techniques used by all search engines, not just Google. Search engines have existed before Google, and despite Google's monopoly on search, "Google clone" is a poor term to describe all search engines when alternatives with unique features (e.g. DuckDuckGo) exist.
- deleted 7y ago
  [deleted]
- nostromo 7y ago
  They are literally rebuilding Google by tracking how users use Google and rebuilding the SERPs. If that’s not a Google clone I don’t know what is.
  - wizzwizz4 7y ago
    All search engines have search-engine results pages. They're looking at how users use Google Search because the data's there. They're making a competitor to Google Search. That doesn't mean they're rebuilding Google Search's SERPs, or making a Google Search “clone”; I've got results from Cliqz for queries I'm confident have never been put into Google before, meaning it's functioning as an independent search engine.
    - pheug 7y ago
      >They're looking at how users use Google Search because the data's there This. Having worked in past life for one of their competitors, can confirm - what users click on (in SERP) is one of the most powerful signals for ranking. And who got (almost) all the clicks in the world? Google! That's why it's so damn hard to beat them. It's the unreasonable effectiveness of data: more data (which they have almost all of) usually beats a smarter algorithm, and with 20 years R&D, theirs is surely not dumb. Do the clicks belong to users or Google, that's an interesting question, though.
- adventskalender 7y ago
  How many real alternatives exist, though? DuckDuckGo is based on Bing, afaik. So there are Google and Bing - any others?
julvo 7y ago
And even if it does work for a while, there still needs to be the original signal to copy I'd say they need to start somewhere. Using other search engine's results is a ways to get things started, so that they can build their own index on crawled content later. For now, it's great to see another competitor for Google coming up.
riazrizvi 7y ago
Google search worked well enough for me when it was launched 22 years ago. So the patents to that early version should be expired. I'd be happy to use a competitor's service if they would reintroduce a Search API, and PageRank, just give me a programmatic interface that doesn't throw up Captcha after a few searches. I'd pay a reasonable price for the compute time + profit margin.
- otterley 7y ago
  If Google went back to the original algorithms it would return even worse results. Much of the noise can be attributed to two factors: (1) the explosive growth of the Internet itself, which was much much smaller and had more focused and authoritative content 20 years ago; and (2) aggressive SEO tactics today that would easily fool the early versions of PageRank.
  - jermaustin1 7y ago
    I kind of miss the early directories for sites. It probably can't happen now, since there are billions of websites, but it was a simpler time, and finding something you needed wasn't THAT hard.
    - macintux 7y ago
      I would strongly disagree. I was at a dinner party in the mid-90s and we had a long-running discussion about how impossible it was to find anything using Altavista and the other search engines. Yahoo's directory was even worse. The best idea anyone had, which I thought wouldn't scale but didn't have better ideas, was implementing a keyword registry à la AOL.
  - riazrizvi 7y ago
    The point about a Search API is that it opens up the problem to a world of 3rd party developers, as opposed to Google putting up competitive defenses by restricting programmatic access. A few years ago I built a web directory that organized and filtered Alexa's top 1MM websites, using PageRank, semantic clustering, and natural language processing. It worked really well, gave me a holistic picture of the internet that was quite different to how Yahoo presented it through their directory. However building it was time-consuming because Google would keep shutting out the ipaddress I was using, so I'd have to keep driving to the next Starbucks up the street to resume my scripts. Now compare the world of search to the world of mobile phones. Imagine if mobile phones only came with proprietary apps, and there were no app stores. That's where search is right now.
udkl 7y ago
Being successfully might not mean being at the top of the pyramid. DDG is a an example of a successful search engine. Further, they don't have to be better than google in the quality of their search results. As soon as the results have good relevance, it's good competition. What matters is that they are 'good enough', that there is a hint of competition to the Google-Bing monopoly. Offering privacy centric 'competition' is what they say their main aim is and what their success should be judged upon.
- mda 7y ago
  DDG is not a real search engine, I think this project is way more impressive than DDG.
  - colejohnson66 7y ago
    DDG absolutely is a search engine; It’s a system (engine) that lets you search. From Wikipedia[0]: > A web search engine or Internet search engine is a software system that is designed to carry out web search (Internet search), which means to search the World Wide Web in a systematic way for particular information specified in a textual web search query. Just because they don’t do the crawling like Google and Bing doesn’t mean they aren’t a search engine. [0]: https://en.wikipedia.org/wiki/Web_search_engine https://en.wikipedia.org/wiki/Web_search_engine
    - palerdot 7y ago
      This is tautology. In that case, everyone can just put up their own html page with a search box, proxying the results from Google or Bing, and can claim that they have built a search engine ...
      - deleted 7y ago
        [deleted]

gojomo 7y ago

Indeed, using the (query -> result) mapping in their 'Human Web' logs, from other search engines, is a roundabout way of copying both the simple word-to-doc-index, and the complicated ranking logic, of other search engines. A sufficiently strict intellectual-property regime might find this a copyright violation. But without any inside info, I strongly suspect Google & other incumbents already do similarly-indirect modeling of their competitors' behavior, via extensive query/click-trail mining, in ways that ultimately feed into improvements of their own systems. So, they might not want to press the issue. Still, this creates a dependency on the competitor you were hoping to displace, where most of your earlier values comes from "drafting" in the easy-path they've already cleared.

solso 7y ago
[Disclaimer I work for Cliqz] Copying, learning would be a bit more precise, and I'm not kidding. We do not answer queries 1 to 1, query-logs are used to build a more concise model of the page. It's not just a cache, we started like this, but we quickly learn to answer unseen queries. A sufficient strict IP might find text snipped a copyright violation too. It's a trick area. Personally, I'm at peace as we get the content of web pages, as everyone else. As for the dependency, it would be if we were not able to generate our own synthetic queries, which we are. So even if all other search engines of the world were to disappear, we would still be able to operate. That was not always the case, as you pointed out.
- lawl 7y ago
  > A sufficient strict IP might find text snipped a copyright violation too. It's a trick area. Personally, I'm at peace as we get the content of web pages, as everyone else. Interesting you bring this up. Wasn't the company funding cliqz in favor of text snippets being copyright violations when the big G does it? (German/EU Leistungsschutzrecht) [0] I mean, I think bootstrapping from google query logs is fine, but following the money it does seem like a double standard. Cliqz "stealing" from Google SERPs is fine, but Google news "stealing" text snippets isn't? [0]: https://leistungsschutzrecht.info/stimmen-zum-lsr/presseartikel/burda-zieht-sich-aus-vg-media-zur-ck https://leistungsschutzrecht.info/stimmen-zum-lsr/pressearti... (Relevant quote: Das Leistungsschutzrecht halte man nach wie vor nicht für falsch. Man setze sogar weiterhin auf ein euop. Leistungsschutzrecht, mit dessen Hilfe man sich erhofft, endlich Geld von Google zu erhalten. sloppy translation: We still consider the [we-want-money-for-google-news-snippets-law] to not be the wrong approach. We in fact contiue hoping it will work out on a european scale)
  - solso 7y ago
    [Disclaimer I work for Cliqz] This is 100% personal opinion, not a spokesman neither for Cliqz nor Burda. As all German media? Yes, it seems that they were lobbying for it, not clear if they still are part of the consortium or not. Why? No idea, really. Perhaps they are diversifying, lobbying on one hand, trying to build a competitor with another. But to make an assessment of the "goodness" of their intentions I prefer to stick to the facts: besides complaining to regulators, or not, they do fund a potential alternative. That's very commendable, cannot name many companies that are crazy/adventurous enough to put a ton of money on something as risky as what Cliqz is trying to do. To sum up, if the lobbying is a minus, the building is a massive plus, a clear positive outcome IMHO.

commoner 7y ago

The article describes techniques used by all search engines, not just Google. Search engines have existed before Google, and despite Google's monopoly on search, "Google clone" is a poor term to describe all search engines when alternatives with unique features (e.g. DuckDuckGo) exist.

deleted 7y ago
[deleted]
nostromo 7y ago
They are literally rebuilding Google by tracking how users use Google and rebuilding the SERPs. If that’s not a Google clone I don’t know what is.
- wizzwizz4 7y ago
  All search engines have search-engine results pages. They're looking at how users use Google Search because the data's there. They're making a competitor to Google Search. That doesn't mean they're rebuilding Google Search's SERPs, or making a Google Search “clone”; I've got results from Cliqz for queries I'm confident have never been put into Google before, meaning it's functioning as an independent search engine.
  - pheug 7y ago
    >They're looking at how users use Google Search because the data's there This. Having worked in past life for one of their competitors, can confirm - what users click on (in SERP) is one of the most powerful signals for ranking. And who got (almost) all the clicks in the world? Google! That's why it's so damn hard to beat them. It's the unreasonable effectiveness of data: more data (which they have almost all of) usually beats a smarter algorithm, and with 20 years R&D, theirs is surely not dumb. Do the clicks belong to users or Google, that's an interesting question, though.
adventskalender 7y ago
How many real alternatives exist, though? DuckDuckGo is based on Bing, afaik. So there are Google and Bing - any others?

julvo 7y ago

And even if it does work for a while, there still needs to be the original signal to copy I'd say they need to start somewhere. Using other search engine's results is a ways to get things started, so that they can build their own index on crawled content later. For now, it's great to see another competitor for Google coming up.

riazrizvi 7y ago

Google search worked well enough for me when it was launched 22 years ago. So the patents to that early version should be expired. I'd be happy to use a competitor's service if they would reintroduce a Search API, and PageRank, just give me a programmatic interface that doesn't throw up Captcha after a few searches. I'd pay a reasonable price for the compute time + profit margin.

otterley 7y ago
If Google went back to the original algorithms it would return even worse results. Much of the noise can be attributed to two factors: (1) the explosive growth of the Internet itself, which was much much smaller and had more focused and authoritative content 20 years ago; and (2) aggressive SEO tactics today that would easily fool the early versions of PageRank.
- jermaustin1 7y ago
  I kind of miss the early directories for sites. It probably can't happen now, since there are billions of websites, but it was a simpler time, and finding something you needed wasn't THAT hard.
  - macintux 7y ago
    I would strongly disagree. I was at a dinner party in the mid-90s and we had a long-running discussion about how impossible it was to find anything using Altavista and the other search engines. Yahoo's directory was even worse. The best idea anyone had, which I thought wouldn't scale but didn't have better ideas, was implementing a keyword registry à la AOL.
- riazrizvi 7y ago
  The point about a Search API is that it opens up the problem to a world of 3rd party developers, as opposed to Google putting up competitive defenses by restricting programmatic access. A few years ago I built a web directory that organized and filtered Alexa's top 1MM websites, using PageRank, semantic clustering, and natural language processing. It worked really well, gave me a holistic picture of the internet that was quite different to how Yahoo presented it through their directory. However building it was time-consuming because Google would keep shutting out the ipaddress I was using, so I'd have to keep driving to the next Starbucks up the street to resume my scripts. Now compare the world of search to the world of mobile phones. Imagine if mobile phones only came with proprietary apps, and there were no app stores. That's where search is right now.

udkl 7y ago

Being successfully might not mean being at the top of the pyramid. DDG is a an example of a successful search engine. Further, they don't have to be better than google in the quality of their search results. As soon as the results have good relevance, it's good competition. What matters is that they are 'good enough', that there is a hint of competition to the Google-Bing monopoly. Offering privacy centric 'competition' is what they say their main aim is and what their success should be judged upon.

mda 7y ago
DDG is not a real search engine, I think this project is way more impressive than DDG.
- colejohnson66 7y ago
  DDG absolutely is a search engine; It’s a system (engine) that lets you search. From Wikipedia[0]: > A web search engine or Internet search engine is a software system that is designed to carry out web search (Internet search), which means to search the World Wide Web in a systematic way for particular information specified in a textual web search query. Just because they don’t do the crawling like Google and Bing doesn’t mean they aren’t a search engine. [0]: https://en.wikipedia.org/wiki/Web_search_engine https://en.wikipedia.org/wiki/Web_search_engine
  - palerdot 7y ago
    This is tautology. In that case, everyone can just put up their own html page with a search box, proxying the results from Google or Bing, and can claim that they have built a search engine ...
    - deleted 7y ago
      [deleted]