5 ms·

Alas, it’s 2016 and there’s still no serious competitor to Google. And there will not be any time soon. Writing an efficient crawler for what we call the "mode

by romaniv 11y ago

Alas, it’s 2016 and there’s still no serious competitor to Google. And there will not be any time soon. Writing an efficient crawler for what we call the "modern" web is not something a small or even median-size company can pull off. Google enjoys a tremendous competitive advantage: people specifically optimize webpages for what it can and cannot do. So any newcomer to the field will have to replicate tons of technologies Google had years to perfect (in addition to solving problems like storage, search logic and bandwidth management).

hotcool 11y ago
Newcomers might be better off approaching the challenge in a totally different way.
- whitegrape 11y ago
  I remember DDG did side-by-side comparisons, but did they ever just query Google themselves and then do their own stuff on top? Anyway I'm not sure Google can be beaten at all by any particular newcomer, instead they might suffer death by a thousand cuts as more niche engines like Shodan show up and bring in context-specific searches that are orders of magnitude better than Google's more complicated all-web-they-approve-of-and-find-relevant-to-your-account search.
  - deleted 11y ago
    [deleted]
- romaniv 11y ago
  A newcomer can index, store, search and display data differently, but first they still have to get that data. Which means dealing with JavaScript, paywalls and other fun stuff like that. Besides, regardless of what you do, you would need to have tons of storage, bandwidth, CPU power and a high-availability infrastructure.
  - grey-area 11y ago
    The vast majority of the data on the internet is dross or spam, if someone finds a way to index only quality content, the crawling burden could be radically smaller.
ErikAugust 11y ago
You are perpetuating a myth for the sake of a monopoly. A novel solution could be designed and implemented by a small company but no one dares.
- webmasterraj 11y ago
  Maybe. Search is a resource-intensive algorithmic problem. So you need one of two things to beat Google: more resources or a much better algorithm. You're not going to get the first unless you're Facebook or Amazon or God, but maybe you can build a smarter algorithm. You are up against an army of some of the smartest computer scientists and mathematicians ever assembled -- but what you have going for you is a complete lack of inertia or legacy. You could try crazy things that Google might not, because they won't think it'll work. If you get lucky, one of those blows up. But you have to get very lucky (this is the Innovator's Dilemma in a nutshell). Anyone want to give it a shot?
  - vmorgulis 11y ago
    > ..more resources or a much better algorithm. Or a limited search space. For myself, it could be HN, SO and Wikipedia. Content can even be static and downloaded once or regularly.
  - jib 11y ago
    You can also niche your space. Hoogle would be an example of that. If you know the searcher cares about Haskell functions only I imagine you can beat Google in that space. That solution probably expands to other interest spheres.
  - dragandj 11y ago
    Additionally, you'll still have to find a way to make money. Even if you manage to get on par with Google's search results, it is difficult to replicate their advertising cash cow, and even more difficult to invent a completely new monetization strategy and make it successful.
- adevine 11y ago
  If a small company could design a credible alternative to Google, I'm sure it would have been done - the incentives are too great. Entrenched businesses are RARELY displaced from their top position. Instead, what usually happens is the world changes around them, and the entrenched business is ill suited to compete in the new world. Nobody ever managed to seriously challenge Microsoft for desktop/laptop OS dominance: https://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Desktop_and_laptop_computers https://en.wikipedia.org/wiki/Usage_share_of_operating_syste.... The issue is that desktop/laptop OS dominance is no longer as important as it once was.
vonklaus 11y ago
You are not correct. Apache Foundation, electron/chromium & elastic are the open source components of google. meme-explorer is the paradigm but was funded by drape/jpl_nasa so work was suspended on it. heirarchical crawl index and DNS rebuild(or similar model) will lead to search platform and fix discovery, monopoly and monetization. note that Google has solved a HUGE problem, but their task is now impossible. You can't simply use a textbook and some booleans and quotations(which I am not sure they even respect) to deliver results for a billion people. Individuals and the market will calibrate their own results.
- coolsunglasses 11y ago
  I maintain an Elasticsearch client and have used it for work quite a bit, but let me just say >elastic hahahahahahaa you have not used Elasticsearch I see :)

by romaniv 11y ago

hotcool 11y ago

Newcomers might be better off approaching the challenge in a totally different way.

whitegrape 11y ago
I remember DDG did side-by-side comparisons, but did they ever just query Google themselves and then do their own stuff on top? Anyway I'm not sure Google can be beaten at all by any particular newcomer, instead they might suffer death by a thousand cuts as more niche engines like Shodan show up and bring in context-specific searches that are orders of magnitude better than Google's more complicated all-web-they-approve-of-and-find-relevant-to-your-account search.
- deleted 11y ago
  [deleted]
romaniv 11y ago
A newcomer can index, store, search and display data differently, but first they still have to get that data. Which means dealing with JavaScript, paywalls and other fun stuff like that. Besides, regardless of what you do, you would need to have tons of storage, bandwidth, CPU power and a high-availability infrastructure.
- grey-area 11y ago
  The vast majority of the data on the internet is dross or spam, if someone finds a way to index only quality content, the crawling burden could be radically smaller.

ErikAugust 11y ago

You are perpetuating a myth for the sake of a monopoly. A novel solution could be designed and implemented by a small company but no one dares.

webmasterraj 11y ago
Maybe. Search is a resource-intensive algorithmic problem. So you need one of two things to beat Google: more resources or a much better algorithm. You're not going to get the first unless you're Facebook or Amazon or God, but maybe you can build a smarter algorithm. You are up against an army of some of the smartest computer scientists and mathematicians ever assembled -- but what you have going for you is a complete lack of inertia or legacy. You could try crazy things that Google might not, because they won't think it'll work. If you get lucky, one of those blows up. But you have to get very lucky (this is the Innovator's Dilemma in a nutshell). Anyone want to give it a shot?
- vmorgulis 11y ago
  > ..more resources or a much better algorithm. Or a limited search space. For myself, it could be HN, SO and Wikipedia. Content can even be static and downloaded once or regularly.
- jib 11y ago
  You can also niche your space. Hoogle would be an example of that. If you know the searcher cares about Haskell functions only I imagine you can beat Google in that space. That solution probably expands to other interest spheres.
- dragandj 11y ago
  Additionally, you'll still have to find a way to make money. Even if you manage to get on par with Google's search results, it is difficult to replicate their advertising cash cow, and even more difficult to invent a completely new monetization strategy and make it successful.
adevine 11y ago
If a small company could design a credible alternative to Google, I'm sure it would have been done - the incentives are too great. Entrenched businesses are RARELY displaced from their top position. Instead, what usually happens is the world changes around them, and the entrenched business is ill suited to compete in the new world. Nobody ever managed to seriously challenge Microsoft for desktop/laptop OS dominance: https://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Desktop_and_laptop_computers https://en.wikipedia.org/wiki/Usage_share_of_operating_syste.... The issue is that desktop/laptop OS dominance is no longer as important as it once was.

vonklaus 11y ago

You are not correct. Apache Foundation, electron/chromium & elastic are the open source components of google. meme-explorer is the paradigm but was funded by drape/jpl_nasa so work was suspended on it. heirarchical crawl index and DNS rebuild(or similar model) will lead to search platform and fix discovery, monopoly and monetization. note that Google has solved a HUGE problem, but their task is now impossible. You can't simply use a textbook and some booleans and quotations(which I am not sure they even respect) to deliver results for a billion people. Individuals and the market will calibrate their own results.

coolsunglasses 11y ago
I maintain an Elasticsearch client and have used it for work quite a bit, but let me just say >elastic hahahahahahaa you have not used Elasticsearch I see :)