5 ms·

Agreed. There are only a few players with a large web index now. Google, Bing, Yandex, Gigablast, Baidu, Naver. Others that I can think of are IXQuick (not sur

by boyter 12y ago

Agreed. There are only a few players with a large web index now. Google, Bing, Yandex, Gigablast, Baidu, Naver. Others that I can think of are IXQuick (not sure if they use their own index or not) and Yioop (smallish index). There really is a lack of large players indexing the web. Its also possible that we will see the rise of niche search engines, such as iconfinder.

thomasfoster96 12y ago
Ixquick is sort of like a proxy for Google search results. It's run by the same people who run StartPage.com
- walterbell 12y ago
  They claim not to track users, any idea how they make money?
  - boyter 12y ago
    You don't have to track users to make money. It's just a way of making more money hence its occurring. DuckDuckGo makes a business not tracking users and I believe made over 1.2 million last year based on its donations.
    - thomasfoster96 12y ago
      Where did you manage to find their donation numbers from last year? I thought they'd stopped disclosing their donation amounts a few years ago (even then they were in the 6 figures for revenue).
      - boyter 12y ago
        Cannot remember sorry. The numbers were published as 6 donations of $25,000 to different organisations. I got the link from twitter. I think of the groups was a women's interest group if that helps.
        thomasfoster96 12y ago
        Ah right no worries. They used to post the donation breakdown on https://duck.co/ https://duck.co/ but the last one I could find was in 2011.
  - thomasfoster96 12y ago
    They show keyword-based ads. Same as DuckDuckGo and other search sites that claim to not track users. A search engine doesn't need to track people to make money, they just make much more money if they target ads better by tracking.
mark_l_watson 12y ago
A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).
- thomasfoster96 12y ago
  Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index. What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.
- boyter 12y ago
  Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though. The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.
  - bobajeff 12y ago
    I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
    - boyter 12y ago
      Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top. I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches. I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.
  - 12y ago
sytelus 12y ago
There are claims that Google indexes 150B pages today. Even if you settle with 1/3rd of this and take in to account that average page size for HTML alone is 200KB, it would be evident that you need 10s of millions of dollars to pay towards hardware. Remember fetching pages costs, storing them, processing them etc costs lots of money and building even semi-scalable index which have in-memory, SSD based layers is super expensive. And we haven't even started the major cost factor i.e. human employees. Today's search engines are not just indexes of web pages. They give direct answers for "when did Lincoln died", they show detailed street and satellite maps with StreetViews in search results, they act as business directory, they act as people finder, they have elaborate image search (again super expensive to do), they recognize objects in images, they have freshest news search for thousands of sources, they can do video search, they have scans of 100s of thousands of books, they catalog millions of products and so on. Even getting some of this high quality data such as satellite maps, books, business listings, product catalogues etc would cost ~100 million dollars in licensing deals. Even if you decided to get world's most productive programmers you will need at least 300 people in my most minimal estimation and more than 2-4 years before you can build anything that has non-negligible chance of competing. That's about $75M of cost per year right there at the ultra-minimal end. Of course, this is assuming you already had bigger breakthrough than that of PageRank and that you can beat state of the art machine learning and natural language processing techniques. Hopefully you can now see, for all intent and purposes, search business is closed to attack via startups. I can imagine Facebook and Apple would sooner or later get in to this business but it would be uphill battle for them primarily because of lack of talent that search engine business requires and more importantly lack of data that only Google has for billions of queries and users doing them. You can build AirBnB with smart college hires but building search engines needs truly the cream of the crop who are perfect rare blend of Computer Scientists and part time mathematicians while also being an exceptionally productive applied programmers. Google has been working for years to sweep away pretty much all talent in this area and it would take most competitors significant effort to build this kind of army. PS: DuckDuckGo is not a "real" search engine. It has a very little of its own index and for most queries they just rearrange results fetched mostly from Bing while inserting links from their own little index. If Bing shuts them out for leaching off of them, they would be toast.
- 12y ago

by boyter 12y ago

thomasfoster96 12y ago

Ixquick is sort of like a proxy for Google search results. It's run by the same people who run StartPage.com

walterbell 12y ago
They claim not to track users, any idea how they make money?
- boyter 12y ago
  You don't have to track users to make money. It's just a way of making more money hence its occurring. DuckDuckGo makes a business not tracking users and I believe made over 1.2 million last year based on its donations.
  - thomasfoster96 12y ago
    Where did you manage to find their donation numbers from last year? I thought they'd stopped disclosing their donation amounts a few years ago (even then they were in the 6 figures for revenue).
    - boyter 12y ago
      Cannot remember sorry. The numbers were published as 6 donations of $25,000 to different organisations. I got the link from twitter. I think of the groups was a women's interest group if that helps.
      - thomasfoster96 12y ago
        Ah right no worries. They used to post the donation breakdown on https://duck.co/ https://duck.co/ but the last one I could find was in 2011.
- thomasfoster96 12y ago
  They show keyword-based ads. Same as DuckDuckGo and other search sites that claim to not track users. A search engine doesn't need to track people to make money, they just make much more money if they target ads better by tracking.

mark_l_watson 12y ago

A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).

thomasfoster96 12y ago
Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index. What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.
boyter 12y ago
Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though. The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.
- bobajeff 12y ago
  I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
  - boyter 12y ago
    Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top. I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches. I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.
- 12y ago

sytelus 12y ago

There are claims that Google indexes 150B pages today. Even if you settle with 1/3rd of this and take in to account that average page size for HTML alone is 200KB, it would be evident that you need 10s of millions of dollars to pay towards hardware. Remember fetching pages costs, storing them, processing them etc costs lots of money and building even semi-scalable index which have in-memory, SSD based layers is super expensive. And we haven't even started the major cost factor i.e. human employees. Today's search engines are not just indexes of web pages. They give direct answers for "when did Lincoln died", they show detailed street and satellite maps with StreetViews in search results, they act as business directory, they act as people finder, they have elaborate image search (again super expensive to do), they recognize objects in images, they have freshest news search for thousands of sources, they can do video search, they have scans of 100s of thousands of books, they catalog millions of products and so on. Even getting some of this high quality data such as satellite maps, books, business listings, product catalogues etc would cost ~100 million dollars in licensing deals. Even if you decided to get world's most productive programmers you will need at least 300 people in my most minimal estimation and more than 2-4 years before you can build anything that has non-negligible chance of competing. That's about $75M of cost per year right there at the ultra-minimal end. Of course, this is assuming you already had bigger breakthrough than that of PageRank and that you can beat state of the art machine learning and natural language processing techniques. Hopefully you can now see, for all intent and purposes, search business is closed to attack via startups. I can imagine Facebook and Apple would sooner or later get in to this business but it would be uphill battle for them primarily because of lack of talent that search engine business requires and more importantly lack of data that only Google has for billions of queries and users doing them. You can build AirBnB with smart college hires but building search engines needs truly the cream of the crop who are perfect rare blend of Computer Scientists and part time mathematicians while also being an exceptionally productive applied programmers. Google has been working for years to sweep away pretty much all talent in this area and it would take most competitors significant effort to build this kind of army. PS: DuckDuckGo is not a "real" search engine. It has a very little of its own index and for most queries they just rearrange results fetched mostly from Bing while inserting links from their own little index. If Bing shuts them out for leaching off of them, they would be toast.

12y ago