3 ms·

Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done

Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done on the cheap. Just trying to think of ways to get away from the filter bubble and ad-based businesses. I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects. Thanks for the info.

posterboy 10y ago
But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit
- jasode 10y ago
  The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
  - posterboy 10y ago
    Talk about feeding the troll ... Your argument is invalid, simply because google didn't become big because they had more money than others, but because the technology was an advantage. Sure, expenses are a limiting factor along the way, but you use anecdotal evidence to make a point of that. Look at traffic instead, is google also single handedly stemming the whole infrastructure? The question was about other search engines, non profit at that, and in doubt of that, about the possibility. Whether or not it would be expensive and whether or not the wikimedia foundation uses a lot of their money for meet ups and the like, there are possibly other limiting factors, that might ease the financial problem when overcome.
    - jasode 10y ago
      >, but because the technology was an advantage. Is "technology advantage" another way of saying their PageRank algorithm of mathematically iterating on a linear algebra problem was superior to other approaches from Lycos/Excite/Yahoo? Well, yes. But that algorithm has to run on thousands of CPUs which require lots of electricity and expensive programmers to program it. The algorithm has to be expressed on real-world things that cost lots of money. Google got $25 million of VC money in 1999. 18 months later, they figured out Adwords and that brought in enough money to self-fund expansion (buy more computers and harddrives). The IPO in 2004 brought another $1.9 billion to pay for datacenters. The superior algorithm must combine with expensive massive hardware scale to deliver quality results. >The question was about other search engines, non profit at that If you read jkaunisv1's comment more carefully, he wanted the homemade search quality to be "close" to Google's quality. You can't achieve that on a nonprofit budget of $12 million a year. (See my links for "javascript" to get an idea of what $12 million buys you.) To be fair to IA, their main mission is archiving and not cutting-edge search algorithm quality.

posterboy 10y ago

But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit

jasode 10y ago
The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
- posterboy 10y ago
  Talk about feeding the troll ... Your argument is invalid, simply because google didn't become big because they had more money than others, but because the technology was an advantage. Sure, expenses are a limiting factor along the way, but you use anecdotal evidence to make a point of that. Look at traffic instead, is google also single handedly stemming the whole infrastructure? The question was about other search engines, non profit at that, and in doubt of that, about the possibility. Whether or not it would be expensive and whether or not the wikimedia foundation uses a lot of their money for meet ups and the like, there are possibly other limiting factors, that might ease the financial problem when overcome.
  - jasode 10y ago
    >, but because the technology was an advantage. Is "technology advantage" another way of saying their PageRank algorithm of mathematically iterating on a linear algebra problem was superior to other approaches from Lycos/Excite/Yahoo? Well, yes. But that algorithm has to run on thousands of CPUs which require lots of electricity and expensive programmers to program it. The algorithm has to be expressed on real-world things that cost lots of money. Google got $25 million of VC money in 1999. 18 months later, they figured out Adwords and that brought in enough money to self-fund expansion (buy more computers and harddrives). The IPO in 2004 brought another $1.9 billion to pay for datacenters. The superior algorithm must combine with expensive massive hardware scale to deliver quality results. >The question was about other search engines, non profit at that If you read jkaunisv1's comment more carefully, he wanted the homemade search quality to be "close" to Google's quality. You can't achieve that on a nonprofit budget of $12 million a year. (See my links for "javascript" to get an idea of what $12 million buys you.) To be fair to IA, their main mission is archiving and not cutting-edge search algorithm quality.