5 ms·

Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like htt

Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?

jasode 10y ago
Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
- jkaunisv1 10y ago
  Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done on the cheap. Just trying to think of ways to get away from the filter bubble and ad-based businesses. I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects. Thanks for the info.
  - posterboy 10y ago
    But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit
    - jasode 10y ago
      The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
      - posterboy 10y ago
        Talk about feeding the troll ... Your argument is invalid, simply because google didn't become big because they had more money than others, but because the technology was an advantage. Sure, expenses are a limiting factor along the way, but you use anecdotal evidence to make a point of that. Look at traffic instead, is google also single handedly stemming the whole infrastructure? The question was about other search engines, non profit at that, and in doubt of that, about the possibility. Whether or not it would be expensive and whether or not the wikimedia foundation uses a lot of their money for meet ups and the like, there are possibly other limiting factors, that might ease the financial problem when overcome.
        jasode 10y ago
        >, but because the technology was an advantage. Is "technology advantage" another way of saying their PageRank algorithm of mathematically iterating on a linear algebra problem was superior to other approaches from Lycos/Excite/Yahoo? Well, yes. But that algorithm has to run on thousands of CPUs which require lots of electricity and expensive programmers to program it. The algorithm has to be expressed on real-world things that cost lots of money. Google got $25 million of VC money in 1999. 18 months later, they figured out Adwords and that brought in enough money to self-fund expansion (buy more computers and harddrives). The IPO in 2004 brought another $1.9 billion to pay for datacenters. The superior algorithm must combine with expensive massive hardware scale to deliver quality results. >The question was about other search engines, non profit at that If you read jkaunisv1's comment more carefully, he wanted the homemade search quality to be "close" to Google's quality. You can't achieve that on a nonprofit budget of $12 million a year. (See my links for "javascript" to get an idea of what $12 million buys you.) To be fair to IA, their main mission is archiving and not cutting-edge search algorithm quality.
- deleted 10y ago
  [deleted]
- TeMPOraL 10y ago
  I believe you could drop 99 of those 100 petabytes and your average user wouldn't notice. The web is full of crap. I don't think you have to match Google's size from the start to be able to compete.
  - jkaunisv1 10y ago
    That was my thought as well. Especially if in some way this could run as a "personal" search engine. So I could fine tune it to avoid crawling and indexing certain sites - I really don't need to see anything from CNN, HuffPo, etc. If I have that urge, I can always Google it.
    - jasode 10y ago
      >TeMPOraL wrote: I believe you could drop 99 of those 100 petabytes and your average user wouldn't notice. A general purpose search engine would need that 99 petabytes of bad webpages to help machine learning algorithms classify the new and unknown web content as good or bad. >if in some way this could run as a "personal" search engine. So I could fine tune it to avoid crawling and indexing certain sites What you want sounds more like a "whitelist" of good sites to archive and a blacklist of sites to avoid. With the smaller storage requirements of the whitelist content, build a limited inverted index[1]. I agree that would be useful for a lot of personal uses but it's not really a homemade version of Google. Your method requires post-hoc reasoning and curation. Google's machine learning algorithm can make intelligent rankings on new content of new websites that don't exist yet. [1]https://en.wikipedia.org/wiki/Inverted_index https://en.wikipedia.org/wiki/Inverted_index

jasode 10y ago

Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...

jkaunisv1 10y ago
Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done on the cheap. Just trying to think of ways to get away from the filter bubble and ad-based businesses. I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects. Thanks for the info.
- posterboy 10y ago
  But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit
  - jasode 10y ago
    The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
    - posterboy 10y ago
      Talk about feeding the troll ... Your argument is invalid, simply because google didn't become big because they had more money than others, but because the technology was an advantage. Sure, expenses are a limiting factor along the way, but you use anecdotal evidence to make a point of that. Look at traffic instead, is google also single handedly stemming the whole infrastructure? The question was about other search engines, non profit at that, and in doubt of that, about the possibility. Whether or not it would be expensive and whether or not the wikimedia foundation uses a lot of their money for meet ups and the like, there are possibly other limiting factors, that might ease the financial problem when overcome.
      - jasode 10y ago
        >, but because the technology was an advantage. Is "technology advantage" another way of saying their PageRank algorithm of mathematically iterating on a linear algebra problem was superior to other approaches from Lycos/Excite/Yahoo? Well, yes. But that algorithm has to run on thousands of CPUs which require lots of electricity and expensive programmers to program it. The algorithm has to be expressed on real-world things that cost lots of money. Google got $25 million of VC money in 1999. 18 months later, they figured out Adwords and that brought in enough money to self-fund expansion (buy more computers and harddrives). The IPO in 2004 brought another $1.9 billion to pay for datacenters. The superior algorithm must combine with expensive massive hardware scale to deliver quality results. >The question was about other search engines, non profit at that If you read jkaunisv1's comment more carefully, he wanted the homemade search quality to be "close" to Google's quality. You can't achieve that on a nonprofit budget of $12 million a year. (See my links for "javascript" to get an idea of what $12 million buys you.) To be fair to IA, their main mission is archiving and not cutting-edge search algorithm quality.
deleted 10y ago
[deleted]
TeMPOraL 10y ago
I believe you could drop 99 of those 100 petabytes and your average user wouldn't notice. The web is full of crap. I don't think you have to match Google's size from the start to be able to compete.
- jkaunisv1 10y ago
  That was my thought as well. Especially if in some way this could run as a "personal" search engine. So I could fine tune it to avoid crawling and indexing certain sites - I really don't need to see anything from CNN, HuffPo, etc. If I have that urge, I can always Google it.
  - jasode 10y ago
    >TeMPOraL wrote: I believe you could drop 99 of those 100 petabytes and your average user wouldn't notice. A general purpose search engine would need that 99 petabytes of bad webpages to help machine learning algorithms classify the new and unknown web content as good or bad. >if in some way this could run as a "personal" search engine. So I could fine tune it to avoid crawling and indexing certain sites What you want sounds more like a "whitelist" of good sites to archive and a blacklist of sites to avoid. With the smaller storage requirements of the whitelist content, build a limited inverted index[1]. I agree that would be useful for a lot of personal uses but it's not really a homemade version of Google. Your method requires post-hoc reasoning and curation. Google's machine learning algorithm can make intelligent rankings on new content of new websites that don't exist yet. [1]https://en.wikipedia.org/wiki/Inverted_index https://en.wikipedia.org/wiki/Inverted_index