8 ms·

Yes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.

by ori_b 10y ago

ksk 10y ago
But again, you could also argue that there would be fewer Linux exploits if it were closed source. And Linux exploits have definitely caused loss of revenue for web hosting companies, and preventing that kind of abuse does carry weight too. With search engines, there's two things. If the core goal of a search engine is to link to ads, (or pages with ads) then bad-actors will be incentivized because there's money to be made. If the core goal was simply to link to content then the incentive goes away, and so does the problem.
- TeMPOraL 10y ago
  > If the core goal was simply to link to content then the incentive goes away, and so does the problem. No, it doesn't, because the core goal of even a perfectly non-profit search engine is to sort content by relevance. That's ultimately a general-AI-complete problem that would require your computer to read your mind in order to solve it perfectly. So necessarily, a search engine is using some heuristics to approximate relevance, and those heuristics are open to gaming. Thus the cat-and-mouse game between search engines and SEO people. There's way too much content to just link to all of it.
  - jkaunisv1 10y ago
    Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?
    - jasode 10y ago
      Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
      - jkaunisv1 10y ago
        Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done on the cheap. Just trying to think of ways to get away from the filter bubble and ad-based businesses. I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects. Thanks for the info.
        posterboy 10y ago
        But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit
        jasode 10y ago
        The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
        posterboy 10y ago
  - ksk 10y ago
    The SEO people are only involved because google links to ads, or pages with ads. For the sake of argument lets say google opensources their engine. One could create a forked search portal that never linked to ads or pages with ads. The motivation to game such an engine would be greatly reduced IMO.
    - TeMPOraL 10y ago
      I disagree. Outside the startup world, a lot of SEO is done simply to drive traffic to you so that you have chance to convert some of it into paying customers. All the small businesses that pay SEO companies to spam the shit out of the Internet don't do it for ad revenue, they do it just for organic traffic. This motivation won't go away as long as people can use the Web to make money, so we're stuck with the pressure to game search results.
      - ksk 10y ago
        Maybe I should have differentiated between link-farm type SEO and web marketing. The former is more likely to cause people to abuse the system because simply getting people to visit your website or click a link provides monetary compensation. If you drove traffic to your website I could make a hand-wavy free market argument that if your product wasn't good, it wouldn't sell and you wouldn't have money to pay someone to drive traffic to your website. I fully accept that its my own opinion.
- jasode 10y ago
  >With search engines, [...] If the core goal was simply to link to content then the incentive goes away, and so does the problem. Why would the incentives go away for bad actors to game the ranking algorithms so that their bad content rises to the top of results? Your chain of logic doesn't make sense. E.g. Since the core goal of SMTP is exchanging electronic mail -- it means the incentive for for advertising (SPAM) goes away and the problem solves itself?!? Why would spammers and bad actors let the idealized goal of technology stop them from abusing it?
  - ksk 10y ago
    >Why would the incentives go away for bad actors to game the ranking algorithms so that their bad content rises to the top of results? Then you're talking about bad-actors who are not motivated strictly by financial gain, but by the successful spreading of their bad content. That is not really a problem specific to search engines. It applies to any medium that has the ability to disseminate information. > E.g. Since the core goal of SMTP is exchanging electronic mail -- it means the incentive for for advertising (SPAM) goes away and the problem solves itself?!? That is not comparing like to like. Unlike mail servers, google plays a very active role in choosing what kind of content it indexes. My point was that if they chose to simply index information and never link to ads or pages with ads, then the incentive to game the results is greatly reduced. (For e.g. - Google Scholar.)
    - jasode 10y ago
      >bad-actors who are not motivated strictly by financial gain, but by the successful spreading of their bad content. The "bad content" showing up on the top of page 1 of the search results is directly related to financial gain. That's why the SEO industry exists! >if they chose to simply index information and never link to ads or pages with ads, then the incentive to game the results is greatly reduced. The flaw in your logic is that it's the ads that's the primary motivation for bad actors to push their unwanted pages to the top. Let's say I'm a bad author that wrote a bad book about "traveling to Paris". I would want my blog page about me and my book to be at the top of search results when you type "Paris" in the search box. It doesn't matter what ads are showing in the side panel. It also doesn't matter if ads didn't exist at all because every Google user would pay a $9.99/month subscription. Either way, I still want my webpage at the top so that some percentage of search users click on my (non-ad) link and buy my sightseeing book.

Yes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.

by ori_b 10y ago

Yes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.

ksk 10y ago

But again, you could also argue that there would be fewer Linux exploits if it were closed source. And Linux exploits have definitely caused loss of revenue for web hosting companies, and preventing that kind of abuse does carry weight too. With search engines, there's two things. If the core goal of a search engine is to link to ads, (or pages with ads) then bad-actors will be incentivized because there's money to be made. If the core goal was simply to link to content then the incentive goes away, and so does the problem.

TeMPOraL 10y ago
> If the core goal was simply to link to content then the incentive goes away, and so does the problem. No, it doesn't, because the core goal of even a perfectly non-profit search engine is to sort content by relevance. That's ultimately a general-AI-complete problem that would require your computer to read your mind in order to solve it perfectly. So necessarily, a search engine is using some heuristics to approximate relevance, and those heuristics are open to gaming. Thus the cat-and-mouse game between search engines and SEO people. There's way too much content to just link to all of it.
- jkaunisv1 10y ago
  Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?
  - jasode 10y ago
    Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
    - jkaunisv1 10y ago
      Maybe not on par but close - I was expecting limitations like that, didn't realize it'd be in the petabytes! I wasn't saying or expecting that it could be done on the cheap. Just trying to think of ways to get away from the filter bubble and ad-based businesses. I was just daydreaming, without first evaluating technical limits, that there could be a distributed search engine that runs on donated CPU/storage like those SETI and genome projects. Thanks for the info.
      - posterboy 10y ago
        But the info is misleading. The Internet Archive stored 18.5 Petabytes as of Aug. 14, yet it's a non profit
        jasode 10y ago
        The Internet Archive has less ~200 to 400 billion pages. Google has 60+ trillion pages. The IA also doesn't spend money on constantly running map-reduce[1] jobs 24-by-7 which requires a datacenter of cpus and megawatts of electricity. By extension, they don't pay for a small army of programmers to write the ranking algorithms and tune the results. So on their shoestring nonprofit budget of $12 million a year, you won't be able to get good results from search queries. E.g. type "javascript" in the IA search box and the first 10 results are junk.[2] At the top is a Javascript Bible book from 2010 that's rated 3 out of 5 stars on Amazon. Compare IA with the "javascript" results from google.com[3]. It will take a lot more money than $12 million a year[4] to upgrade IA to be similar quality to Google. The increased costs would very likely exceed the financial support of their most generous donors. [1]https://en.wikipedia.org/wiki/Google_Panda https://en.wikipedia.org/wiki/Google_Panda [2]https://archive.org/search.php?query=javascript https://archive.org/search.php?query=javascript [3]https://www.google.com/search?q=javascript https://www.google.com/search?q=javascript [4]http://blog.archive.org/2015/12/26/the-internet-archive-telethon-pt-1/ http://blog.archive.org/2015/12/26/the-internet-archive-tele...
        posterboy 10y ago
- ksk 10y ago
  The SEO people are only involved because google links to ads, or pages with ads. For the sake of argument lets say google opensources their engine. One could create a forked search portal that never linked to ads or pages with ads. The motivation to game such an engine would be greatly reduced IMO.
  - TeMPOraL 10y ago
    I disagree. Outside the startup world, a lot of SEO is done simply to drive traffic to you so that you have chance to convert some of it into paying customers. All the small businesses that pay SEO companies to spam the shit out of the Internet don't do it for ad revenue, they do it just for organic traffic. This motivation won't go away as long as people can use the Web to make money, so we're stuck with the pressure to game search results.
    - ksk 10y ago
      Maybe I should have differentiated between link-farm type SEO and web marketing. The former is more likely to cause people to abuse the system because simply getting people to visit your website or click a link provides monetary compensation. If you drove traffic to your website I could make a hand-wavy free market argument that if your product wasn't good, it wouldn't sell and you wouldn't have money to pay someone to drive traffic to your website. I fully accept that its my own opinion.
jasode 10y ago
>With search engines, [...] If the core goal was simply to link to content then the incentive goes away, and so does the problem. Why would the incentives go away for bad actors to game the ranking algorithms so that their bad content rises to the top of results? Your chain of logic doesn't make sense. E.g. Since the core goal of SMTP is exchanging electronic mail -- it means the incentive for for advertising (SPAM) goes away and the problem solves itself?!? Why would spammers and bad actors let the idealized goal of technology stop them from abusing it?
- ksk 10y ago
  >Why would the incentives go away for bad actors to game the ranking algorithms so that their bad content rises to the top of results? Then you're talking about bad-actors who are not motivated strictly by financial gain, but by the successful spreading of their bad content. That is not really a problem specific to search engines. It applies to any medium that has the ability to disseminate information. > E.g. Since the core goal of SMTP is exchanging electronic mail -- it means the incentive for for advertising (SPAM) goes away and the problem solves itself?!? That is not comparing like to like. Unlike mail servers, google plays a very active role in choosing what kind of content it indexes. My point was that if they chose to simply index information and never link to ads or pages with ads, then the incentive to game the results is greatly reduced. (For e.g. - Google Scholar.)
  - jasode 10y ago
    >bad-actors who are not motivated strictly by financial gain, but by the successful spreading of their bad content. The "bad content" showing up on the top of page 1 of the search results is directly related to financial gain. That's why the SEO industry exists! >if they chose to simply index information and never link to ads or pages with ads, then the incentive to game the results is greatly reduced. The flaw in your logic is that it's the ads that's the primary motivation for bad actors to push their unwanted pages to the top. Let's say I'm a bad author that wrote a bad book about "traveling to Paris". I would want my blog page about me and my book to be at the top of search results when you type "Paris" in the search box. It doesn't matter what ads are showing in the side panel. It also doesn't matter if ads didn't exist at all because every Google user would pay a $9.99/month subscription. Either way, I still want my webpage at the top so that some percentage of search users click on my (non-ad) link and buy my sightseeing book.