5 ms·

>Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them That sounds like an argument that would ap

by ksk 10y ago

>Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them That sounds like an argument that would apply against open sourcing anything.

mc32 10y ago
How? Open sourcing most things don't lead to "gaming" systems in a malicious manner.
- ksk 10y ago
  Its the argument that the source must be hidden or else people will release exploits for it. e.g. The fact that Linux is open source has caused people to release exploits for it.
  - Dylan16807 10y ago
    With exploits, you can easily fix them. The problem with this sort of algorithm is that you have people directly competing, and there's no objective measure of site quality. The system has to use what it can measure, and this means it's gamable. Security through obscurity sucks, but it's better than having nothing at all.
    - ksk 10y ago
      >Security through obscurity sucks, but it's better than having nothing at all. Why do you think it would not result in people improving the algorithm to account for people gaming the system?
      - Dylan16807 10y ago
        The algorithm would be better, but the goal is keeping the algorithm as far ahead of the gaming as possible, and that's not assured at all. Though if you could use a parallel universe to do that development...
ori_b 10y ago
Yes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.
- ksk 10y ago
  But again, you could also argue that there would be fewer Linux exploits if it were closed source. And Linux exploits have definitely caused loss of revenue for web hosting companies, and preventing that kind of abuse does carry weight too. With search engines, there's two things. If the core goal of a search engine is to link to ads, (or pages with ads) then bad-actors will be incentivized because there's money to be made. If the core goal was simply to link to content then the incentive goes away, and so does the problem.
  - TeMPOraL 10y ago
    > If the core goal was simply to link to content then the incentive goes away, and so does the problem. No, it doesn't, because the core goal of even a perfectly non-profit search engine is to sort content by relevance. That's ultimately a general-AI-complete problem that would require your computer to read your mind in order to solve it perfectly. So necessarily, a search engine is using some heuristics to approximate relevance, and those heuristics are open to gaming. Thus the cat-and-mouse game between search engines and SEO people. There's way too much content to just link to all of it.
    - jkaunisv1 10y ago
      Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?
      - jasode 10y ago
        Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...

by ksk 10y ago

>Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them That sounds like an argument that would apply against open sourcing anything.

mc32 10y ago

How? Open sourcing most things don't lead to "gaming" systems in a malicious manner.

ksk 10y ago
Its the argument that the source must be hidden or else people will release exploits for it. e.g. The fact that Linux is open source has caused people to release exploits for it.
- Dylan16807 10y ago
  With exploits, you can easily fix them. The problem with this sort of algorithm is that you have people directly competing, and there's no objective measure of site quality. The system has to use what it can measure, and this means it's gamable. Security through obscurity sucks, but it's better than having nothing at all.
  - ksk 10y ago
    >Security through obscurity sucks, but it's better than having nothing at all. Why do you think it would not result in people improving the algorithm to account for people gaming the system?
    - Dylan16807 10y ago
      The algorithm would be better, but the goal is keeping the algorithm as far ahead of the gaming as possible, and that's not assured at all. Though if you could use a parallel universe to do that development...

ori_b 10y ago

Yes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.

ksk 10y ago
But again, you could also argue that there would be fewer Linux exploits if it were closed source. And Linux exploits have definitely caused loss of revenue for web hosting companies, and preventing that kind of abuse does carry weight too. With search engines, there's two things. If the core goal of a search engine is to link to ads, (or pages with ads) then bad-actors will be incentivized because there's money to be made. If the core goal was simply to link to content then the incentive goes away, and so does the problem.
- TeMPOraL 10y ago
  > If the core goal was simply to link to content then the incentive goes away, and so does the problem. No, it doesn't, because the core goal of even a perfectly non-profit search engine is to sort content by relevance. That's ultimately a general-AI-complete problem that would require your computer to read your mind in order to solve it perfectly. So necessarily, a search engine is using some heuristics to approximate relevance, and those heuristics are open to gaming. Thus the cat-and-mouse game between search engines and SEO people. There's way too much content to just link to all of it.
  - jkaunisv1 10y ago
    Are there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?
    - jasode 10y ago
      Would you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...