7 ms·
Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them. Disclosing them publicly may have the parad
by jerf 10y ago
Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them. Disclosing them publicly may have the paradoxical effect of introducing even more distortion into the system.
The only solution I can see is just ensuring there's more competition, and even that isn't a perfect one.
- awinter-py 10y agoyes -- for most topics, the curated information in a wikipedia article is better than the first page of G results. This is partly because curation still works in 2016, but also because G results are easier to monetize. A significant portion of web content is some form of clickbait, i.e. ad-supported pages with no new information on their topic. For now, viral content seems like a bigger threat to search results than abuse by the index operators. (Though that can change in a hurry; some studies have shown that search rankings can be used to influence get-out-the-vote numbers).
- honkhonkpants 10y agoFor most topics the Wikipedia entry is the top result.
- ksk 10y ago>Broadly speaking, the search engines aren't disclosing these algorithms because we know then people will abuse them That sounds like an argument that would apply against open sourcing anything.
- mc32 10y agoHow? Open sourcing most things don't lead to "gaming" systems in a malicious manner.
- ksk 10y agoIts the argument that the source must be hidden or else people will release exploits for it. e.g. The fact that Linux is open source has caused people to release exploits for it.
- Dylan16807 10y agoWith exploits, you can easily fix them. The problem with this sort of algorithm is that you have people directly competing, and there's no objective measure of site quality. The system has to use what it can measure, and this means it's gamable. Security through obscurity sucks, but it's better than having nothing at all.
- ksk 10y ago>Security through obscurity sucks, but it's better than having nothing at all. Why do you think it would not result in people improving the algorithm to account for people gaming the system?
- Dylan16807 10y agoThe algorithm would be better, but the goal is keeping the algorithm as far ahead of the gaming as possible, and that's not assured at all. Though if you could use a parallel universe to do that development...
- ori_b 10y agoYes, it does. however, it carries more weight when applied to a product where the core problem is a cat and mouse game of fighting abuse.
- ksk 10y agoBut again, you could also argue that there would be fewer Linux exploits if it were closed source. And Linux exploits have definitely caused loss of revenue for web hosting companies, and preventing that kind of abuse does carry weight too. With search engines, there's two things. If the core goal of a search engine is to link to ads, (or pages with ads) then bad-actors will be incentivized because there's money to be made. If the core goal was simply to link to content then the incentive goes away, and so does the problem.
- TeMPOraL 10y ago> If the core goal was simply to link to content then the incentive goes away, and so does the problem. No, it doesn't, because the core goal of even a perfectly non-profit search engine is to sort content by relevance. That's ultimately a general-AI-complete problem that would require your computer to read your mind in order to solve it perfectly. So necessarily, a search engine is using some heuristics to approximate relevance, and those heuristics are open to gaming. Thus the cat-and-mouse game between search engines and SEO people. There's way too much content to just link to all of it.
- jkaunisv1 10y agoAre there non-profit search engines? Any domain experts who can speak to how hard/expensive it would be to run your own (a quick google shows me stuff like http://www.gigablast.com/ http://www.gigablast.com/)?
- jasode 10y agoWould you want the non-profit search engine to be on par with Google for quality? Google's storage just for the index of web pages is about 100 petabytes.[1][2] To compare, Wikipedia's storage is only ~50 TB[3] and it's mostly static content with no heavy cpu constantly processing the data. Relatively speaking, Wikipedia's operation is tiny and yet they're regularly running banner ads for donations to keep the lights on. Copying an open-source search engine algorithm from github isn't enough to get a usuable search tool. And buying some 8 terabyte hard drives from Amazon to run a RAID setup in the garage won't store enough data to feed the algorithm. [1]https://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html https://googleblog.blogspot.com/2010/06/our-new-search-index... [2]https://www.quora.com/How-large-is-the-Google-search-index-as-of-mid-2016 https://www.quora.com/How-large-is-the-Google-search-index-a... [3]https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
- rwmj 10y agoAnother "solution" would be to force social media companies to randomly mix in some low percentage of opposing views into someone's feed. Of course this would have strange side-effects, eg. it might mean that a spam filter had to let through a certain percentage of spam.
- stcredzero 10y agoNewsflash: Spammers are more intelligent than your friends and relatives who share political opinions.
- SerLava 10y agoYes, Google will never release their algorithm. They've bowed to pressure before, but for smaller issues like right to be forgotten. If Germany required them to disclose their algorithm, they would absolutely pull out of Germany. No question.
- Tomte 10y agoWhich would solve the problem that's the issue here. (It would create other, bigger problems for many people, but on this single point, it would be the perfect solution)