4 ms·

Well I think you already nailed it with “not even trying”. You don’t need to try or optimize when you’re a monopoly. Competition forces that function, but a lac

Well I think you already nailed it with “not even trying”. You don’t need to try or optimize when you’re a monopoly. Competition forces that function, but a lack thereof enables mediocrity and complacency.

kristiandupont 5y ago
Oh come on! I am no fan of Google and their privacy policies etc., and I agree that results like the sites mentioned in this thread are annoying but I am not so cocky as to think I have any idea what Google is fighting. There are a lot of very smart people from all over the world putting everything they have into getting to the top of that site. It's not a trivial task for them.
- kizer 5y ago
  Fair point. But again, that would be less of an issue if there were like 3 or 4 big search engines with different algorithms, all changing and adapting on their own schedules. Would be harder to game the system then.
- motoxpro 5y ago
  You're so right. It's sad that even if you remove 99% of something, to the outside it looks like you are removing 0% because you have no reference.
- Nextgrid 5y ago
  I disagree. Google has absolutely dropped the ball - I'm not saying that it's easy to algorithmically rank good/bad content, but they're not even trying. A stop-gap solution would be to simply penalize anything with ads. Legitimate websites will still have enough "SEO juice" (for the lack of a better term) to offset the penalty, but brand new copycats with otherwise no inbound links to them from other legitimate websites (and no other business model beyond scammy ads) will never be able to rank high, essentially removing the incentive from setting up these sites in the first place. Not to mention, these problems can be identified, prioritized and then tackled manually one-by-one. Stackoverflow copycats can be dealt with by downloading the SO data dump, parsing it and then severely penalizing any website where the bulk of the content matches the dump. Pinterest can be penalized by simply excluding it from image searches until they actually display the searched image without asking for login. So on and so forth. You can't win them all, and you can't train an algorithm to win every time either, but you can manually observe what's happening and deal with the biggest problems. The problem however is that the current status-quo is good for Google. They've got the monopoly on search (every other one is typically even worse when it comes to these issues), and spammy copycat websites happen to have Google ads or analytics so Google actually benefits from them too.
  - Kiro 5y ago
    How do you know Google is not already doing all those things? You said it yourself, you can't win them all and we're witnessing the result of that. If Google truly did nothing it would be an unimaginable chaos.
    - Nextgrid 5y ago
      Because I’m not seeing the results. Let’s ignore SO copycats for now because that solution requires at least a slight amount of effort and look at the elephant in the room - Pinterest. Dealing with it is a simple “if image_search and domain == pinterest.com then skip” and they’re not even doing that. They have a site that’s been polluting image search results for years, isn’t even trying to hide (unlike the SO copycats which could technically churn out an infinite amount of domains to work around bans) and they can’t even deal with that.
  - TheOtherHobbes 5y ago
    Well, quite. Why would a company that makes its money selling ads penalise sites that show its ads? And when you get to that state, you've pretty much turned the entire web into content marketing for your ad sales business. So - of course search quality suffers. Search quality is not the point.
- MrWiffles 5y ago
  Consider it from a management/budgetary point of view. Super smart people? Oh hell yeah, and I don't fault them for this one bit. But bean counters? Why the hell invest in hiring those people, or budgeting for search quality improvements, when you have essentially no competition to speak of? Makes more sense to put those same smart people on projects that correlate people's activities outside of Google with activities they can observe either directly or indirectly to create a more wholistic picture of the consumer for advertisers. Those are much harder problems to solve, especially now that the public is finally becoming slowly more aware of the massive abuses of privacy the bean counters like I described above have been pushing for years. At the end of the day Google doesn't exist for the good of the public, they exist to return the highest possible investment for shareholders. It's literally the opposite of their job to improve search quality when competition doesn't force them to (because investors see that as a waste of money that could have been spent on further eroding people's privacy to enable higher and higher ad revenues.)
rhdunn 5y ago
One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions. Related is if you have a lot of content from git repositories that are mirrored from different locations (GitHub, GitLab, etc.), all of which are showing the same content. Or if different sites are hosting versions of public domain texts. You don't want to derank those results, even if they are similar to the "copy a popular site" websites.
- lazyeye 5y ago
  Google knows instantly how much time you spend on a page and how quickly you return to the search results because it is a useless copy/paste page from SO.
  - cutemonster 5y ago
    And probably they track content age too? Since they somehow can lookup by content (they can detect duplicated content), there's probably a date remembered too So they could know if a code snippet was already X years old? And from where, originally. (Unless it got edited a lot but that's not the case here?)
- MrWiffles 5y ago
  > One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions. The core function of that is actually pretty simple: 1. Strip all X/HTML tags 2. Run `diff` Sure, it's not perfect, but an organization that pursues academic quantum computing research can sure as hell afford to run the results of the above against an AI to check for similarities.
  - rhdunn 5y ago
    The issue is not figuring out if site X has a copy of a page or text from site Y. The issue is how do you know which site is the source of the information. In other words, the technical bit is easy but determining what to do with that is hard. Especially if you don't want a lot of hard coded exceptions. In the case of things like other sites copying from stack overflow, github, etc. you can figure out which is the source of the information. Lets day you have decided to make github the source of that information, and derank any other sites that also have that information. As a result, you will derank gitlab for having git mirrors of projects, as the source will match that on github. You will also derank sites like lkml as that contains the commit message descriptions, and the patches will partially match content from the kernel source hosted on github. Lets also say you've decided to make wikipedia and other wikimedia sites the source of information. Congratulations, you've now deranked project gutenberg for hosting the same public domain texts as wikimedia, along with other sites like the official sites for authors like Jack London. Plus any blog that includes portions of these to discuss or analyze them. Do you want a blackbox AI making these decisions?

kristiandupont 5y ago

Oh come on! I am no fan of Google and their privacy policies etc., and I agree that results like the sites mentioned in this thread are annoying but I am not so cocky as to think I have any idea what Google is fighting. There are a lot of very smart people from all over the world putting everything they have into getting to the top of that site. It's not a trivial task for them.

kizer 5y ago
Fair point. But again, that would be less of an issue if there were like 3 or 4 big search engines with different algorithms, all changing and adapting on their own schedules. Would be harder to game the system then.
motoxpro 5y ago
You're so right. It's sad that even if you remove 99% of something, to the outside it looks like you are removing 0% because you have no reference.
Nextgrid 5y ago
I disagree. Google has absolutely dropped the ball - I'm not saying that it's easy to algorithmically rank good/bad content, but they're not even trying. A stop-gap solution would be to simply penalize anything with ads. Legitimate websites will still have enough "SEO juice" (for the lack of a better term) to offset the penalty, but brand new copycats with otherwise no inbound links to them from other legitimate websites (and no other business model beyond scammy ads) will never be able to rank high, essentially removing the incentive from setting up these sites in the first place. Not to mention, these problems can be identified, prioritized and then tackled manually one-by-one. Stackoverflow copycats can be dealt with by downloading the SO data dump, parsing it and then severely penalizing any website where the bulk of the content matches the dump. Pinterest can be penalized by simply excluding it from image searches until they actually display the searched image without asking for login. So on and so forth. You can't win them all, and you can't train an algorithm to win every time either, but you can manually observe what's happening and deal with the biggest problems. The problem however is that the current status-quo is good for Google. They've got the monopoly on search (every other one is typically even worse when it comes to these issues), and spammy copycat websites happen to have Google ads or analytics so Google actually benefits from them too.
- Kiro 5y ago
  How do you know Google is not already doing all those things? You said it yourself, you can't win them all and we're witnessing the result of that. If Google truly did nothing it would be an unimaginable chaos.
  - Nextgrid 5y ago
    Because I’m not seeing the results. Let’s ignore SO copycats for now because that solution requires at least a slight amount of effort and look at the elephant in the room - Pinterest. Dealing with it is a simple “if image_search and domain == pinterest.com then skip” and they’re not even doing that. They have a site that’s been polluting image search results for years, isn’t even trying to hide (unlike the SO copycats which could technically churn out an infinite amount of domains to work around bans) and they can’t even deal with that.
- TheOtherHobbes 5y ago
  Well, quite. Why would a company that makes its money selling ads penalise sites that show its ads? And when you get to that state, you've pretty much turned the entire web into content marketing for your ad sales business. So - of course search quality suffers. Search quality is not the point.
MrWiffles 5y ago
Consider it from a management/budgetary point of view. Super smart people? Oh hell yeah, and I don't fault them for this one bit. But bean counters? Why the hell invest in hiring those people, or budgeting for search quality improvements, when you have essentially no competition to speak of? Makes more sense to put those same smart people on projects that correlate people's activities outside of Google with activities they can observe either directly or indirectly to create a more wholistic picture of the consumer for advertisers. Those are much harder problems to solve, especially now that the public is finally becoming slowly more aware of the massive abuses of privacy the bean counters like I described above have been pushing for years. At the end of the day Google doesn't exist for the good of the public, they exist to return the highest possible investment for shareholders. It's literally the opposite of their job to improve search quality when competition doesn't force them to (because investors see that as a waste of money that could have been spent on further eroding people's privacy to enable higher and higher ad revenues.)

rhdunn 5y ago

One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions. Related is if you have a lot of content from git repositories that are mirrored from different locations (GitHub, GitLab, etc.), all of which are showing the same content. Or if different sites are hosting versions of public domain texts. You don't want to derank those results, even if they are similar to the "copy a popular site" websites.

lazyeye 5y ago
Google knows instantly how much time you spend on a page and how quickly you return to the search results because it is a useless copy/paste page from SO.
- cutemonster 5y ago
  And probably they track content age too? Since they somehow can lookup by content (they can detect duplicated content), there's probably a date remembered too So they could know if a code snippet was already X years old? And from where, originally. (Unless it got edited a lot but that's not the case here?)
MrWiffles 5y ago
> One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions. The core function of that is actually pretty simple: 1. Strip all X/HTML tags 2. Run `diff` Sure, it's not perfect, but an organization that pursues academic quantum computing research can sure as hell afford to run the results of the above against an AI to check for similarities.
- rhdunn 5y ago
  The issue is not figuring out if site X has a copy of a page or text from site Y. The issue is how do you know which site is the source of the information. In other words, the technical bit is easy but determining what to do with that is hard. Especially if you don't want a lot of hard coded exceptions. In the case of things like other sites copying from stack overflow, github, etc. you can figure out which is the source of the information. Lets day you have decided to make github the source of that information, and derank any other sites that also have that information. As a result, you will derank gitlab for having git mirrors of projects, as the source will match that on github. You will also derank sites like lkml as that contains the commit message descriptions, and the patches will partially match content from the kernel source hosted on github. Lets also say you've decided to make wikipedia and other wikimedia sites the source of information. Congratulations, you've now deranked project gutenberg for hosting the same public domain texts as wikimedia, along with other sites like the official sites for authors like Jack London. Plus any blog that includes portions of these to discuss or analyze them. Do you want a blackbox AI making these decisions?