5 ms·

> Apparently, only the information that maximizes search advertising revenue is worth organizing and making accessible. I don't even understand the theory of h

> Apparently, only the information that maximizes search advertising revenue is worth organizing and making accessible. I don't even understand the theory of how there is a profit motive here. How does Google make more money by asking webmasters to hide parts of their site from the indexer? Doesn't Occam's Razor say that a more likely explanation is that innocent sites are being caught in the never-ending arms race between spammers and Google? If Google didn't care about the "indie" web, why would Google employees be taking their time to coach indie website operators on how to improve their site's rankings? I know this is a frustrating situation for everyone involved. But the cynicism that says there is some ulterior motive cuts deep, especially coming from someone I respect like you, Matt.

fleitz 12y ago
It's not an ulterior motive, profit is their stated objective.
sunir 12y ago
Hey Josh, I'm genuinely confused by your statement and now curious. Even though you work at Google, you do not see the profit motive in making better search result pages and why Google is investing heavily in improving the search result pages. I would imagine you actually do see the profit motive for that activity, and yet you rejected the statement above by saying, "I don't even understand the theory of how there is a profit motive here. How does Google make more money by asking webmasters to hide parts of their site from the indexer?" Could you explain how these things connect in your view?
- skybrian 12y ago
  Google has always said that advertising does not affect search results. This is one way Google distinguished themselves from other search engines back in the day. There are sometimes quite a lot of ads on the search results page these days, but they are still clearly distinguished, and I haven't heard anything about that policy changing.
  - tripzilch 12y ago
    Gonna disagree with "clearly distinguished". Most non-tech-savvy people I know do not tell them apart. Especially since the light yellow background got even lighter with time until it's hardly distinguishable on a cheap screen.
- haberman 12y ago
  Sorry, I may not have been clear enough. What didn't make sense to me is why there would be a profit motive for ignoring or not valuing "indie" sites.
mbrubeck 12y ago
The cynical side of my concern is that search engines under pressure due to spammers will let niche/indie content bear the brunt of the cost, because that content is not "high quality" from the search engine perspective of "we have valuable ad inventory that lets us monetize these searches." Not all searches are profitable. Similarly, "sophisticated" users are less lucrative because we click fewer ads. A bias toward readily available metrics can easily lead to algorithm changes that benefit the mainstream majority while punishing various niche use cases as regretted but ultimately acceptable collateral damage. The conflict is between "the world's information" (mission) and "the world's information, minus some troublesome long-tail bits that few people care about and aren't profitable anyway" (business motive). > why would Google employees be taking their time to coach indie website operators on how to improve their site's rankings This "coaching" is actually Google pushing the spam problem onto content authors, promoting "solutions" that benefit Google while still hurting legit publishers. "Spam has gotten so bad that our algorithms can't classify it without misclassifying you. You can either hide some of your content as a proof that it's not link spam, or have us hide even more on presumption that it is." Yes, this gives Google a powerful tool for weeding out fake/malicious content, but it also hurts the user experience and the mission in the long run, compared to solutions that allow for more manual review, more use of inbound link disavowal, or finer-grained penalties that target only the possibly "low-quality" pages without spilling over to entire sites. Those solutions would probably shift more cost to Google, however. What's really bad is that some of Google's recent recommendations (e.g. "use the noindex robots meta tag") will remove real knowledge not just from Google but from other present and future search engines (even if those engines wouldn't have the same problems Google did with that content). And even if the algorithm's limitations/errors are eventually resolved, the recommendations will live on much longer in legacy content (and in SEO folklore), meaning that some content will be erased from search permanently for no reason. These are long-term costs that are borne not by Google but by users, publishers, and competing search engines. Personally, I want to be able to search ALL THE THINGS even if it means that I sometimes have to wade through more crap, and that it's harder work for the search engine to weed out people who are gaming the system. Maybe Google can't offer me that without degrading its more mainstream users. Even if that's the case, it still feels like a departure from principles (or maybe a declaration of defeat) for Google to be making that choice. Even more so when it hurts the ability of other search engines to step in and index the stuff that Google doesn't want. The reason I want to see Google hew strictly to "the mission" is that it offers a way to let these big-picture issues override things that are more immediately measurable. I know I became an extensive user of Google services mainly because of the way its mission-oriented thinking was visible in its product design choices. That intangible feeling was more important than any particular feature. Keeping it alive requires the ability to say, "Screw it, we're not going to go this route no matter its obvious benefits, simply because it's not in line with the mission." But I also know that it's not possible to do that in every decision; even here in the non-profit, mission-driven Mozilla project, principles alone can't win every fight. (See http://robert.ocallahan.org/2014/05/unnecessary-dichotomy.html http://robert.ocallahan.org/2014/05/unnecessary-dichotomy.ht... for an interesting take on this.) And these things tend to look different from the inside; maybe Panda was a case where de-indexing legit content was the least-bad path available. But from the outside, every time "pragmatism" wins over "principle" is a little chink in the armor. And for Google, the outside view is the only one I have.
mrshoe 12y ago
> I know this is a frustrating situation for everyone involved. But the cynicism that says there is some ulterior motive cuts deep, especially coming from someone I respect like you, Matt. Ha! HN loves GOOG. Try frequenting this site as an Apple employee...
- jfb 12y ago
  You get used to it.
rasz_pl 12y ago
>I don't even understand the theory of how there is a >profit motive here. > >How does Google make more money by asking webmasters to >hide parts of their site from the indexer? Maybe they get paid by competition. After all removing competition with clever TOS tricks is right up Googles alley: http://pando.com/2014/05/29/after-google-bought-nest-it-removed-one-of-the-companys-biggest-competitors-from-search-results/ http://pando.com/2014/05/29/after-google-bought-nest-it-remo... Looks like google saw what bookface was doing and copied. Remove from existence and charge for exposure.

fleitz 12y ago

It's not an ulterior motive, profit is their stated objective.

sunir 12y ago

Hey Josh, I'm genuinely confused by your statement and now curious. Even though you work at Google, you do not see the profit motive in making better search result pages and why Google is investing heavily in improving the search result pages. I would imagine you actually do see the profit motive for that activity, and yet you rejected the statement above by saying, "I don't even understand the theory of how there is a profit motive here. How does Google make more money by asking webmasters to hide parts of their site from the indexer?" Could you explain how these things connect in your view?

skybrian 12y ago
Google has always said that advertising does not affect search results. This is one way Google distinguished themselves from other search engines back in the day. There are sometimes quite a lot of ads on the search results page these days, but they are still clearly distinguished, and I haven't heard anything about that policy changing.
- tripzilch 12y ago
  Gonna disagree with "clearly distinguished". Most non-tech-savvy people I know do not tell them apart. Especially since the light yellow background got even lighter with time until it's hardly distinguishable on a cheap screen.
haberman 12y ago
Sorry, I may not have been clear enough. What didn't make sense to me is why there would be a profit motive for ignoring or not valuing "indie" sites.

mbrubeck 12y ago

The cynical side of my concern is that search engines under pressure due to spammers will let niche/indie content bear the brunt of the cost, because that content is not "high quality" from the search engine perspective of "we have valuable ad inventory that lets us monetize these searches." Not all searches are profitable. Similarly, "sophisticated" users are less lucrative because we click fewer ads. A bias toward readily available metrics can easily lead to algorithm changes that benefit the mainstream majority while punishing various niche use cases as regretted but ultimately acceptable collateral damage. The conflict is between "the world's information" (mission) and "the world's information, minus some troublesome long-tail bits that few people care about and aren't profitable anyway" (business motive). > why would Google employees be taking their time to coach indie website operators on how to improve their site's rankings This "coaching" is actually Google pushing the spam problem onto content authors, promoting "solutions" that benefit Google while still hurting legit publishers. "Spam has gotten so bad that our algorithms can't classify it without misclassifying you. You can either hide some of your content as a proof that it's not link spam, or have us hide even more on presumption that it is." Yes, this gives Google a powerful tool for weeding out fake/malicious content, but it also hurts the user experience and the mission in the long run, compared to solutions that allow for more manual review, more use of inbound link disavowal, or finer-grained penalties that target only the possibly "low-quality" pages without spilling over to entire sites. Those solutions would probably shift more cost to Google, however. What's really bad is that some of Google's recent recommendations (e.g. "use the noindex robots meta tag") will remove real knowledge not just from Google but from other present and future search engines (even if those engines wouldn't have the same problems Google did with that content). And even if the algorithm's limitations/errors are eventually resolved, the recommendations will live on much longer in legacy content (and in SEO folklore), meaning that some content will be erased from search permanently for no reason. These are long-term costs that are borne not by Google but by users, publishers, and competing search engines. Personally, I want to be able to search ALL THE THINGS even if it means that I sometimes have to wade through more crap, and that it's harder work for the search engine to weed out people who are gaming the system. Maybe Google can't offer me that without degrading its more mainstream users. Even if that's the case, it still feels like a departure from principles (or maybe a declaration of defeat) for Google to be making that choice. Even more so when it hurts the ability of other search engines to step in and index the stuff that Google doesn't want. The reason I want to see Google hew strictly to "the mission" is that it offers a way to let these big-picture issues override things that are more immediately measurable. I know I became an extensive user of Google services mainly because of the way its mission-oriented thinking was visible in its product design choices. That intangible feeling was more important than any particular feature. Keeping it alive requires the ability to say, "Screw it, we're not going to go this route no matter its obvious benefits, simply because it's not in line with the mission." But I also know that it's not possible to do that in every decision; even here in the non-profit, mission-driven Mozilla project, principles alone can't win every fight. (See http://robert.ocallahan.org/2014/05/unnecessary-dichotomy.html http://robert.ocallahan.org/2014/05/unnecessary-dichotomy.ht... for an interesting take on this.) And these things tend to look different from the inside; maybe Panda was a case where de-indexing legit content was the least-bad path available. But from the outside, every time "pragmatism" wins over "principle" is a little chink in the armor. And for Google, the outside view is the only one I have.

mrshoe 12y ago

> I know this is a frustrating situation for everyone involved. But the cynicism that says there is some ulterior motive cuts deep, especially coming from someone I respect like you, Matt. Ha! HN loves GOOG. Try frequenting this site as an Apple employee...

jfb 12y ago
You get used to it.

rasz_pl 12y ago

>I don't even understand the theory of how there is a >profit motive here. > >How does Google make more money by asking webmasters to >hide parts of their site from the indexer? Maybe they get paid by competition. After all removing competition with clever TOS tricks is right up Googles alley: http://pando.com/2014/05/29/after-google-bought-nest-it-removed-one-of-the-companys-biggest-competitors-from-search-results/ http://pando.com/2014/05/29/after-google-bought-nest-it-remo... Looks like google saw what bookface was doing and copied. Remove from existence and charge for exposure.