3 ms·

This is a common initial reaction, but it misses an important point: Google's process for deciding what to index isn't perfect. There's no perfect solution. Man

This is a common initial reaction, but it misses an important point: Google's process for deciding what to index isn't perfect. There's no perfect solution. Many valuable, high-quality pieces of content that people would find useful never make it into Google's index. In the past, users could refine their search keywords until they found what they wanted. This approach doesn't work as well now. The main reason? The content you're looking for might not be in the index at all. Google's selective indexing aims to reduce spam, but it also limits the diversity and depth of discoverable information.

kstrauser 2y ago
I understand what you're saying, and while I'm sympathetic to that position, you've gotta lay a lot of that at the feet of spammers who try every trick they can think of to fill the results with crap. The same thing happened with email. Maybe I want my doctor to send me information about, I don't know, herbal cholesterol treatments, but she can't because they get caught in a spam filter. Countless terabytes of junk web content make it hard to find the needle in the haystack. Frankly, I don't want to see that low-value part of the web. Every time it shows up in search results, that's one less decent result appearing on my screen. I suspect that anyone who says they want to see everything doesn't really appreciate what everything means. There will never be a page on Experts Exchange that I want to see. I'll never be grateful for a result from answers.com. Although those sites are capable of creating useful information, their business model been to dump a ton of junk on search engines with the hope that people accidentally click on it so they can make some ad money. I picked on them because I can name them off the top of my head. There are a million sites just like them. And none of them have anything to offer me whatsoever. I'd hate for Google's changes to remove the site maintained by that expert kid in Tampa who has precisely the answer I need, along with part numbers and wiring diagrams. If Google has to nuke the Experts Exchanges and answers.com of the world so that the Tampa kid's site does start showing up, yay!
emurlin 2y ago
> Many valuable, high-quality pieces of content that people would find useful never make it into Google's index. I see your point in the light of the article (not indexed = not visible), but it feels like the things that _do_ make it need to follow very particular content and style patterns to rank high. Anecdotally, this observation comes from searching for any term and seeing the results: they are usually similar-looking plausible-looking-but-actually-low-quality results that seem to follow the same or similar structure and have the same content. This does indeed limit the diversity and depth of information, but I'm not so sure it reduces spam, as these low-quality sites seem to be as prevalent as ever before, if not more. From experience writing articles to a small tech blog, this means that it's quite difficult to get well-researched articles to rank well, even if they're indexed. For example, I've written an article on how to block hotlinking (I've just checked, and Google says it's indexed). If you search for this, my article on a not-so-well-known blog is nowhere to be found(*)(**), and this is somewhat expected, for a myriad of reasons. The problem isn't that my post doesn't rank, but rather that none of the top-ranking (or even not-so-top-ranking) results are wrong. They are either about how to do this on cPanel or whatever, which is ineffective (but granted, could be what people are looking for), or instructions using the `Referer` header, which is ineffective. These days, browsers offer headers like `Cross-Origin-Resource-Policy` which can completely solve the particular issue of hotlinking, unlike `Referer` which is easily bypassed using `referrerpolicy="no-referrer"`. However, because most 'authorities' seem to be wrong on this issue, the correct result isn't displayed, because it's a hard problem to solve algorithmically (or even manually). (*) This doesn't affect just Google, though. (**) Because it's indexed, adding the right keywords (which you wouldn't do in this case unless you already knew the answer) does bring it up, although from federated high-authority sites instead of the original canonical source.

kstrauser 2y ago

I understand what you're saying, and while I'm sympathetic to that position, you've gotta lay a lot of that at the feet of spammers who try every trick they can think of to fill the results with crap. The same thing happened with email. Maybe I want my doctor to send me information about, I don't know, herbal cholesterol treatments, but she can't because they get caught in a spam filter. Countless terabytes of junk web content make it hard to find the needle in the haystack. Frankly, I don't want to see that low-value part of the web. Every time it shows up in search results, that's one less decent result appearing on my screen. I suspect that anyone who says they want to see everything doesn't really appreciate what everything means. There will never be a page on Experts Exchange that I want to see. I'll never be grateful for a result from answers.com. Although those sites are capable of creating useful information, their business model been to dump a ton of junk on search engines with the hope that people accidentally click on it so they can make some ad money. I picked on them because I can name them off the top of my head. There are a million sites just like them. And none of them have anything to offer me whatsoever. I'd hate for Google's changes to remove the site maintained by that expert kid in Tampa who has precisely the answer I need, along with part numbers and wiring diagrams. If Google has to nuke the Experts Exchanges and answers.com of the world so that the Tampa kid's site does start showing up, yay!

emurlin 2y ago

> Many valuable, high-quality pieces of content that people would find useful never make it into Google's index. I see your point in the light of the article (not indexed = not visible), but it feels like the things that _do_ make it need to follow very particular content and style patterns to rank high. Anecdotally, this observation comes from searching for any term and seeing the results: they are usually similar-looking plausible-looking-but-actually-low-quality results that seem to follow the same or similar structure and have the same content. This does indeed limit the diversity and depth of information, but I'm not so sure it reduces spam, as these low-quality sites seem to be as prevalent as ever before, if not more. From experience writing articles to a small tech blog, this means that it's quite difficult to get well-researched articles to rank well, even if they're indexed. For example, I've written an article on how to block hotlinking (I've just checked, and Google says it's indexed). If you search for this, my article on a not-so-well-known blog is nowhere to be found(*)(**), and this is somewhat expected, for a myriad of reasons. The problem isn't that my post doesn't rank, but rather that none of the top-ranking (or even not-so-top-ranking) results are wrong. They are either about how to do this on cPanel or whatever, which is ineffective (but granted, could be what people are looking for), or instructions using the `Referer` header, which is ineffective. These days, browsers offer headers like `Cross-Origin-Resource-Policy` which can completely solve the particular issue of hotlinking, unlike `Referer` which is easily bypassed using `referrerpolicy="no-referrer"`. However, because most 'authorities' seem to be wrong on this issue, the correct result isn't displayed, because it's a hard problem to solve algorithmically (or even manually). (*) This doesn't affect just Google, though. (**) Because it's indexed, adding the right keywords (which you wouldn't do in this case unless you already knew the answer) does bring it up, although from federated high-authority sites instead of the original canonical source.