4 ms·
No, disallow means that you are not allowed to crawl the page. You have to crawl the page to no you cannot index it. But how do you index it if you do not crawl
by bryanrasmussen 7y ago
No, disallow means that you are not allowed to crawl the page. You have to crawl the page to no you cannot index it. But how do you index it if you do not crawl the page, well if another page that you can crawl and index points to the page you cannot index as authoritative on a keyword then it be in the index with that keyword, even if you do not have the actual crawled content of the page.
- henryfjordan 7y agoIt really feels like they need to allow a list of `noindex` pages in the robots.txt then...
- wayoutthere 7y agoThe whole point is they don’t want you to easily opt out of it.
- dredmorbius 7y agoKeep walking down that path to find v. heavy regulation.
- wayoutthere 7y agoNot if you buy the government first. I think they’re already a victim of their own greed though. It’s bad enough I started using DDG for search because the results are now more relevant. Google’s advertising algorithms are designed to subtly nudge sites into paying for placement — which means there’s a “non-content” element to the search results that makes it into the user experience. I feel like there was a tipping point a year or two ago where the results just stopped being useful — The best analogy I can find is how search engines used to be in the days before AltaVista. Then AltaVista came out and the results were far more relevant (if not perfect). Google -> DDG feels like that in 2019. That “non-content” element will only grow over time as Google seeks revenue growth — growth across all of Google’s non-advertising revenue streams combined are not enough to move the needle compared to the scale their ad business has — of which search ads are by far the most profitable. So they will further try to monetize search; it’s their cash cow but I think a small player like DDG could easily overtake them as the quality of Google’s search results (to the end user) continue to decline.
- dredmorbius 7y agoAgreed re: DDG search quality. It's my own default and preferred choice. Google remains useful for Scholar and Books, but relevance and deceptive ads on SERPS is rapidly declining.
- LunaSea 7y agoRight, but how do you index a page you weren't supposed to crawl in the first place?
- mijamo 7y agoIt's like recommending a book you haven't read, and newspapers do that every day. Basically Google finds the link in other places -> oh that must be interesting, I'm indexing it, without even reading it. So they don't have the actual content, and just use the texts from the sites that link to it.
- LunaSea 7y agoBut they do have the actual content since they show the meta title an description on top of what I assume is heavy NLP to drive the search engine itself.