2 ms·
Right, but how do you index a page you weren't supposed to crawl in the first place?
by LunaSea 7y ago
Right, but how do you index a page you weren't supposed to crawl in the first place?
- mijamo 7y agoIt's like recommending a book you haven't read, and newspapers do that every day. Basically Google finds the link in other places -> oh that must be interesting, I'm indexing it, without even reading it. So they don't have the actual content, and just use the texts from the sites that link to it.
- LunaSea 7y agoBut they do have the actual content since they show the meta title an description on top of what I assume is heavy NLP to drive the search engine itself.