28 ms·
The crawler still hits the actual "URL" he found, so that provides sanity checking. It's just another way to discover (potential) new URLs.
by anonymouz 14y ago
The crawler still hits the actual "URL" he found, so that provides sanity checking. It's just another way to discover (potential) new URLs.
- protomyth 14y agoI see your point and the sanity check is good. My problem with the whole thing is that interpreting text is fraught with problems and I just don't see the value from a search perspective. Something served as plain/text either is a problem with the server or just a plain text file. Either way, it seems like a poor value thing to add to a search.