2 ms·
It just means a crawler cannot retrieve the page; in other words, a crawler cannot GET a URL for whom there's an exclusion pattern in robots.txt .
by ajays 16y ago
It just means a crawler cannot retrieve the page; in other words, a crawler cannot GET a URL for whom there's an exclusion pattern in robots.txt .