3 ms·
> the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. That seems like something that could be slowed down slightly with li
by noja 10y ago
> the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds.
That seems like something that could be slowed down slightly with little negative impact, and a big positive impact (no annoyed people, no interaction with support).
- Animats 10y agoOur user is waiting. We rate sites on demand, and the user is looking at a rotating "busy" icon in their browser search results during the rating process.