3 ms·
You’re missing the part about the human who interacts with the webpage.
by hk__2 2mo ago
You’re missing the part about the human who interacts with the webpage.
- ronsor 2mo agoDoesn't matter. Even by the typical robots.txt definition, a bot has to operate more or less fully autonomously. If the user initiated the request, it doesn't count, and it shouldn't follow robots.txt.