3 ms·
User-agents can be used for web-servers to differentiate between bots/crawlers and actual human beings. But there could be a generic user-agent instead of the o
by ClikeX 7y ago
User-agents can be used for web-servers to differentiate between bots/crawlers and actual human beings. But there could be a generic user-agent instead of the ones we have now.
Note: This only matters for bots that serve to aid your website. Like search engine crawlers being given instructions what pages to ignore.
- Jamwinner 7y agoMost crawlers/bots, particularly the ones you want to be able to identify, spoof their user agent, making this mostly moot.