3 ms·
All of the crawlers present themselves as being from one of the major companies, even if they’re not. Setting user-agent headers is easy.
by Aurornis 2y ago
All of the crawlers present themselves as being from one of the major companies, even if they’re not.
Setting user-agent headers is easy.
- cyanydeez 2y agoat the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.
- CaptainFever 2y agoI highly doubt that people are using AI agent features so frequently and so concentrated-ly that it brings down websites.