2 ms·
Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://devel
by wilg 2mo ago
Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth https://developers.google.com/crawling/docs/crawlers-fetcher...
Seems like the crawler companies would be incentivized to not want to take responsibility for people spoofing their user agents.
- gavinhking 2mo agoYup, in fact most of them are already. That's one of the ways this data is verifying whether the visits are spoofed or not: https://knownagents.com/insights#spoofing-and-security https://knownagents.com/insights#spoofing-and-security