Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously n

Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously need to send their page/article to Google if they want it to be indexed but may not want to send the same page content to a normal user: https://www.256kilobytes.com/content/show/1934/spoofing-your-user-agent-as-googlebot-avoid-paywalls-circumvent-log-in-requirements-and-exploit-other-quality-loopholes https://www.256kilobytes.com/content/show/1934/spoofing-your... This was common even back in the mid-2000s: https://www.avivadirectory.com/bethebot/ https://www.avivadirectory.com/bethebot/ https://developers.google.com/search/blog/2006/09/how-to-verify-googlebot https://developers.google.com/search/blog/2006/09/how-to-ver...