Here's how to make the site accessible to search engines, but not common crawl bots: https://commoncrawl.org/big-picture/frequently-asked-questions/ https://co

Here's how to make the site accessible to search engines, but not common crawl bots: https://commoncrawl.org/big-picture/frequently-asked-questions/ https://commoncrawl.org/big-picture/frequently-asked-questio... e.g.: User-agent: CCBot Disallow: /