3 ms·
Sure, depends on how accessibly to people you want it to be. Most legit search engines are going to honor robots.txt and you can disallow access. Next level w
by matt_heimer 5mo ago
Sure, depends on how accessibly to people you want it to be.
Most legit search engines are going to honor robots.txt and you can disallow access.
Next level would be using something like rate limiting controls and/or Cloudflare's bot fight mode to start blocking the bad bots. You start to annoy some people here.
Next would be putting the content behind some form of auth.
- cute_boi 5mo agoI don't know why we are trusting cloudflare when they are the one creating crawlers. https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/ https://developers.cloudflare.com/browser-run/quick-actions/...