3 ms·

Does this respect robots.txt?

• deleted 2y ago
  [deleted]
• srameshc 2y ago
  Valid question and I am sure it doesn't.
• danirod 2y ago
  I hope this gets answered. Also I've checked their docs to see if there is any mention about the user agents or IP ranges they use for scraping, with no luck.
• tompec 2y ago
  It does respect robots.txt when crawling. I'll add more details about this in the docs.
  • pryelluw 2y ago
    I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.