3 ms·
It does respect robots.txt when crawling. I'll add more details about this in the docs.
by tompec 2y ago
It does respect robots.txt when crawling. I'll add more details about this in the docs.
- pryelluw 2y agoI appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.