3 ms·
Yeah and even better: share blocklists of known AI crawler IPs so we can just block them. Robots.txt is too voluntary.
by wkat4242 2y ago
Yeah and even better: share blocklists of known AI crawler IPs so we can just block them. Robots.txt is too voluntary.