3 ms·
Do AI crawlers actually follow robots.txt rules? They didn't care about rules when they trained them on copyrighted works so why would they follow the rules her
by hiccuphippo 4mo ago
Do AI crawlers actually follow robots.txt rules? They didn't care about rules when they trained them on copyrighted works so why would they follow the rules here?