4 ms·The law generally rewards good faith attempts, and robots.txt is an established commercial standard.by eru 2y agoThe law generally rewards good faith attempts, and robots.txt is an established commercial standard.