3 ms·
the facebook bug is funny but the real issue is that robots.txt is a gentleman's agreement from 1994 and half the crawlers in 2026 don't care what it says.
by _slih 7mo ago
the facebook bug is funny but the real issue is that robots.txt is a gentleman's agreement from 1994 and half the crawlers in 2026 don't care what it says.