3 ms·Obituary: Farewell to robots.txt (1994-2025)14 points by donohoe 1y agomitchbob 1y agohttps://archive.ph/ZvSlq https://archive.ph/ZvSlq