3 ms·
so it's ok if they ignore robots.txt and just vacuum up every scrap of data they can, but not when someone else does it to them, iiuc
by vrighter 2y ago
so it's ok if they ignore robots.txt and just vacuum up every scrap of data they can, but not when someone else does it to them, iiuc