3 ms·
The Wayback Machine has ignored robots.txt for a few years at this point. The only way to get them to stop scraping or remove content is by asking them directly
by wl 11mo ago
The Wayback Machine has ignored robots.txt for a few years at this point. The only way to get them to stop scraping or remove content is by asking them directly.