4 ms·
Perhaps access to archived pages should only be blocked when the ia_archiver user agent specifically is denied in the robots.txt. That way they aren't inadverte
by ldjb 10y ago
Perhaps access to archived pages should only be blocked when the ia_archiver user agent specifically is denied in the robots.txt. That way they aren't inadvertently blocked by a generic robots.txt that denies everything (which sometimes occurs with parked domains), but there's still a way to deny the Wayback Machine if you really need to.
- JoshTriplett 10y agoThis, exactly. Or, even better, use a separate user-agent ("ia_retroactive" or similar) for retroactive removals.
- smsm42 10y agoWell, then everybody would claim the same right and you'd have to maintain full list of bots and keep it up to date. This doesn't sound scalable, * should still mean "everybody".
- ldjb 10y agorobots.txt only tells bots not to crawl the website. It doesn't say anything about indexing or archival of pages. It would be perfectly possible to have the Wayback Machine respect the robots.txt and not have it crawl or archive any new pages, whilst making pages that have already been archived accessible unless a specific user agent has been denied.