2 ms·
This is an old problem I could have sworn there were promises they were going to change their procedures. The question I have is how fast is the content remove
by 1vuio0pswjnm7 5y ago
This is an old problem I could have sworn there were promises they were going to change their procedures.
The question I have is how fast is the content removed after the domain name registration changes, i.e., is there is a window of time between the appearance of a new robots.txt and the next scheduled crawl, and if so, is it be possible to "rescue" the content, as ArchiveTeam would do, during that window, before it disappears.
If this is possible, there could be a service for monitoring changes to domain name registrations for sites that have large amounts of historical content. I would happily volunteer to set up such a service.
- joe_the_user 5y agoWell, "complain on hn" has been a way to get stuff from Google. Maybe someone at archive will notice this thread.
- hidden-spyder 5y agoI'm curious. What changes has Google made due to complains on HN?