3 ms·
> They keep the data and just don’t display it. I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this some
by rovr138 8y ago
> They keep the data and just don’t display it.
I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
- slededit 8y agoSure see here: https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...