4 ms·

And what about sites that do not have robots.txt? Does IA snapshot those?

• nlitened 5y ago
  Yes. By default, if you put information on the web, it is assumed that you are okay with people seeing it.