4 ms·

I'm not surprised. Some people think humans read robots.txt and get super angry when the crawler doesn't understand.

• c3534l 7y ago
  I read robots.txt, but I'm not a massive corporation.
• deleted 7y ago
  [deleted]