3 ms·

robots.txt is an obsolete honor system that has no relevance today. Nobody can be blamed for not having one.

• deepsun 2y ago
  Yet it's a first step to tell bots something, not a HN post.