3 ms·
It makes sense. It's not great but it does make sense. Also, do big crawlers even observe robots.txt?
by scrum-treats 3y ago
It makes sense. It's not great but it does make sense. Also, do big crawlers even observe robots.txt?
- deleted 3y ago[deleted]