3 ms·
Sorry, we'll make the wording more clear. At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future
by tsergiu 12y ago
Sorry, we'll make the wording more clear.
At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.
- spacefight 12y agoPlease add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.