3 ms·
The demo is magical. Excellent work! One question about this FAQ: > Does ParseHub respect robots.txt? > We're working on an admin panel to give webmast
by ihiram 12y ago
The demo is magical. Excellent work!
One question about this FAQ:
> Does ParseHub respect robots.txt?
> We're working on an admin panel to give webmasters full transparency and control. We'll have more info soon.
What does that mean? I read it as ParseHub does not respect robots.txt, which as a content owner is a bit disappointing. Would you elaborate on your thinking?
- tsergiu 12y agoSorry, we'll make the wording more clear. At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.
- spacefight 12y agoPlease add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.