3 ms·
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU a
by Findecanor 2y ago
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags?
These are getting more important now, especially in the EU after the DSM directive.
- jancurn 2y agoNot yet, but it’s on the roadmap