3 ms·
> All a robots.txt is is a polite request to please follow the rules in it At least in my country (Germany), respecting robots.txt is a legal requirement for d
by majewsky 3y ago
> All a robots.txt is is a polite request to please follow the rules in it
At least in my country (Germany), respecting robots.txt is a legal requirement for data mining. See German Copyright Code, section 44b: https://www.gesetze-im-internet.de/urhg/__44b.html https://www.gesetze-im-internet.de/urhg/__44b.html
(IANAL)