3 ms·
Or in Germany, see §44b UrhG. [1] https://www.gesetze-im-internet.de/urhg/__44b.html https://www.gesetze-im-internet.de/urhg/__44b.html
by mschuster91 3y ago
Or in Germany, see §44b UrhG.
[1] https://www.gesetze-im-internet.de/urhg/__44b.html https://www.gesetze-im-internet.de/urhg/__44b.html
- nmlracx 3y agoIt seems to me that Twitter's robots.txt only allows Googlebot: https://twitter.com/robots.txt https://twitter.com/robots.txt Therefore, this disallows other bots in "maschinenlesbarer Form" and the scraping is illegal.
- mschuster91 3y agoThis is fascinating. Seems like at least Bing openly ignores robots.txt?
- gnfargbl 3y agoGoogle is, to a first approximation, the web's only search engine. As such I think there's an argument to be made that declaring your bot's User-Agent as Googlebot is a technical necessity, similar to how every browser declares itself as Mozilla.
- deleted 3y ago[deleted]