3 ms·
OTOH: ByteDance intern responsible for spamming your web server with crawlers that ignore robots.txt given permanent position with a raise, now in management.
by kazinator 2y ago
OTOH: ByteDance intern responsible for spamming your web server with crawlers that ignore robots.txt given permanent position with a raise, now in management.
- 123yawaworht456 2y agohonoring robots.txt is an informal courtesy, not international law.
- davemp 2y agoNot breaking the law is just about the lowest bar you can set for an organization.
- not_a_bot_4sho 2y agoWe can go lower
- hnfong 2y agoFYI, we're still not sure whether the scraped AI training datasets involve copyright infringement.