4 ms·
the first few words of the article are: > Last Sunday I discovered some abusive bot behaviour [...]
by woodrowbarlow 1y ago
the first few words of the article are:
> Last Sunday I discovered some abusive bot behaviour [...]
- mostlysimilar 1y ago> The robots.txt for the site in question forbids all crawlers, so they were either failing to check the policies expressed in that file, or ignoring them if they had.
- foobarbecue 1y agoYeah but the abusive behavior is ignoring robots.txt and scraping to train AI. Following commented URLs was not the crime, just evidence inadvertently left behind.