4 ms·
We can both agree on the unfortunate future. But I think we can also both agree hoping that people will abide by your robots.txt file wont save us from that un
by bnchrch 2y ago
We can both agree on the unfortunate future.
But I think we can also both agree hoping that people will abide by your robots.txt file wont save us from that unfortunate future.
- brigadier132 2y agoI think the difference is that now that the sites doing the scraping don't have a mutually beneficial relationship with the creator.
- bnchrch 2y agoNo, the difference is that theres nothing forcing you to abide by a robots.txt As long as that content is valuable it will be scraped and used. Theres no solution to this. So the structure of content sites will have to adapt. How? I dont know, likely will vary by publication and content type. Is it fair? Probably not Is it stoppable? No. Not without heavy, global legislation. (read impossible)
- jaredwiener 2y agoThis is such a weird take. If a house is robbed, you don't give the burglars a pass and say "well their security system wasn't very good." Yes, the defensive measures probably need improvement. But that doesn't mean that the act they are defending against is wrong.
- bnchrch 2y agoMy point is that it isnt a security system. Its a sign on the door that says "Please don't rob me" while the door is unlocked and no body is home.
- jaredwiener 2y agoRight. And my point is that it would still be wrong to rob that person.
- bnchrch 2y agoWe don't disagree there. In such a large ungoverned system as the internet morals don't dictate what people do / dont do. So it doesnt matter whats wrong / right. What does matter is what system participants will / wont do And they will scrape your site if you don't put up any real defences.