3 ms·
It ticks me off that bots no longer respect robots.txt files at all. The authors of these things are complete assholes. If you’re one of them, gfy.
by ThomW 1y ago
It ticks me off that bots no longer respect robots.txt files at all. The authors of these things are complete assholes. If you’re one of them, gfy.
- cyanydeez 1y ago[flagged]
- whatevermom 1y ago[flagged]
- oblio 1y agoWhy do you do it?
- jeffhuys 1y agoProfit in terms of knowledge or just actual money (guessing)
- whatevermom 1y agoFor Fun & Profit
- thowaway7564902 1y agoYou might as well say gfy to anyone using chat bots, search engines and price comparison tools. They're the one's who financially incentivize the scrapers.
- mandmandam 1y agoThat doesn't logic. Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself. I wonder where you ever got the idea that it does.
- braiamp 1y agoBecause there's a market to serve, that you refuse to serve, so the stop gap solution is for third parties to acquire the liabilities and risks for compensation.
- mandmandam 1y agoFinancial incentive ≠ moral justification. You're shifting the frame, to one where morality doesn't come into it. You're asserting some kind of market inevitability, which is probably the same sort of rationalization arms and people traffickers use to sleep at night.
- hinkley 1y agoHoney potting the robots file is handy for the ones who don’t just ignore it but go looking for trouble.