4 ms·
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them dow
by TrevorFSmith 3mo ago
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
- maxglute 3mo agoI suppose it's like trying to enforce no picture rule in public.
- LoganDark 3mo agoIt's most definitely unethical.
- cr125rider 3mo agoI think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
- fooqux 3mo agoWhy do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
- theptip 3mo agoWhy do you feel entitled to dictate what user agent I use, if it’s well-behaved?
- getnormality 3mo agoWhy do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?
- Dylan16807 3mo agoSite owners should not have full control over what users do. Their wishes do not apply past certain boundaries. For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.
- pooploop64 3mo agoThere actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
- Dylan16807 3mo agoWhen we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
- theptip 3mo agoA request is a request. None of your business what I do with it after that.
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- mxkopy 3mo agoAccessibility, saving time, personal preference. Any number of reasonable things that don’t put undue stress on the host
- tadfisher 3mo agoThis page is only viewable in Internet Explorer 5.5. Please switch to a supported browser.
- stronglikedan 3mo agoIf it's just impersonating me to help me better consume the content as if I were the one driving, then it's perfectly ethical, and not even related to a bot ban. Shades of gray and all...
- matheusmoreira 3mo agoBecause I should be able to choose whatever user agent I want. The owner gets to "forbid" things on his computer, not on mine.
- est 3mo ago> Is it ethical to scrape when a site has explicitly blocked bots Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.
- yogorenapan 3mo agoI know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there. I've only ever gotten in trouble in the UK
- lemagedurage 3mo agoUse cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.