3 ms·
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve webs
by rao-v 3mo ago
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
- Catloafdev 3mo agoIf it weren't a real problem, these types of articles and services wouldn't exist.
- inigyou 3mo agoPlenty of complaints exist about things that are not real problems.
- Catloafdev 3mo agoWell that's not what's happening here, lol.
- TZubiri 3mo agoWhy the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.
- inigyou 3mo agoSurely that depends on the motivation for the ban.
- hananova 3mo agoNo. When someone says you’re not welcome, you’re not welcome. Regardless of the reason.
- inigyou 3mo agoIf I get kicked out of Epstein island because I refuse to **** a child, that doesn't make me a bad actor.
- TZubiri 3mo agoAnd then going there 1 million times under fake identities? Yeah, I'm sure that's not a bad actor.
- hananova 3mo agoThank you, you get it. Either you get shown the door because you refuse to take part in unethical behavior, in which case you wouldn’t want to return anyway. Or you get shown the door because you’re simply not welcome or because YOU are the unethical one, in which case no means no. I’m sure someone will be able to come up with some kind of edge case where this logic doesn’t work, but that doesn’t matter. This is about websites saying no to agents, bots, and scrapers. And no means no.
- inigyou 3mo agoWhat if I'm a journalist trying to expose what's happening on the island?
- TZubiri 3mo agoEver heard the phrase that the exception proves the rule?
- trucks-refinish 3mo ago[dead]
- duskwuff 3mo ago> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
- wiredfool 3mo agoWhat’s more, they will scale up with increased resources on the site. If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
- solid_fuel 3mo agoI don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.