8 ms·
This attitude, and by proxy this business are the epitome of selfish entitlement. You state that you believe you deserve access to others’ resources, at their
by devmor 4mo ago
This attitude, and by proxy this business are the epitome of selfish entitlement.
You state that you believe you deserve access to others’ resources, at their cost, despite their clear attempts to stop you from using them, simply because you want it.
- irjustin 4mo agoPersonally I consider it fair game in "price wars". Dynamic pricing designed to extract every penny out. Then why shouldn't I be allowed to monitor your pricing changes?
- Banditoz 4mo agoDude, it's a web request. It's not that deep.
- devmor 4mo agoDude, as someone who runs web servers, my pockets are not that deep. I’m struggling to keep the websites of hundreds of hobbyists and small businesses alive right now because of people like this.
- kjkjadksj 4mo agoYou can already access these resources. What does it matter if you do the clicking or you have headless chrome do the clicking while you make a cup of coffee?
- baby_souffle 4mo agoThis is my entire point! And it’s why scrapers will always win; absolutely worst case, I get a screenshot of the content and have to process it further.
- fc417fc802 4mo agoI'd counter that your attitude is a techno-authoritarian one. Why should anyone have any say over how I access and use a publicly available resource? At least so long as my actions don't directly cause technical problems for the service operator.
- hirsin 4mo agoLook, it wasn't _my_ request that made the server fall over, it must have been one of the other several thousand thoughtless scrapers running on the website that caused it to die.
- fc417fc802 4mo agoIf you're claiming that the operators of high volume AI scrapers that wantonly disregard rate limits and all common sense are unethical then I'm right there with you. But that's not at all what was described upthread nor is it the only way in which bots get used by any stretch of the imagination. As far as anti-bot countermeasures go I quite like proof of work solutions since those disproportionately impact high volume scrapers without noticeably impeding a small hobby project. Unfortunately the operators of many major websites appear to want something akin to DRM with the excuse of bots used merely as window dressing.
- ssl-3 4mo agoThere was a time when a person could walk through a few department stores every week (or even every day) just to take note of some prices along the way, and ultimately tabulate them to try to identify and snatch up the best deal once it happens. And if everyone did this, it'd be a real problem. The stores would be clogged up by geeks writing notes in little books with Parker Jotters and just basically wasting space and taking up air conditioning while they sleuth out the best way to put the screws to the company for a few measly dollars. That'd be awful. But not many people ever did that in stores, and not many individual people are doing that today with the web. It's really not a problem. (And if a website in 2026 can't stand the burn of several thousand personal scrapers that are operated by people who actually want to buy stuff from it, then maybe that system simply sucks and needs to be rethought.)
- giancarlostoro 4mo agoI mean, this is how Google was built.
- cryptonym 4mo agoNot a fair statement. Google wasn't built on bypassing bot protections. Google is providing a service to the websites they crawl. They try to not crawl when we don't want them (robots.txt, clear user-agent, no-index no-follow...).
- giancarlostoro 4mo ago> Google is providing a service to the websites they crawl. Yeah they're building an LLM and making it pointless to visit the websites.
- cryptonym 4mo agoLet's agree on "Google was providing a service". Current and future state can be questionable. Btw you can still block it.