4 ms·
> Who uses service providers like this? I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I alw
by baby_souffle 4mo ago
> Who uses service providers like this?
I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS.
I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time.
I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I can do analysis.
> These kinds of services inevitably make the web more human-hostile and expensive.
I would rather not have to spend more time circumventing stupid bot detection things. I would be more than happy to pay for access to some of this data that I cannot access any other way.. but sure, let's keep burning resources on a cat and mouse game that scrapers will always be able to win.
- arianvanp 4mo agoThe litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not They do not.
- mannanj 4mo agohas anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.
- baby_souffle 4mo agoI don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS). "pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.
- inigyou 4mo ago"we don't negotiate with terrorists"
- cindyllm 4mo ago[dead]
- devmor 4mo agoThis attitude, and by proxy this business are the epitome of selfish entitlement. You state that you believe you deserve access to others’ resources, at their cost, despite their clear attempts to stop you from using them, simply because you want it.
- irjustin 4mo agoPersonally I consider it fair game in "price wars". Dynamic pricing designed to extract every penny out. Then why shouldn't I be allowed to monitor your pricing changes?
- Banditoz 4mo agoDude, it's a web request. It's not that deep.
- devmor 4mo agoDude, as someone who runs web servers, my pockets are not that deep. I’m struggling to keep the websites of hundreds of hobbyists and small businesses alive right now because of people like this.
- kjkjadksj 4mo agoYou can already access these resources. What does it matter if you do the clicking or you have headless chrome do the clicking while you make a cup of coffee?
- pwdisswordfishs 4mo ago> I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. Have you considered offering, as penitence, a public feed to share the information that this process produces?
- saltmate 4mo agoDid you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email.
- baby_souffle 4mo ago> Did you ask them for an RSS feed? Lots of people are pretty reasonable for such requests if you write a nice email. Yep! They're busy people or just don't feel the need to do anything beyond hit the "publish" button on their CMS and call it good and that's fine / why I have a robot to make an RSS for me :).
- saltmate 4mo agoToo bad, lots of CMSs have that built-in, probably a checkbox away.