3 ms·
"I live in fear of an AI scraper figuring out how to scrape all of these files and bankrupting me overnight." What is the best way around this for a hobby proj
by weinzierl 2y ago
"I live in fear of an AI scraper figuring out how to scrape all of these files and bankrupting me overnight."
What is the best way around this for a hobby project similar to botsin.space.
I don't mind the service going down in case of a DOS attack. I want to handle TLS myself though (so no free Cloudflare).
Most important thing is my good sleep at night, so no fine print that allows the provider to pass on the cost to me in case something goes wrong. (If that means higher fixed cost, that's how it is, I'm not asking for a dream house, just reliable cost control).
- _fat_santa 2y agoI could see a huge business opportunity here. In Amazon S3 there is a concept of "requestor pays" so you can have an S3 bucket that loads of people download data from yet you aren't hit with the bill and instead the "requestor" pays for the bandwidth. With AI scraping becoming more of a thing, a cloud platform could roll out a feature where an AI is allowed to scrape the daylights out of your site, but they must pay for the bandwidth or bandwidth + premium. In this scenario you wouldn't wake up bankrupt but instead with a windfall of cash because TikTok decided to scrape all your stuff.
- kraftman 2y agoA dedicated server with no bandwidth cap?