3 ms·
Your statements are somewhat contradictory. You said that you were being scraped so you had to shut the service down due to costs... but then you also say that
by whalesalad 1y ago
Your statements are somewhat contradictory. You said that you were being scraped so you had to shut the service down due to costs... but then you also say that this not a high-bandwidth or high-volume service.
- jasonthorsness 1y agoI was the only regular user (it's basically a POC/concept site), so it was extremely low volume except for the scraper(s), who suddenly became high-volume. I could have done more research into scraper/bot blocking but I didn't have the time and needed to respond to the scrapers immediately, so I pulled the plug. EDIT: I have also since learned of Vercel's bot/scraper blocking features so I'm also going to try turning those on and see if it stops the scraping.
- nox_pp 1y agoFor https://luxweather.com https://luxweather.com I detect bots/scrapers by user agent and just serve them a fake forecast, brought our bill way down :)
- jasonthorsness 1y agoHa that's a great idea; proactive anti-bot solutions like that are hilarious