3 ms·
I was the only regular user (it's basically a POC/concept site), so it was extremely low volume except for the scraper(s), who suddenly became high-volume. I co
by jasonthorsness 1y ago
I was the only regular user (it's basically a POC/concept site), so it was extremely low volume except for the scraper(s), who suddenly became high-volume. I could have done more research into scraper/bot blocking but I didn't have the time and needed to respond to the scrapers immediately, so I pulled the plug.
EDIT: I have also since learned of Vercel's bot/scraper blocking features so I'm also going to try turning those on and see if it stops the scraping.
- nox_pp 1y agoFor https://luxweather.com https://luxweather.com I detect bots/scrapers by user agent and just serve them a fake forecast, brought our bill way down :)
- jasonthorsness 1y agoHa that's a great idea; proactive anti-bot solutions like that are hilarious