2 ms·
I do something similar with a different user hostile website with lots of user generated content: scraped all the content, and built a viewer. I would love to n
by ThatPlayer 3y ago
I do something similar with a different user hostile website with lots of user generated content: scraped all the content, and built a viewer. I would love to not have to, but the contributors I follow are still on the site instead of the less user hostile alternatives. Running a web scraper is easier than convincing all the users to change to a different site, and the (user-generated) content is good enough that I'm willing to spend the time to do it.
- flyinghamster 3y agoBrowser extensions (either bespoke or existing content blockers) could be another way to deal with user-hostile sites. Unfortunately, with either approach, it becomes a cat-and-mouse game the second the extension or scraper becomes popular.
- ThatPlayer 3y agoThe site I'm scraping will also just deny users access to the content regularly. It's not even a per-user rate limit, but seems to be site-wide. Unless you pay. They've recently also started banning too many requests, so a simple refresh script I used to use no longer works. So what I need isn't just a simple content blocker, but also an auxiliary database, and maybe a browser extension to pull data from that source instead.