4 ms·
Scraping from a central server has two problems: 1) you have to pay for it 2) you can be blocked by the content providers Would it be possible to leverage yo
by bambax 16y ago
Scraping from a central server has two problems:
1) you have to pay for it
2) you can be blocked by the content providers
Would it be possible to leverage your end users instead?
While your users read articles in your app, the app probably does little; couldn't it use that (mostly idle) time to scrape other articles, and send back to your central server the scraped/cleaned content? Then all the server does is store and serve.
(Of course you'd have to warn users and check for the type of connection (only do this when the user is on wifi?) but it would be quite legitimate for your app to download content that has actually been requested by the user).
- iflyplanes 16y agoI think the best way to do this would be to inform the end users of the goal (to lower costs and provide better availability), then allow them to turn on the feature if they wish. A checkbox in the settings/preferences pane and maybe a popup option box on first startup could be enough to convince a significant number of users to help out.