3 ms·
Just an (evil) idea: Create some free iPhone app/game, scrape the site in user's iPhone and send to your server.
by tszming 15y ago
Just an (evil) idea: Create some free iPhone app/game, scrape the site in user's iPhone and send to your server.
- gtani 15y agoLess evil: read about how duckduckgo uses Boss and Bing APIs http://www.gabrielweinberg.com/blog/2010/08/thoughts-on-yahoo-boss-monetization-ii.html http://www.gabrielweinberg.com/blog/2010/08/thoughts-on-yaho... ----------------- and check http://www.reddit.com/r/datasets http://www.reddit.com/r/datasets and infochimps, maybe dataset is already out there -------------------- and ...never spider from home, you'll always screw something up: referrer HTTP header, user agent, random wait time between requests, http://streamhacker.com/2010/10/04/perils-web-crawling/ http://streamhacker.com/2010/10/04/perils-web-crawling/ http://news.ycombinator.com/item?id=2463058 http://news.ycombinator.com/item?id=2463058 see first comment
- bengl3rt 15y agoI thought of something similar, though less surreptitious. I run a website that relies on a fairly enormous quantity of scraped data. To keep its dataset current I need a huge, continuous scraping cluster. I thought about letting people run a little thing in their system tray so that when the network interface on their computer is idle, it becomes part of my scraping cluster. I would be completely upfront with them about what it is doing - it would serve as a non-monetary way for them to contribute meaningfully to the operation of my site.