3 ms·
I'm hoarding profile data diffs from a well known social network. Been crawling every single user for the past 2 years and saving the changes. Had to stop doing
by kaivi 9y ago
I'm hoarding profile data diffs from a well known social network. Been crawling every single user for the past 2 years and saving the changes. Had to stop doing it last month, after storage costs became too much.
- SIRHAMY 9y agoWoah, do you have an end goal with this. Willing to let people get their hands on it?
- kaivi 9y agoI'd like to set it up as a subscription service like Domaintools. I'm also crawling and storing user's comments, friends, posts and links to media published in other groups. It will make a great tool for profiling a person, but I'm not sure on how (il)legal it will be.
- j_s 9y agoCan you share any of the techniques you use to avoid being blocked for scraping? Interesting to hear that storage was too expensive. I'm sure you could find personal sponsors here who would help with funding in exchange for access to the data - even with a 'no commercial use' type of restriction!