5 ms·
Foursquare dataset free to download and analyze
- nicholassmith 13y agoThat doesn't look like Foursquare has handed that over. What's the legality of scraping a service for their data in this way?
- onedev 13y agoThat doesn't look completely legal to me either.
- joshu 13y agothen re-releasing it with distribution terms is probably not legal at all.
- ozh 13y agoI don't get why scraping publicly and freely accessible data would be illegal. The redistribution under their own terms is another matter, though.
- chimeracoder 13y ago> publicly and freely accessible data would be illegal. Data is not "publicly and freely accessible" if accessing it requires you to agree to separate terms of service for it that restrict your ability to access and redistribute it. (Whether or not one believes the data should be freely and public accessible is a separate matter, but given the above, it's hard to make the case that it is). Amusingly, this data still isn't "freely accessible", because these people have attached their own, separate terms to reusing and redistributing the data.
- mvanvoorden 13y agoIf you have to apply to specific terms, but are able to access everything without complying to these terms, this effectively means that the data is publicly and freely accessible. Rules on itself don't restrict anything, enforcement does.
- jorgeortiz85 13y agoI'm a Foursquare engineer. We have explanations of our API policies here: https://developer.foursquare.com/overview/community https://developer.foursquare.com/overview/community We'll be contacting this researcher to ask where they got this data and whether it conforms to our policies.
- nicholassmith 13y agoThanks for responding, I didn't realise Foursquare actually gave so much of their data away freely. Which makes it slightly more seedy someone has scraped the rest and dumped it online.
- boothead 13y agoNo mention of the data format. Is it json, csv what? I know you can always head -n the file but a little hint would be helpful!
- nachi 13y agoIt looks like an ASCII-formatted table. Pretty disappointing that it isn't machine readable out of the box.
- sneak 13y agoObligatory: https://en.wikipedia.org/wiki/AOL_search_data_leak https://en.wikipedia.org/wiki/AOL_search_data_leak http://techcrunch.com/2006/08/06/aol-proudly-releases-massive-amounts-of-user-search-data/ http://techcrunch.com/2006/08/06/aol-proudly-releases-massiv... http://www.nytimes.com/2006/08/09/technology/09aol.html?pagewanted=all http://www.nytimes.com/2006/08/09/technology/09aol.html?page... TL;DR: It's fairly easy to deanonymize datasets like this, provided they are somewhat complete.
- route66 13y agoThe dataset comes with GPS locations of users and venues. With that data alone you can retrieve individual addresses in not too densely populated areas. Missing links will get caught in the social net. We already learned that Warhol's 15 minutes of fame should read 15 megabytes , but, to cut the "it's the users choice to post that data" apologists short: almost no-one I speak to understands the implications of all possible interpretations, classifications and groupings that their online traces allow.
- sneak 13y agoFurthermore, it's not the users' fault that 4sq isn't sufficiently rate limiting or otherwise protecting this data. Why should arbitrary users be able to see the social graph of others they're not friends with? Also, why should people outside of your immediate one-hop social graph be able to see your checkins? Giving it to 4sq for data mining is different than giving it to UMN and/or the whole internet for data mining and/or deanonymization.
- r721 13y agoAlso this: http://33bits.org/2009/05/13/your-morning-commute-is-unique-on-the-anonymity-of-homework-location-pairs/ http://33bits.org/2009/05/13/your-morning-commute-is-unique-...
- davidmat 13y agoCould anyone recommend some solid introductory material on data analysis/data visualisation? I'm thinking this data set seems like a fun way to fill a rainy weekend, going for a dive into these worlds :)
- benmanns 13y agoThe Social Network Analysis course on Coursera started yesterday. [0] 0: https://www.coursera.org/course/sna https://www.coursera.org/course/sna
- m4tthumphrey 13y agoLook's like it's been removed. Damn. Edit: Not removed, just unaccessible. 403.
- annnnd 13y agoI'm quite sure it will surface somewhere. Looking forward to it. :) EDIT: I would love to get my hands on this data... anyone? :)
- mvanvoorden 13y agoI have it, but "The user may not redistribute the data without separate permission." But if you have any questions about it, I can try to answer them, my username here also corresponds to a gmail account I use ;)
- danso 13y agoJesus Christ. The bulk scraping in violation of the TOS is egregious enough, but redistributing it with a mandate that the researchers get credit? For what, scraping a generous public API?
- Irishsteve 13y agoThis is usually common practice. It serves two purposes 1) Makes it easier to find the dataset that was used for experiments. And 2) Improves citation count for the author which is usually important in research.
- mzs 13y agoThat's true, but here is the paper from Microsoft Research and it seems to be lacking in how those data files were generated: http://research.microsoft.com/pubs/156453/icde12_lars.pdf http://research.microsoft.com/pubs/156453/icde12_lars.pdf
- rajbala 13y agoThe data set has been removed?
- lgas 13y agohttp://archive.org/details/201309_foursquare_dataset_umn http://archive.org/details/201309_foursquare_dataset_umn
- xntrk 13y agosounded too good to be true. I guess we'll have to find it on bittorrent.
- galapago 13y agohttp://webcache.googleusercontent.com/search?q=cache:hLI5FqDixY8J:www-users.cs.umn.edu/~sarwat/foursquaredata/+&cd=1&hl=en&ct=clnk http://webcache.googleusercontent.com/search?q=cache:hLI5FqD... (the direct link is not working, but this confirmed that was freely available)
- umnuser 13y agoWhat about this: http://www-users.cs.umn.edu/~baojie/dataset/ http://www-users.cs.umn.edu/~baojie/dataset/
- interskh 13y ago> This data set contains 2153471 users, 1143092 venues, 1021970 check-ins, 27098490 social connections, and 2809581 ratings that users assigned to venues The number of check-ins seems to be low compared to other numbers.
- renownedmedia 13y agoLooks like the data is only up-to-date as of July 2012 (judging from the zip compression times).
- waynesonfire 13y agowhy was this not posted as a torrent?