8 ms·
Data Science of the Facebook World
- geekam 13y agoHow is it possible that Facebook, which owns the data, does not give tools like these but others tap this using their data?
- mtgx 13y agoImagine how much more of this could be done if they did open their data.
- programnature 13y agoThe attitude of companies like Facebook, Google, Twitter is: if the product isn't addictive or useful to Billions of people, its not worth doing. Hence there are vastly more resources dedicated to assimilating eg photos and games into their ecosystem, than into something computationally innovative. This is IMHO a huge mistake, since they could instead be introducing simple forms of programming that takes you on a continuous curve from using the product, to developing for it. There is a huge hunger in the masses for better forms of programming. This point will probably become obvious if the Wolfram Language is successful.
- taliesinb 13y agoI'm sure Facebook's Data Science team does a lot of interesting things internally. They do in fact have some interesting papers [0] and [1], though obviously with more of an 'academic' feel than the blog post. [0]: http://arxiv.org/abs/1111.4503 http://arxiv.org/abs/1111.4503 [1]: http://arxiv.org/pdf/1201.4145 http://arxiv.org/pdf/1201.4145 (edit) Edit: they also have this FB page which has a steady stream of interesting stuff: https://www.facebook.com/data https://www.facebook.com/data
- photorized 13y agoData is their biggest asset, they wouldn't want to just shar it freely.
- jonpeda 13y agoPeople donate their data to support Wolfram's closed-source, paid-license, for-profit program?
- misiti3780 13y agoi was thinking the same thing ..
- mcintyre1994 13y agoThey said they'll use it to support their 'personal analytics' programme [0], which is free via wolframalpha.com - I don't see how this data would help with Mathematica or anything else they charge for? [0] http://www.wolframalpha.com/facebook/ http://www.wolframalpha.com/facebook/
- maxerickson 13y agoThey are using it for marketing. (I mean the post is what it is, I don't mean to whine about it)
- programnature 13y agoWolfram must have have built something people want, are willing to pay for, and even like ?
- stared 13y agoIt is data already on Facebook, so why are you so picky with sharing it further? (Or rather - criticizing people who do so.) Especially, as one can get some nice plots back (also - personal).
- hammock 13y agoIt's taking advantage of non-public Facebook data, which is closed anyway. So, your data is now being used by two places instead of one. More open.
- susi22 13y agoEven though WA is closed source and for-profit, I few them (and the company) as a kind of fellow scientist and not the evil big corporation (like oracle, microsoft etc.). It all depends who ask for my data.
- jonpeda 13y agoThe Mathematica system makes some beautiful, informative graphs, and presumably users can make those graphs with a minimum of fuss and bother. It's technically very nice. Yet, in the entire blog post, is there one insight that wasn't a priori obvious? Maybe the bits about migration. I don't see the "art and science" in this analysis, I see "stamp collecting" (http://en.wikiquote.org/wiki/Ernest_Rutherford http://en.wikiquote.org/wiki/Ernest_Rutherford)
- programnature 13y agoThe progression of interests over time was non-obvious to me. For instance the explosion of interest in travel in the 20s, and the temporary dip in interests like philosophical quotes also in the 20s. Theres a lot of stuff in the post, I wouldn't dismiss it because its TL;DR
- Create 13y agoThe LHC was built as a higgs factory. Or a stamp printing press, if you prefer. btw Mathematica would be little more than a DOE-MACSYMA/Maxima ripoff, not to mention the delusional "NKS" including its author.
- carlob 13y agoSo Rutherford says: "All science is either physics or stamp collecting" and then wins the Nobel prize for… You got that right: chemistry!
- taliesinb 13y agoI did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!
- programnature 13y agoWhat took the most work? What was the most painful or repetitive bit? What new primitives in Mathematica would you like to see?
- taliesinb 13y agoData wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
- programnature 13y agoWhat can Mathematica bring to the DataFrame concept that hasn't been done before? Also: why Go instead of a JVM lang that can interop directly with Mathematica via JLink? Finally: Will Mathematica directly support doing the full stack of this kind of work, including the egress?
- taliesinb 13y ago1. DataFrames themselves? Well, I think they'll get interesting when they can 'know' about high-level entities like cities, countries, zip codes, ip addresses, etc. Basically, everything that Alpha knows and can compute about, we want Mathematica to know and compute with. 2. I used Go because I am very productive in Go and like a lot of things about it. Goroutines are neat. Java is fine, it's just very boilerplatey, and I'm not practiced enough at it to get past that. And I don't see why we can't develop a GoLink as well. 3. Probably not the whole stack, at least in the beginning. But we'll get there. We want to make it really easy to spider websites and so on.
- pbnjay 13y agoHow much of the friends with zero friends is simply because that information is blocked? If my friends "donated" their data, I would show as having 0 friends if I've blocked that information to apps.
- taliesinb 13y agoActually, NONE of the people in our dataset had zero friends. The x-axis starts at 1, not 0. The point is that resampling to remove the friendship paradox shows that there are many more people with single-digit friends than we expected.
- pbnjay 13y agoGiven mcintyre1994's comment, I think this still explains the same situation. People with single-digit friends are simply people who have friends blocked to apps but have multiple friends who've donated data.
- taliesinb 13y agoNo, we can tell when someone's privacy settings have given them a zero friend count, and they're not included in that aggregation.
- mcintyre1994 13y agoAll, since they're friends with the person who donated the data.
- pbnjay 13y agoD'oh. Of course you're right, but simply substitute 1 for 0 and my base questions still stands.
- brown9-2 13y agoVery nice looking graphs, but running "Wolfram Alpha Personal Analytics for Facebook" for my own profile comes with a rather nerve-wracking warning: Wolfram Connection would like to access your public profile, friend list, email address, custom friends lists, News Feed, relationships, birthday, status updates, checkins, education history, hometown, current city, photos, religious and political views, videos, likes and your friends' relationships, birthdays, education histories, hometowns, current cities, photos, religious and political views and videos.
- dusing 13y agoHow do you think they are going to get all the data to analyze your facebook account?
- taliesinb 13y agoYeah, that's data you see in your report. You have to opt in to being a data donor for us to store any of it. Otherwise we just record basic anonymized statistics -- like number of friends, sex, age, etc... and throw all the detailed stuff away. Our privacy policy has more: http://www.wolframalpha.com/fbfaqs.html http://www.wolframalpha.com/fbfaqs.html We also encrypt with public keys like there's no tomorrow.
- brown9-2 13y agoI didn't realize going into it that the report would be that detailed. However, having opted in, I'm definitely impressed by what you get out of it.
- kevinalexbrown 13y agoI found this interesting. What I would love to have seen, however, is a probe into the dynamics. You did a nice abstraction over time as you measured property X as age was varied. I would have loved to have seen the manner in which topics and ideas spread over your network. For instance: If an event occurred in New York, say, how long would it have taken to spread to San Francisco? If there were no progression, topic times would center around the same time. This would indicate that people were getting their information from national, not local sources (e.g. the evening news), then talking about it on facebook. On the other hand, if a local topic was spread on facebook alone, we should see some sort of progression. It's possible that this progression could take more interesting forms besides geolocation, but that might require a more extensive network. A simple experiment would work like this: A few thousand people who are not friends but have a similar interest (say an interest in Elizabeth Warren) post independently a video of her. This particular esoteric interest is unlikely to be valued a priori by their friends, but perhaps they are compelled to repost the information. What's the threshold of "esotericness" such that it won't "go viral?" Is there a way to predict virality as a function of how popular it is to begin with? Is there no actual progression across the network, but rather a small bump in topic expression, until it is picked up by larger media sources at which point the entire network is inundated with people reposting Elizabeth Warren recaps from HuffPo et al? The reason this is interesting is that it sheds insight into the role of social networks: are we fundamentally disposed toward central sources like the NYTimes, or is facebook a fundamental sharing mechanism? That is, do I post on facebook just to have my views expressed, validated, and challenged, so that they might change the world over a few years? Or do I post on facebook to have my views propagate across the world much more quickly? Finally, a question: How did you estimate the power law? I know how difficult it is to do this (e.g. not linear regression on a log-log scale). Did you compare the power law fit to other, similar distributions, like lognormal? Preferential attachment is indeed a beautiful theoretical result, because it implies the existence of power law degree distributions. Unfortunately, many networks are not as well represented by power laws as by alternative distributions, which casts doubt on the preferential attachment hypothesis as is. (Also, many sampling methods give rise to fictive power laws). That said, a fat tail can still be interesting. In any case, this is a beautiful piece of work.
- 13y ago
- austinl 13y agoI've been doing Facebook network visualization for a while now with Gephi. Here are some of the graphs I came up with: http://visualizingpolitics.wordpress.com/2012/05/02/facebook-network-visualizations/ http://visualizingpolitics.wordpress.com/2012/05/02/facebook...
- taliesinb 13y agoNice! Do you know if Gephi can do something similar to the summarization that we did using cluster diagrams? The whole "ball of hair" problem doesn't have any other real solution, I don't think (well, unless you use edge clustering, but that doesn't help in-group connections).
- austinl 13y agoI'm 90% certain that it can, but I haven't worked with it in a while. I remember a setting in the display options that simply combined all the dots of one color into one large group.
- taliesinb 13y agoCool! I wonder how one could combine the best of both worlds... what we're really talking about here is a hierarchy of graph plots in which you can drill down to each node = graph at a lower level.
- photorized 13y agoOne thing that bugs me is how comments are linked to "interest". There are many topics that interest people (passive consumption), that do not necessarily translate into engaging in a conversation with others publicly. As a marketing term - sure, that would be a good indicator of interest. Since this article is more scientific than marketing-oriented, I would clarify what some of the metrics mean (or don't mean). Excellent, fantastic visualizations though!
- taliesinb 13y agoYou're right. A more ambitious thing might get a bit closer to people's "real interests" would be to follow posted links and topic model the contents of those links.
- photorized 13y agoI am very interested in that angle - let's connect when you have time, twitter @iTrendTV
- CurtMonash 13y agoIntroducing "data science for Facebook" in 2013 is ... odd. All the more so because Jeff Hammerbacher is often credited with coining the term "data science", and he started doing it at -- that's right -- Facebook.
- taliesinb 13y agoWell, the title is "Data Science of the Facebook World", there is no "for". The data comes from our "Personal Analytics for Facebook" product. Didn't know Jeff Hammerbacher coined "data science" at Facebook -- that's interesting!
- greiskul 13y agoI wonder if the higher friends count of Brazilian users is caused by the previous use of Orkut, where it was popular to try to have as many friends as possible.
- taliesinb 13y agoHa! That is interesting! I didn't know about that... will tell Stephen.
- sskates 13y agoWow- this is awesome! It's really cool how people's friend distribution by age is a convolution of their age and the age of the general facebook population. It's also scary in a way to see a snapshot of how I'm likely to change in the future with regards to my clusters of friends, my relationship status, and what I'll talk about.
- xk_id 13y agoThe traditional way to plot the assortativity by age is using a scatter plot / heatmap. This is similar to what they did for country homophily on p12 of the Facebook anatomy paper. The result would be a plot with a prominent diagonal, illustrating that "same attracts same". That aside, imo, Facebook is an incredibly idiosyncratic "app", which makes almost no sense. And yet, it gave us so many opportunities for interesting discussions, like the insights in this blog post. Nice job.
- taliesinb 13y agoYeah, we tried a couple. Those heatmaps I think are quite hard to read.. because it is natural to want to take marginals, but you can't easily do that visually. I think this whole "octile plot" thing turned out quite nicely. It's in a sense a way of 'slicing' the CDF into 8 even strips and projecting them onto a single axis. It's quite intuitive to read, too. Facebook seems to use it too for some of their papers.
- deleted 13y ago[deleted]