5 ms·

Mining social networks

45 points by iamelgringo 16y ago

akshayubhat 16y ago
Interesting. Sadly it does not mentions any practical tools for doing social network analysis: Here are few that I know: For small networks (up to a million or two million nodes such as Wikipedia Link graph from 2009) Following libraries provide code to handle and manipulate Network datasets: 1: SNAP by Prof. Jure Leskovec [ http://snap.stanford.edu http://snap.stanford.edu ] written in C++ 2: Networkx by Lanl [ http://networkx.lanl.gov/ http://networkx.lanl.gov/ ] written in Python, esp. good for fast prototyping There are few Databases for storing networks, e.g. Neo4J http://neo4j.org/ http://neo4j.org/ . Additionally there is a Graph Processing Language called as Gremlin http://wiki.github.com/tinkerpop/gremlin/ http://wiki.github.com/tinkerpop/gremlin/ . For Large networks with millions and billions of nodes, one can use Hadoop / Map-Reduce or Apache Hama [still in nascent stage]. Google has a special system known as Pregel which it uses to perform scalable computations over large networks.
- Elite 16y ago
  I'm watching one of the 2 hours videos from the Stanford professor. He looks like a college freshman! No offense intended. Do you have any resources on how to use these tools (especially the python library), and examples of implementation? I'm interested in learning more and using these tools on real data, but don't necessarily want to spend the time learning all of the theory behind it.
  - akshayubhat 16y ago
    Yup he is young, he did a post doc for a year at Cornell [under Prof. Kleinberg] after his PhD, and directly became Asst. Prof at Stanford. Well you can start with reading this book http://www.cs.cornell.edu/home/kleinber/networks-book/ http://www.cs.cornell.edu/home/kleinber/networks-book/ for overview, regarding application of these techniques are considered you can look for papers at recent WWW, NIPS,ICML conferences. The most popular and well studied areas are Link Prediction and Community Detection. The SNAP library comes with some good example code. You can also have a look at Divisi project at MIT Media Lab if you are interested in reasoning/ analogy over the networks. For real datasets, there are a lot of encyclopedic datasets such as Wikipedia / DBPedia /Semantic Web/ Music Brainz, as well as social ones such as Twitter follower network dataset. If you are in a university, you can even get full Web Graph from Yahoo [for research use alone].
loup-vaillant 16y ago
So, phone companies spy on us to make money, help the police (probably without a warrant), and spread propaganda. <sarcasm> How surprising. </sarcasm> Seriously, it's not like we couldn't have foreseen: http://www.softwarefreedom.org/news/2010/feb/01/freedom-cloud-software-freedom-privacy-and-securit/ http://www.softwarefreedom.org/news/2010/feb/01/freedom-clou...
dsplittgerber 16y ago
does anyone know of a secure, non-free alternative to gmail , where one wouldn't have (as much) to worry about having your privacy violated?
- dododo 16y ago
  http://www.hushmail.com/ http://www.hushmail.com/ perhaps? although sadly: http://en.wikipedia.org/wiki/Hushmail#Controversy http://en.wikipedia.org/wiki/Hushmail#Controversy
  - dsplittgerber 16y ago
    thanks, I knew about hushmail, but their usability and search function are not up to par. It's funny - I'd gladly pay for a technogically sophisticated alternative to gmail, just not to have all email data with google.
    - hushmail 16y ago
      If you'll give us another chance re usability, make an account & fill out this beta invite form: https://www.hushmail.com/contact/preview/ https://www.hushmail.com/contact/preview/ We have a new + faster + easier-to-use webmail coming in the next few months. It's a huge improvement over what we have now. Search is still in the current folder only though, so that might not be quite what you're looking for. (btw if anyone wants to help us build a better webmail, we're hiring PHP devs in Vancouver)
- bobfunk 16y ago
  Rackspace has a solid email hosting for businesses. Part of their sales-pitch is "Ad-free and Privacy Guaranteed," but eventually it always comes down to a question of trust.
- cageface 16y ago
  Keep in mind that your privacy is only as secure as that of your correspondents. I kind of gave up on the idea of keeping email private when I realized that most of the people I write are non-techy and use GMail, Facebook etc. naively.
rjurney 16y ago
Reading a social network analysis textbook, one is struck that all the examples involve rigorous data collection by social scientists with clipboards. Hundreds and thousands of hours of work for a small graph. Modern social networks have given us an abundance of data against which to leverage a backlog of techniques that were previously constrained by manual data collection. Phone companies use this stuff in ways that may be unethical, but it can just as easily be used in ways to empower you to leverage your own network. There are exciting potentials we are just beginning to see.
iamwil 16y ago
So this was back in 2008, but I read this from Bruce Schneider: http://www.schneier.com/blog/archives/2008/10/data_mining_for_1.html http://www.schneier.com/blog/archives/2008/10/data_mining_fo... <quote>But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."</quote> So did something change in the last two years? I mean Palantir is making a crap ton for presumably something similar. Was this ever a problem, or was it overcome?
- Alex3917 16y ago
  "Was this ever a problem, or was it overcome?" The problems Schneier is talking about have very different characteristics than the ones in this article. So the problems Schneier is talking about are still very real. But solving little pieces of the problem is definitely possible, especially with a good amount of human mediation.
  - doron 16y ago
    Human mediatiation is key. Skilled intelligence analysis made by well versed operators. even then it is very hard. As one intelligence analyst once told me " it is looking for a needle in stack of needles"
  - iamwil 16y ago
    Ahh, so difference in data mining is that it implies there's no human in the loop. Whereas palantir situates itself as a tool for intelligence, rather than as an oracle.
- iamelgringo 16y ago
  Finding out "Who is a terrorist, and what are they going to bomb?" is a much tougher problem than, "What information can I glean from Facebook, MySpace and Twitter to indicate this person might be a higher credit risk?" For instance, I imagine a high number of updates on a Facebook profile containing the word "drunk" might lead to a higher risk profile for auto insurance.
theprodigy 16y ago
This is why facebook is worth billions. They can mine so much social data and deliver targeted ads to the proper influencers.

45 points by iamelgringo 16y ago

Interesting. Sadly it does not mentions any practical tools for doing social network analysis: Here are few that I know: For small networks (up to a million or two million nodes such as Wikipedia Link graph from 2009) Following libraries provide code to handle and manipulate Network datasets: 1: SNAP by Prof. Jure Leskovec [ http://snap.stanford.edu http://snap.stanford.edu ] written in C++ 2: Networkx by Lanl [ http://networkx.lanl.gov/ http://networkx.lanl.gov/ ] written in Python, esp. good for fast prototyping There are few Databases for storing networks, e.g. Neo4J http://neo4j.org/ http://neo4j.org/ . Additionally there is a Graph Processing Language called as Gremlin http://wiki.github.com/tinkerpop/gremlin/ http://wiki.github.com/tinkerpop/gremlin/ . For Large networks with millions and billions of nodes, one can use Hadoop / Map-Reduce or Apache Hama [still in nascent stage]. Google has a special system known as Pregel which it uses to perform scalable computations over large networks.

Elite 16y ago
I'm watching one of the 2 hours videos from the Stanford professor. He looks like a college freshman! No offense intended. Do you have any resources on how to use these tools (especially the python library), and examples of implementation? I'm interested in learning more and using these tools on real data, but don't necessarily want to spend the time learning all of the theory behind it.
- akshayubhat 16y ago
  Yup he is young, he did a post doc for a year at Cornell [under Prof. Kleinberg] after his PhD, and directly became Asst. Prof at Stanford. Well you can start with reading this book http://www.cs.cornell.edu/home/kleinber/networks-book/ http://www.cs.cornell.edu/home/kleinber/networks-book/ for overview, regarding application of these techniques are considered you can look for papers at recent WWW, NIPS,ICML conferences. The most popular and well studied areas are Link Prediction and Community Detection. The SNAP library comes with some good example code. You can also have a look at Divisi project at MIT Media Lab if you are interested in reasoning/ analogy over the networks. For real datasets, there are a lot of encyclopedic datasets such as Wikipedia / DBPedia /Semantic Web/ Music Brainz, as well as social ones such as Twitter follower network dataset. If you are in a university, you can even get full Web Graph from Yahoo [for research use alone].

loup-vaillant 16y ago

So, phone companies spy on us to make money, help the police (probably without a warrant), and spread propaganda. <sarcasm> How surprising. </sarcasm> Seriously, it's not like we couldn't have foreseen: http://www.softwarefreedom.org/news/2010/feb/01/freedom-cloud-software-freedom-privacy-and-securit/ http://www.softwarefreedom.org/news/2010/feb/01/freedom-clou...

dsplittgerber 16y ago

does anyone know of a secure, non-free alternative to gmail , where one wouldn't have (as much) to worry about having your privacy violated?

dododo 16y ago
http://www.hushmail.com/ http://www.hushmail.com/ perhaps? although sadly: http://en.wikipedia.org/wiki/Hushmail#Controversy http://en.wikipedia.org/wiki/Hushmail#Controversy
- dsplittgerber 16y ago
  thanks, I knew about hushmail, but their usability and search function are not up to par. It's funny - I'd gladly pay for a technogically sophisticated alternative to gmail, just not to have all email data with google.
  - hushmail 16y ago
    If you'll give us another chance re usability, make an account & fill out this beta invite form: https://www.hushmail.com/contact/preview/ https://www.hushmail.com/contact/preview/ We have a new + faster + easier-to-use webmail coming in the next few months. It's a huge improvement over what we have now. Search is still in the current folder only though, so that might not be quite what you're looking for. (btw if anyone wants to help us build a better webmail, we're hiring PHP devs in Vancouver)
bobfunk 16y ago
Rackspace has a solid email hosting for businesses. Part of their sales-pitch is "Ad-free and Privacy Guaranteed," but eventually it always comes down to a question of trust.
cageface 16y ago
Keep in mind that your privacy is only as secure as that of your correspondents. I kind of gave up on the idea of keeping email private when I realized that most of the people I write are non-techy and use GMail, Facebook etc. naively.

rjurney 16y ago

Reading a social network analysis textbook, one is struck that all the examples involve rigorous data collection by social scientists with clipboards. Hundreds and thousands of hours of work for a small graph. Modern social networks have given us an abundance of data against which to leverage a backlog of techniques that were previously constrained by manual data collection. Phone companies use this stuff in ways that may be unethical, but it can just as easily be used in ways to empower you to leverage your own network. There are exciting potentials we are just beginning to see.

iamwil 16y ago

So this was back in 2008, but I read this from Bruce Schneider: http://www.schneier.com/blog/archives/2008/10/data_mining_for_1.html http://www.schneier.com/blog/archives/2008/10/data_mining_fo... <quote>But the authors conclude the type of data mining that government bureaucrats would like to do--perhaps inspired by watching too many episodes of the Fox series 24--can't work. "If it were possible to automatically find the digital tracks of terrorists and automatically monitor only the communications of terrorists, public policy choices in this domain would be much simpler. But it is not possible to do so."</quote> So did something change in the last two years? I mean Palantir is making a crap ton for presumably something similar. Was this ever a problem, or was it overcome?

Alex3917 16y ago
"Was this ever a problem, or was it overcome?" The problems Schneier is talking about have very different characteristics than the ones in this article. So the problems Schneier is talking about are still very real. But solving little pieces of the problem is definitely possible, especially with a good amount of human mediation.
- doron 16y ago
  Human mediatiation is key. Skilled intelligence analysis made by well versed operators. even then it is very hard. As one intelligence analyst once told me " it is looking for a needle in stack of needles"
- iamwil 16y ago
  Ahh, so difference in data mining is that it implies there's no human in the loop. Whereas palantir situates itself as a tool for intelligence, rather than as an oracle.
iamelgringo 16y ago
Finding out "Who is a terrorist, and what are they going to bomb?" is a much tougher problem than, "What information can I glean from Facebook, MySpace and Twitter to indicate this person might be a higher credit risk?" For instance, I imagine a high number of updates on a Facebook profile containing the word "drunk" might lead to a higher risk profile for auto insurance.

theprodigy 16y ago

This is why facebook is worth billions. They can mine so much social data and deliver targeted ads to the proper influencers.