6 ms·

I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. Th

I did the analysis and worked with Stephen on the science side of it. If anyone would like to ask questions about what we did, I'd be happy to answer them. There's still lots more interesting stuff to do, but it was enough for a blog post. Suggest away if you think we missed something obvious!

programnature 14y ago
What took the most work? What was the most painful or repetitive bit? What new primitives in Mathematica would you like to see?
- taliesinb 14y ago
  Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
  - programnature 14y ago
    What can Mathematica bring to the DataFrame concept that hasn't been done before? Also: why Go instead of a JVM lang that can interop directly with Mathematica via JLink? Finally: Will Mathematica directly support doing the full stack of this kind of work, including the egress?
    - taliesinb 14y ago
      1. DataFrames themselves? Well, I think they'll get interesting when they can 'know' about high-level entities like cities, countries, zip codes, ip addresses, etc. Basically, everything that Alpha knows and can compute about, we want Mathematica to know and compute with. 2. I used Go because I am very productive in Go and like a lot of things about it. Goroutines are neat. Java is fine, it's just very boilerplatey, and I'm not practiced enough at it to get past that. And I don't see why we can't develop a GoLink as well. 3. Probably not the whole stack, at least in the beginning. But we'll get there. We want to make it really easy to spider websites and so on.
  - clebio 14y ago
    How long did the whole piece take to put together, and what's the rough break-down of time spent on each component (data wrangling, finding useful sorts, visualizations, write-up)? Thanks!
    - taliesinb 14y ago
      Fulltime, around 6 weeks. Breakdown is hard to say. I wasted a lot of time trying to do things the "traditional" way by loading into SQL, querying, etc, but it was actually much faster to process things in memory (I have a 16 gig machine). Intensive stuff was parallelized in Go and used ordinary filesystem with directory prefix tries for performance. Writeup was mostly SW. He's worked on it maybe an afternoon a week for the last month. I really enjoy visualizations and can iterate extremely fast (e.g. ChordPlot took half an hour). Don't know why M is not the defacto standard for dataviz people. Tweaking takes a long time, and design iterated with me on getting things looking really nice. All in all, most of my time was spent building tools to easily create multidimensional histograms. The nice thing is that those tools are clearly useful enough we'll integrate them into Mathematica, so the cost is somewhat amortized. NLP took a few weeks of Etienne's time... once again, amortized. Most of that is wrangling, really, and building tools to understand the deficiencies of your training set. Naive Bayes works surprisingly well, the magic is in the tooling and "human intelligence" you iterate with.
dekhn 14y ago
how would you effectively compete with distributed computing frameworks such as Pregel, MapReduce, and Dremel, when Mathematica is primarily used as a desktop application for in-RAM datasets? I know that Mathematica supports various parallelism options (such as multicore and grid), but frankly to gather real information requires much deeper probing, which far higher numbers of people, graph clustering/centrality on billions of nodes, edges etc. Mathematica's core routines seem to provide multicore implementations, but distributed algorithms require you to implement your own code on top of mathematica, meaning you'll never see the full performance/behavior of the finely tuned Mathematica im,plementation.
- taliesinb 14y ago
  There is already HadoopLink. LibraryLink allows you to write C or C++ that gets dynamically linked into the kernel at runtime (no restart required), which gives you freedom to create your own threads and do your own thing [and crash the kernel]. A lot of kernel development happens that way now. You can even synthesize C code from Mathematica (there is a symbolic subset of C in it already) and have Mathematica run the appropriate build process for you, so things can get pretty interesting with that alone. Out-of-core processing of large datasets is already on the roadmap for Mathematica 10. We plan to have a domain-specific language to describe and work with external [or in-memory] datasets in an efficient way, translating as appropriate to the native database query languages. Our 'native' format will be HDF5. Ultimately, though, I think we'll rely on code generation to compile Mathematica to LLVM or transpile it to Go, so that we can distribute chunks of computation out to a cluster using M as command-and-control. The idea would be that you can create and test large processing pipelines from inside Mathematica and then distribute them across a cluster in an ad-hoc way, then visualize the progress, track errors, and analyze the results. Notebooks are really good for that kind of lightweight UI. This isn't a new idea, but in a language as dynamic as Mathematica, I think it could be especially powerful. Of course, it is also tricky because type inference would be a big part of making this idea possible in a dynamically typed, symbolic language like Mathematica. But not impossible, I don't think. And functional languages already have demonstrated advantages in this type of situation -- take stream fusion in Haskell.
  - 14y ago
thesnark 14y ago
Is there a description available of the algorithm used to generate the 'cluster diagrams'? Really nice work.
- taliesinb 14y ago
  Thanks! Some of the plots use this: http://reference.wolfram.com/mathematica/ref/CommunityGraphPlot.html http://reference.wolfram.com/mathematica/ref/CommunityGraphP... The underlying community detection uses: http://reference.wolfram.com/mathematica/ref/FindGraphCommunities.html http://reference.wolfram.com/mathematica/ref/FindGraphCommun... If you look under "Method", there are a bunch of different methods to use that I'm told correspond to various landmark papers in the field. If you know about community detection, you'll recognize which methods correspond to which papers, but if you don't, why do you care? At least, that's our philosophy for documentation, but I'm not sure I entirely agree with that philosophy.
  - p3r1 14y ago
    I found the methods very interesting and thanks for the work. Have you thought to add the Surprise method? http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0024195 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna... It appears to solve the resolution limit of modularity.
    - taliesinb 14y ago
      Thanks, I've forwarded that to our graph algorithms group.
nswanberg 14y ago
Did your 2012 Strange Loop talk touch on the technology you used? Do you know if the video of the talk is still going to be released? https://thestrangeloop.com/news/strange-loop-2012-video-schedule https://thestrangeloop.com/news/strange-loop-2012-video-sche...
- taliesinb 14y ago
  No, I invented most of the technology over the last month. The naive Bayes and corpus wrangling was done by my colleague Etienne Bernard: http://www.wolframscience.com/summerschool/2012/alumni/bernard.html http://www.wolframscience.com/summerschool/2012/alumni/berna...
stared 14y ago
Are you planning to share some of that data in a reusable format (e.g. csv), not only as end-user plots (nice, BTW)? It would be great to play with it a bit.
- taliesinb 14y ago
  I'll have to check. Certainly there can be no harm in releasing some of the more aggregated data (i.e. that was behind the plots). Perhaps tweet at me so I don't forget -- @taliesinb
  - stared 14y ago
    Great! Tweeted.
stared 14y ago
When it comes to "questions I would ask [the data]": which interests are likely shared by friends? (Can you make a correlation table, or a graph out of it?) And - is correlations of one's interest vs friends interests the same as correlation of one's interests with itself.
- taliesinb 14y ago
  That's a good idea! Though if I remember correctly, interests aren't canonicalized, so it might be pretty messy. And I'm not sure if people fill them in non-ironically any more.
  - stared 14y ago
    Well, it is always "people how clicked on FB that they like X", regardless whether it is shallow or deep interest, genuine or ironic, or random, or "because my friends like it and I want to be as cool as them" etc. If more fine grained, I wouldn't be surprised to see ties between seemingly exclusive things... e.g. in this http://meta.stackoverflow.com/questions/157976/map-of-all-se-sites-except-the-3-biggest http://meta.stackoverflow.com/questions/157976/map-of-all-se... (not interesting, but participation in particular Q&A sites) "christianity", "judaism" and "islam" are in the same category (as opposed to people apathetic to that topic).
    - reeses 14y ago
      That's not precisely correct. In the old days, we typed in our favorite bands/books/music. The attempted canonicalization created some amusing interpretations of song as band, book as author, band as book, etc. It has also made it exceptionally annoying to get feed updates from the 900 bands, actors, movies, songs, etc. that I said I liked and now have to "unlike". I think you could correlate people within a certain confidence, but because of the nature of the data, you would have to expect a surprisingly large dissimilarity of interests within a clique over a certain account age based on this noise. Not 90%, but higher than the real value.
      - 14y ago
ececconi 14y ago
This is some really awesome work. How much time do you spend analyzing social network information? Do you think analyzing information about social networks will bring out real world insights about people and relationships?
- taliesinb 14y ago
  Quite a bit -- I've been analyzing my own data for years now. I did some network analysis stuff like this for Twitter long before it was built into Mathematica (rant: someone at Twitter needs to make a 'graph query' API call so that it doesn't take 3 hours to get a single graph of your own network). I would link to the relevant posts at taliesinb.net, but Posterous is down 7 days ahead of schedule. I think it can bring out real world insights. You just have to be very cautious and not leap to conclusions because they seem to tell an interesting story. Although it is somewhat disturbing how "gendered" the wall post topic distributions are.
susi22 14y ago
The only visualization I didn't like was the chord diagram: http://blog.stephenwolfram.com/data/uploads/2013/04/chordplot-movers-by-state3.png http://blog.stephenwolfram.com/data/uploads/2013/04/chordplo... Did you try this visualization?: http://bl.ocks.org/mbostock/4062006 http://bl.ocks.org/mbostock/4062006
- taliesinb 14y ago
  Yes, I did. We'll have to agree to disagree. I think the visualization you linked to is much harder to read, because the visual weight accorded to each edge is a non-linear (and somewhat arbitrary) function of the 'true' weight. It also doesn't scale well with number of vertices. Whereas with the chord diagram, your eye is naturally drawn to the big arrows, and you can easily follow them. It's also bidirectional in a more straightforward way.
szhorvat 14y ago
Several questions :-) 1. I noticed the interactive slider, embedded into the webpage. That's not what vanilla Mma 9 can do. Is there a simple way we can do the same without the CDF plugin (e.g. a package I'm not aware of) or is this future functionality? 2. The graph with the migration data looks nice. Mma 9 can't do an edge layout like this (curved edges) by default. Is this again custom code (custom Graphics or custom EdgeRenderingFunction) or is it future functionality? 3. There's the part with the frequency of various graph motives (the number of edges, triangles, and other weird shapes in graphs). How did you count these? Was it done using Mma? Some of these motifs are easy to count (there are simple expressions in therms of the adjacency matrix), but some others like the (1-2, 2-3, 1-3, 3-4) subgraph are not so easy. 4. How did you make the word clouds? Is the algorithm written in Mma? Here's a nice but slow one (the worse answer by myself): http://mathematica.stackexchange.com/questions/2334/how-to-create-word-clouds http://mathematica.stackexchange.com/questions/2334/how-to-c...
- taliesinb 14y ago
  1. https://gist.github.com/taliesinb/5464092 https://gist.github.com/taliesinb/5464092 . Hopefully we can build this in. 2. Custom Graphics. We don't (yet) support doing anything interesting with graph weights, which this relies heavily on. I think this is a good candidate for M10 -- name would probably be ChordPlot. I came up with a "GraphForm" construct that allows you to patch various graph properties into various visual parameters (size, color, edge weight, etc). That turns out to be quite useful. 3. Tally[list, IsomorphicGraphQ] . Isn't that cool? 4. Awesome! Nice code. We plan to create ImageCloud and WordCloud functions for M10. WordCloud will be specialized for representing word frequencies and so on. ImageCloud will be the general case: accept a list of images [potentially with transparency], and then find a nice layout given desired sizes. So much cool dataviz will be possible with this! Like country flags...

programnature 14y ago

What took the most work? What was the most painful or repetitive bit? What new primitives in Mathematica would you like to see?

taliesinb 14y ago
Data wrangling. So I wrote my own "DataFrame" -- we have an official one coming to Mathematica 10, too. Also, binning. There is a nice theory for multidimensional binning and aggregation [that I haven't seen anyone describe explicitly so far]. So I wrote primitives. They play nicely with plotting, statistics, etc. That'll also be in Mathematica 10. Lots of Go for data egress. It's perfect for it.
- programnature 14y ago
  What can Mathematica bring to the DataFrame concept that hasn't been done before? Also: why Go instead of a JVM lang that can interop directly with Mathematica via JLink? Finally: Will Mathematica directly support doing the full stack of this kind of work, including the egress?
  - taliesinb 14y ago
    1. DataFrames themselves? Well, I think they'll get interesting when they can 'know' about high-level entities like cities, countries, zip codes, ip addresses, etc. Basically, everything that Alpha knows and can compute about, we want Mathematica to know and compute with. 2. I used Go because I am very productive in Go and like a lot of things about it. Goroutines are neat. Java is fine, it's just very boilerplatey, and I'm not practiced enough at it to get past that. And I don't see why we can't develop a GoLink as well. 3. Probably not the whole stack, at least in the beginning. But we'll get there. We want to make it really easy to spider websites and so on.
- clebio 14y ago
  How long did the whole piece take to put together, and what's the rough break-down of time spent on each component (data wrangling, finding useful sorts, visualizations, write-up)? Thanks!
  - taliesinb 14y ago
    Fulltime, around 6 weeks. Breakdown is hard to say. I wasted a lot of time trying to do things the "traditional" way by loading into SQL, querying, etc, but it was actually much faster to process things in memory (I have a 16 gig machine). Intensive stuff was parallelized in Go and used ordinary filesystem with directory prefix tries for performance. Writeup was mostly SW. He's worked on it maybe an afternoon a week for the last month. I really enjoy visualizations and can iterate extremely fast (e.g. ChordPlot took half an hour). Don't know why M is not the defacto standard for dataviz people. Tweaking takes a long time, and design iterated with me on getting things looking really nice. All in all, most of my time was spent building tools to easily create multidimensional histograms. The nice thing is that those tools are clearly useful enough we'll integrate them into Mathematica, so the cost is somewhat amortized. NLP took a few weeks of Etienne's time... once again, amortized. Most of that is wrangling, really, and building tools to understand the deficiencies of your training set. Naive Bayes works surprisingly well, the magic is in the tooling and "human intelligence" you iterate with.

dekhn 14y ago

how would you effectively compete with distributed computing frameworks such as Pregel, MapReduce, and Dremel, when Mathematica is primarily used as a desktop application for in-RAM datasets? I know that Mathematica supports various parallelism options (such as multicore and grid), but frankly to gather real information requires much deeper probing, which far higher numbers of people, graph clustering/centrality on billions of nodes, edges etc. Mathematica's core routines seem to provide multicore implementations, but distributed algorithms require you to implement your own code on top of mathematica, meaning you'll never see the full performance/behavior of the finely tuned Mathematica im,plementation.

taliesinb 14y ago
There is already HadoopLink. LibraryLink allows you to write C or C++ that gets dynamically linked into the kernel at runtime (no restart required), which gives you freedom to create your own threads and do your own thing [and crash the kernel]. A lot of kernel development happens that way now. You can even synthesize C code from Mathematica (there is a symbolic subset of C in it already) and have Mathematica run the appropriate build process for you, so things can get pretty interesting with that alone. Out-of-core processing of large datasets is already on the roadmap for Mathematica 10. We plan to have a domain-specific language to describe and work with external [or in-memory] datasets in an efficient way, translating as appropriate to the native database query languages. Our 'native' format will be HDF5. Ultimately, though, I think we'll rely on code generation to compile Mathematica to LLVM or transpile it to Go, so that we can distribute chunks of computation out to a cluster using M as command-and-control. The idea would be that you can create and test large processing pipelines from inside Mathematica and then distribute them across a cluster in an ad-hoc way, then visualize the progress, track errors, and analyze the results. Notebooks are really good for that kind of lightweight UI. This isn't a new idea, but in a language as dynamic as Mathematica, I think it could be especially powerful. Of course, it is also tricky because type inference would be a big part of making this idea possible in a dynamically typed, symbolic language like Mathematica. But not impossible, I don't think. And functional languages already have demonstrated advantages in this type of situation -- take stream fusion in Haskell.
- 14y ago

thesnark 14y ago

Is there a description available of the algorithm used to generate the 'cluster diagrams'? Really nice work.

taliesinb 14y ago
Thanks! Some of the plots use this: http://reference.wolfram.com/mathematica/ref/CommunityGraphPlot.html http://reference.wolfram.com/mathematica/ref/CommunityGraphP... The underlying community detection uses: http://reference.wolfram.com/mathematica/ref/FindGraphCommunities.html http://reference.wolfram.com/mathematica/ref/FindGraphCommun... If you look under "Method", there are a bunch of different methods to use that I'm told correspond to various landmark papers in the field. If you know about community detection, you'll recognize which methods correspond to which papers, but if you don't, why do you care? At least, that's our philosophy for documentation, but I'm not sure I entirely agree with that philosophy.
- p3r1 14y ago
  I found the methods very interesting and thanks for the work. Have you thought to add the Surprise method? http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0024195 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjourna... It appears to solve the resolution limit of modularity.
  - taliesinb 14y ago
    Thanks, I've forwarded that to our graph algorithms group.

nswanberg 14y ago

Did your 2012 Strange Loop talk touch on the technology you used? Do you know if the video of the talk is still going to be released? https://thestrangeloop.com/news/strange-loop-2012-video-schedule https://thestrangeloop.com/news/strange-loop-2012-video-sche...

taliesinb 14y ago
No, I invented most of the technology over the last month. The naive Bayes and corpus wrangling was done by my colleague Etienne Bernard: http://www.wolframscience.com/summerschool/2012/alumni/bernard.html http://www.wolframscience.com/summerschool/2012/alumni/berna...

stared 14y ago

Are you planning to share some of that data in a reusable format (e.g. csv), not only as end-user plots (nice, BTW)? It would be great to play with it a bit.

taliesinb 14y ago
I'll have to check. Certainly there can be no harm in releasing some of the more aggregated data (i.e. that was behind the plots). Perhaps tweet at me so I don't forget -- @taliesinb
- stared 14y ago
  Great! Tweeted.

stared 14y ago

When it comes to "questions I would ask [the data]": which interests are likely shared by friends? (Can you make a correlation table, or a graph out of it?) And - is correlations of one's interest vs friends interests the same as correlation of one's interests with itself.

taliesinb 14y ago
That's a good idea! Though if I remember correctly, interests aren't canonicalized, so it might be pretty messy. And I'm not sure if people fill them in non-ironically any more.
- stared 14y ago
  Well, it is always "people how clicked on FB that they like X", regardless whether it is shallow or deep interest, genuine or ironic, or random, or "because my friends like it and I want to be as cool as them" etc. If more fine grained, I wouldn't be surprised to see ties between seemingly exclusive things... e.g. in this http://meta.stackoverflow.com/questions/157976/map-of-all-se-sites-except-the-3-biggest http://meta.stackoverflow.com/questions/157976/map-of-all-se... (not interesting, but participation in particular Q&A sites) "christianity", "judaism" and "islam" are in the same category (as opposed to people apathetic to that topic).
  - reeses 14y ago
    That's not precisely correct. In the old days, we typed in our favorite bands/books/music. The attempted canonicalization created some amusing interpretations of song as band, book as author, band as book, etc. It has also made it exceptionally annoying to get feed updates from the 900 bands, actors, movies, songs, etc. that I said I liked and now have to "unlike". I think you could correlate people within a certain confidence, but because of the nature of the data, you would have to expect a surprisingly large dissimilarity of interests within a clique over a certain account age based on this noise. Not 90%, but higher than the real value.
    - 14y ago

ececconi 14y ago

This is some really awesome work. How much time do you spend analyzing social network information? Do you think analyzing information about social networks will bring out real world insights about people and relationships?

taliesinb 14y ago
Quite a bit -- I've been analyzing my own data for years now. I did some network analysis stuff like this for Twitter long before it was built into Mathematica (rant: someone at Twitter needs to make a 'graph query' API call so that it doesn't take 3 hours to get a single graph of your own network). I would link to the relevant posts at taliesinb.net, but Posterous is down 7 days ahead of schedule. I think it can bring out real world insights. You just have to be very cautious and not leap to conclusions because they seem to tell an interesting story. Although it is somewhat disturbing how "gendered" the wall post topic distributions are.

susi22 14y ago

The only visualization I didn't like was the chord diagram: http://blog.stephenwolfram.com/data/uploads/2013/04/chordplot-movers-by-state3.png http://blog.stephenwolfram.com/data/uploads/2013/04/chordplo... Did you try this visualization?: http://bl.ocks.org/mbostock/4062006 http://bl.ocks.org/mbostock/4062006

taliesinb 14y ago
Yes, I did. We'll have to agree to disagree. I think the visualization you linked to is much harder to read, because the visual weight accorded to each edge is a non-linear (and somewhat arbitrary) function of the 'true' weight. It also doesn't scale well with number of vertices. Whereas with the chord diagram, your eye is naturally drawn to the big arrows, and you can easily follow them. It's also bidirectional in a more straightforward way.

szhorvat 14y ago

Several questions :-) 1. I noticed the interactive slider, embedded into the webpage. That's not what vanilla Mma 9 can do. Is there a simple way we can do the same without the CDF plugin (e.g. a package I'm not aware of) or is this future functionality? 2. The graph with the migration data looks nice. Mma 9 can't do an edge layout like this (curved edges) by default. Is this again custom code (custom Graphics or custom EdgeRenderingFunction) or is it future functionality? 3. There's the part with the frequency of various graph motives (the number of edges, triangles, and other weird shapes in graphs). How did you count these? Was it done using Mma? Some of these motifs are easy to count (there are simple expressions in therms of the adjacency matrix), but some others like the (1-2, 2-3, 1-3, 3-4) subgraph are not so easy. 4. How did you make the word clouds? Is the algorithm written in Mma? Here's a nice but slow one (the worse answer by myself): http://mathematica.stackexchange.com/questions/2334/how-to-create-word-clouds http://mathematica.stackexchange.com/questions/2334/how-to-c...

taliesinb 14y ago
1. https://gist.github.com/taliesinb/5464092 https://gist.github.com/taliesinb/5464092 . Hopefully we can build this in. 2. Custom Graphics. We don't (yet) support doing anything interesting with graph weights, which this relies heavily on. I think this is a good candidate for M10 -- name would probably be ChordPlot. I came up with a "GraphForm" construct that allows you to patch various graph properties into various visual parameters (size, color, edge weight, etc). That turns out to be quite useful. 3. Tally[list, IsomorphicGraphQ] . Isn't that cool? 4. Awesome! Nice code. We plan to create ImageCloud and WordCloud functions for M10. WordCloud will be specialized for representing word frequencies and so on. ImageCloud will be the general case: accept a list of images [potentially with transparency], and then find a nice layout given desired sizes. So much cool dataviz will be possible with this! Like country flags...