4 ms·
The most well-known practitioners of this sort of 'topological' approach are Ayasdi, they have some slick demos [1]. The general name for this idea is topologic
by taliesinb 12y ago
The most well-known practitioners of this sort of 'topological' approach are Ayasdi, they have some slick demos [1]. The general name for this idea is topological data analysis [2].
I replicated this particular experiment in WL, of course, because it's a 5-minute thing to do [3], and I could actually do the community detection the author alluded to.
But I noticed that the correlation matrix itself is much more suggestive than the graph ends up being, with or without community detection. Take a look at the correlation matrix (note that MatrixPlot does some clever combination of rank and absolute value to get high dynamic range):
http://imgur.com/WKn029o http://imgur.com/WKn029o
The tri-diagonal structure is because the original dataset is derived from the pixel counts from successive 4x4 tiles on NIST written-digit images [4].
Those 8x8 matrix of tiles is flattened onto the 64 random variables, so the large correlation with tiles on the left and right explain the 1-off-diagonal orange lines, the other two diagonals are offset by 8 and correspond the high correlation with the tiles above and below. That's the 'connectivity kernel' of a 2D manifold, so to speak.
The curious squiggles in all the other blocks of this matrix are unusual. I don't know what's going on there. Maybe something interesting.
[1] http://www.ayasdi.com/ http://www.ayasdi.com/
[2] http://en.wikipedia.org/wiki/Topological_data_analysis http://en.wikipedia.org/wiki/Topological_data_analysis
[3] https://www.wolframcloud.com/objects/c7927909-448d-4502-9c1a-ff65243c1f5a https://www.wolframcloud.com/objects/c7927909-448d-4502-9c1a...
[4] https://archive.ics.uci.edu/ml/machine-learning-databases/optdigits/optdigits.names https://archive.ics.uci.edu/ml/machine-learning-databases/op...
- taliesinb 12y agoIt turns out that the graph is a bit more interesting when you can understand what part of the original image each node comes from. I've color coded each variable, along with a little legend image: http://imgur.com/TeOspAj http://imgur.com/TeOspAj
- jburroni 12y agoThis idea is actually very interesting!
- jburroni 12y agoThank you for your feedback. I din't know that topological data analysis even exist! I have some published and unpublished research on the topic. I had used this particular dataset by random. I wanted to use something with integers, large sample size and with high dimension. I'm glad you did the community detection stuff. I found it useful when the dimension is really large (in the order of thousand variables). For instance, when doing text mining. very nice comment!