5 ms·
Ask HN: How would you spatialize higher dimensional data?
I wrangle a lot of social media datasets, which could be anything from an email database to an old 4chan archive. I'm primarily interested in the metadata network dynamics rather than the content (although I might mine it for hashtags of length/complexity, but I am not very interested in NLP). Sometimes it's just a few thousand items, sometimes orders of magnitude larger.
Typically when I represent this data visually I use force-directed layouts or tree/radial ones if it's definitely hierarchical. And I'm pretty good at the visualization side - filtering, community detection, backbone extraction, layer decomposition, motif identification, time windowing and so on. I enjoy starting out with a giant hairball and translating it into something intuitively comprehensible and explicable. I also experiment a lot with other dataviz paradigms - treemaps, hyperbolic geometry and so on.
But one thing that bothers me about many layout algorithms is that they're so arbitrary; once you go past a certain level of complexity you can get quite different outcomes from the same dataset and starting procedures. And it's difficult to represent agents and their activity on the same graph, eg influencers and their tweets. I am constantly trying to think of ways to map the raw data to Cartesian space or on the interior of a sphere, not unlike a planetarium. But I have some kind of cognitive or knowledge-based block to understanding this, which I can't seem to get around.
How would you spatialize a complex network - eg suppose you had unlimited access to the HN API and plugged it into a particle system or an FPS game engine?
I'm more interested in creating a vivid, explorable impression than scientific modeling. I just struggle with the concept of locality because (in this example) any HN user could reply to or vote/flag any other as there's no 'travel time' between threads or users. And without a coherent spatial metaphor, I can't think of good mappings to 2 or 3 dimensions of space and another of time.
Thanks for your help!
- ByersReason 4y agohttps://arxiv.org/abs/1806.09823 https://arxiv.org/abs/1806.09823 - you can use this to find nearest neighbors to reduce the number of elements you are considering (or maybe not). Then you need: https://en.wikipedia.org/wiki/Dimensionality_reduction https://en.wikipedia.org/wiki/Dimensionality_reduction
- clusterhacks 4y agoThis is a super-interesting question. I don't think it is quite the answer to your question, but I have seen combined heatmaps/dendrograms that feel like a possibility for this? Maybe the heatmap deals with activity between nodes and the dendrogram handles the community cluster relationships? I see them often in bioinformatics Ex: https://en.wikipedia.org/wiki/Dendrogram#/media/File:Heatmap_RNAseqV2_1.png https://en.wikipedia.org/wiki/Dendrogram#/media/File:Heatmap...
- sargstuff 4y agoGephi [1] is a software suite to visualize social media datasets. Demo / feature pages [2][3] show different approaches to visualizing different social media datasets. Uncertain about current status of software support/development. ----------------------- [1] https://en.wikipedia.org/wiki/Gephi https://en.wikipedia.org/wiki/Gephi [2] http://www.martingrandjean.ch/gephi-introduction/ http://www.martingrandjean.ch/gephi-introduction/ [3] Gephi home page : https://github.com/gephi/gephi/releases https://github.com/gephi/gephi/releases
- sargstuff 4y agoSifting through Gephi references: https://en.wikipedia.org/wiki/Visual_analytics https://en.wikipedia.org/wiki/Visual_analytics Projection styles : https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduc... https://en.wikipedia.org/wiki/Principal_component_analysis https://en.wikipedia.org/wiki/Principal_component_analysis
- anigbrowl 4y agoI know, I've been using it for a decade. Also a fan of Cytoscape and Socviz, as well as a bunch of more obscure tools in this area. I should have mentioned my tools, thanks for reminding me.
- sargstuff 4y agoPerhaps view of data at different locality scales in different windows? figure 2 of subspace explorer in https://www.cs.uic.edu/~wilkinson/Publications/visualpatterndiscovery.pdf https://www.cs.uic.edu/~wilkinson/Publications/visualpattern...
- sargstuff 4y agoOn the 'more animated' side of things : https://hypertools.readthedocs.io/en/latest/ https://hypertools.readthedocs.io/en/latest/ ** different visual demonstrations : https://www.datavis.ca/gallery/delights.php https://www.datavis.ca/gallery/delights.php Edward Tufte has looked at various ways to present data : https://www.amazon.com/Edward-R-Tufte/e/B000APET3Y/ref=aufs_dp_fta_dsk https://www.amazon.com/Edward-R-Tufte/e/B000APET3Y/ref=aufs_... / https://www.edwardtufte.com/tufte/ https://www.edwardtufte.com/tufte/ ** Make the quantum leap and use fennman diagrams to construct the network ( https://www.ias.edu/ideas/2009/arkani-hamed-oconnell-feynman-diagrams https://www.ias.edu/ideas/2009/arkani-hamed-oconnell-feynman... ) ** So, could take the idea of scatter plot matrix in https://www.cs.uic.edu/~wilkinson/Publications/sorting.pdf https://www.cs.uic.edu/~wilkinson/Publications/sorting.pdf and use the various plots to literally build a picture as represented in Excel cells as "picture" : https://www.youtube.com/watch?v=UBX2QQHlQ_I https://www.youtube.com/watch?v=UBX2QQHlQ_I and https://jpg.space/mmmatto/exhibition/Mathcastles-%3E-Sandcastles https://jpg.space/mmmatto/exhibition/Mathcastles-%3E-Sandcas... Hypercastle explorer for terraforms : https://www.youtube.com/watch?v=1jD6F_6_Yak https://www.youtube.com/watch?v=1jD6F_6_Yak Although, that kind of starts to get into coding. ;) https://direct.mit.edu/leon/article-abstract/48/4/375/45993/Light-Pattern-Writing-Code-with-Photographs?redirectedFrom=fulltext#.VsXePvkrKUk https://direct.mit.edu/leon/article-abstract/48/4/375/45993/... http://lightpattern.info/ http://lightpattern.info/ https://esolangs.org/wiki/ObjectArt https://esolangs.org/wiki/ObjectArt https://www.dangermouse.net/esoteric/piet.html https://www.dangermouse.net/esoteric/piet.html ** Obligitory HN 'lisp' link : https://web.archive.org/web/20110504131632/http://sas.uwaterloo.ca/Quail/features.html https://web.archive.org/web/20110504131632/http://sas.uwater...
- anigbrowl 4y agoSome of these definitely look intriguing. There's a lot of good techniques that have fallen by the wayside for years because they required way more computing power than was practical at the time they were prototyped. https://treevis.net/ https://treevis.net/ has an especially comprehensive collection of network visualization tropes. I'm not averse to coding and do some in this area already, though in the Lego-assembly fashion of mixing and matching things in Jupyter notebooks with Python or Julia rather than product engineering.
- anigbrowl 4y agoMany great new avenues of inquiry here - I thought the thread had died for lack of interest. Thanks to everyone!
- gus_massa 4y agoLocality means make a lot of comments in the same threads. Note that it will make close contacts people that agree a lot and also people that disagree a lot. I'd try using the 2nd and 3rd eigenvector of the Laplacian of gtaph. I heard a lot of good properties of them, but I never used them so I'm not sure how tricky is to get a nice graph. Something like https://math.stackexchange.com/questions/3853424/what-does-the-value-of-eigenvectors-of-a-graph-laplacian-matrix-mean https://math.stackexchange.com/questions/3853424/what-does-t...
- sargstuff 4y agoPick your statistical software package & chart to generate code for! https://chartmaker.visualisingdata.com https://chartmaker.visualisingdata.com
- sargstuff 4y agoSome stories not safe for work viewing! Longform data journalism stories (aka visual statistics case studies): https://pudding.cool/ https://pudding.cool/ Small corrolation of book covers (~5,000) : https://pudding.cool/2019/07/book-covers/ https://pudding.cool/2019/07/book-covers/
- sargstuff 4y agoMeta-visualization / statistical use cases links site : https://www.tableau.com/learn/articles https://www.tableau.com/learn/articles sample links: https://www.tableau.com/learn/articles/data-science-blogs https://www.tableau.com/learn/articles/data-science-blogs https://www.tableau.com/learn/articles/best-data-visualization-blogs https://www.tableau.com/learn/articles/best-data-visualizati...
- sargstuff 4y agohttps://digitalarrtarchive.siggraph.org/exhibition/siggraph-2009-information-aesthetics-showcase/ https://digitalarrtarchive.siggraph.org/exhibition/siggraph-...