10 ms·

Datashader: turns even the largest data into images, accurately

237 points by luu 8y ago

BubRoss 8y ago
This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These too have been problems solved a long time ago with a lot less resources. The book advanced renderman are is a good place to start, and of course, opengl uses basic z-buffer rendering. Unless you need sorted transparency (and accumulating points for brightness definitely does not), then you can rasterize as much as you want in any order, without holding anything in memory except for the image buffer. It doesn't matter how many point/particles you have, you don't have to do anything to make it scale.
- Eli_P 8y ago
  What are those fractals on the 3rd picture? They remind me of Lissajous curves used in oscilloscopes.
  - gilleain 8y ago
    I think they are "symmetric icons" : http://www.alunw.freeuk.com/symiconroom.html http://www.alunw.freeuk.com/symiconroom.html Can't remember where I've seen them before - possibly in a book on Symmetry.
    - jbednar 8y ago
      Yes, those are symmetric icons, a type of attractor. See http://datashader.org/topics/strange_attractors.html http://datashader.org/topics/strange_attractors.html for all the details.
- jameskilton 8y ago
  And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly know everything. To the Datashader team: I apologize for the above comment. Good job in building and launching a tool for others to use, and great choices for examples!
  - RodericDay 8y ago
    As a newcomer to the field, parent post is far more welcoming than the bevy of trash Ninja flavor-of-the-week bullshit that makes it seem impossible to catch up. I only actually got my bearings and self-confidence as a programmer when I realized that most of the people pushing blogs with subscriptions about "cutting-edge" tech were literally snake-oil salesmen and shovel merchants. That coding wasn't actually different from anything else I had learned in my life, and that there were some fundamentals I could latch onto, and grow from there upwards. All this nonsense about the field experiencing a revolution that upends all existing knowledge year-after-year is far more mentally taxing.
  - corysama 8y ago
    “Dropbox is just rsync on a cron job” and “OP’s project is not completely novel” are unfortunately common tropes on HN... Like you said. Starting with “Great job launching. What are the benefits of using Dropbox over, say just rsync + cron?” would go a long way towards improving the environment around here.
  - BubRoss 8y ago
    My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets into a territory of reinventing the wheel but naming it 'the flattened infinite curvature hypersphere'. So, before you get too self righteous at least realize that it is the delivery and lack of context and precedent, not the actual work that is the problem.
- douglaswlance 8y ago
  It doesn't matter if something has been done before if the new way hooks into new systems. Tools do not exist in a vacuum. All tools are apart of a system of tools that should always be considered when evaluating any component piece.
- sevensor 8y ago
  I also was expecting something new, but in their defense they've made a very appealing version of something old. I'm sure there are a lot of people out there who haven't thought about saturation with "large" data sets before.
- IanCal 8y ago
  > thinking up fancy names for reinventing the wheel They're not claiming to have reinvented the wheel, they're just explaining what it is. > 'Turning data into images' isn't exactly a new concept. No, but doing so on large data accurately (the last word is important that you cut off) is not something I know I can easily achieve in a different python library faster. I'd like to know if I could.
  - CyberDildonics 8y ago
    > but doing so on large data accurately What do you mean by 'large' or 'accurate' ? Where would accuracy be lost in any approach?
    - IanCal 8y ago
      If you want to discuss this properly, can you please finish the sentence? What library should I use to do this faster in python without subsampling the data for a start?
    - pwang 8y ago
      This paper on our original ideas of "Abstract Rendering" talks about the kinds of accuracy problems that plague the visualization of large datasets: https://www.semanticscholar.org/paper/Abstract-rendering%3A-out-of-core-rendering-for-Cottam-Lumsdaine/41a157f01b27bcf20143fc578a0db49f4e7b90f3 https://www.semanticscholar.org/paper/Abstract-rendering%3A-... We renamed from Abstract Rendering to Datashader for affordances of human cognition. This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization, as part of an effort to come up with an algebraic process for visual design: https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20tvcg12-kindlmann-2346325.pdf https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20... Using their metrics around "confusers" and "hallucinators", Datashader came out as one of the few things that doesn't suffer from such intrinsic limitations.
      - CyberDildonics 8y ago
        There are a lot of red flags in the abstract of that paper alone. > Rendering techniques are currently a major limiter since they tend to be builtaround central processing with all of the geometric data present. This is completely untrue - OpenGL and virtually all real time rendering is done using z-buffer techniques that were originally used because they don't need all the geometry present. These techniques date back to the 70s and were some of the first hidden surface rendering algorithms. > This paper presents Abstract Rendering (AR), a technique for eliminating the cen-tralization requirement while preserving some forms of interactivity. Interactivity might be novel here so that is what should really be focused on, if anything. I don't think coining a new term and acronym that don't seem to relate to what is happening is a going to be a good choice to communicate the techniques. > AR is based on the observation that pixelsare fundamentally bins, and that rendering is essentially a binning process on a lattice of bins. This observation was made in the early 80s and has been the backbone of renderman renderers for almost 40 years. Renderman calls them 'buckets'. > This approach enables: (1) rendering onlarge datasets without requiring large amounts of working memory, Renderman originally rendered film resolution images with high resolution textures with only 10MB of memory. > (3) a direct means of distributing the rendering task across processes, Giving different threads their own buckets is standard for any non-toy renderer. Distributing buckets across multiple computers is part of many toolsets. > high-performanceinteraction techniques on large datasets This is the only part that has a chance of being novel, but paper only shows basic accumulation of density for adjacency matrices. The visualization are timed in the multiple seconds but look extremely simple, and for some reason are rendered 'out-of-core' on a computer with 144GB of memory even though it seems very unclear that these images couldn't be made with z-buffer rendering in opengl. > This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization It looks like that paper is about the transformations of visualizations for higher dimensional data, not rendering accuracy, so these two things are being conflated even though they are completely separate concepts.
- candu 8y ago
  Ah, the good old "MapReduce is basically functional programming 101" trope, usually resulting from a fundamental misunderstanding of the problem the framework / tool in question solves.
  - TeMPOraL 8y ago
    Well, in that particular case, Google has brought this on themselves by naming their sort-of-a-product after a common FP idiom, and then hyping the hell out of it.
- pwang 8y ago
  You're missing the point of this project. It's not about the feasibility of throwing a billion points at a pile of software, to get an image. I can do that with a simple Python script. It's about doing so to create a meaningful and accurate data visualization, and not just a picture of, say, shiny spheres or a scene from Avatar. I actually have a background in 3D computer graphics, and it's precisely because of my detailed knowledge of raytracing, rasterization, OpenGL, BMRT, photon maps, computational radiometry, BDRFs, computational geometry, and statistical sampling, etc... that when I came to the field of data science & specifically the problem of visualizing large datasets, I realized the total lack of tooling in this space. The field of information visualization lags behind general "computer-generated imagery" by decades. When I first presented my ideas around Abstract Rendering (which became Datashader) to my DARPA collaborators, even to famous visualization people like Bill Cleveland or Jeff Heer, it was clear that I was thinking about the problem in an entirely different way. I recall our DARPA PM asking Hanspeter Pfister how he would visualize a million points, and he said, "I wouldn't. I'd subsample, or aggregate the data." Datashader eats a million points for breakfast. Since you're clearly a computer graphics guy, the way to think about this problem is not one of naive rendering, but rather one of dynamically generating correct primitives & aesthetics at every image scale, so that the viewer has the most accurate understanding of what's actually in the dataset. So it's not just a particle cloud, nor is it nurbs with a normal & texture map; rather, it's a bunch of abstract values from which a data scientist may want to synthesize any combination of geometry and textures. I chose the name "datashader" for a very specific and intentional reason: we are dynamically invoking a shader - usually a bunch of Python code for mathematical transformation - at every point, within a sampling volume (typically a square, but it doesn't have to be). One can imagine drawing a map of the rivers of the US, with the shading based on some function of all industrial plants in its watershed. Both the domain of integration and the function to evaluate are dynamic for each point in the view frustum.
  - BubRoss 8y ago
    > Datashader eats a million points for breakfast. So does opengl on a decade old computer.
johnmarinelli 8y ago
Looks like a really cool project. One thing that I would be interested in see would be using Datashader as a dynamic visualisation library - for example, generative art projects. Probably not the main interest of data visualisation practitioners but hey, if you've got a sweet pipeline to render all those points, why not?
- jbednar 8y ago
  The attractors at http://datashader.org/topics/strange_attractors.html http://datashader.org/topics/strange_attractors.html are probably closer to art than science...
IanCal 8y ago
Datashader is a great project. Very fast, very easy to use. You can throw a lot of data at it in a notebook and get back a zoomable interactive pane. Here's a 2016 talk on it: https://www.youtube.com/watch?v=fB3cUrwxMVY https://www.youtube.com/watch?v=fB3cUrwxMVY There's likely a lot of improvements since then, but that should help show some of the core parts and explain why it's a useful tool.
simplyinfinity 8y ago
Is there anything similar for network graphs?
- jointpdf 8y ago
  Perhaps Gephi? It was used for those Graphcore neural network visualizations (millions of edges and nodes) and they look stunning. [1] https://gephi.org/features/ https://gephi.org/features/ [2] https://www.graphcore.ai/posts/what-does-machine-learning-look-like https://www.graphcore.ai/posts/what-does-machine-learning-lo...
  - simplyinfinity 8y ago
    I've used both gephi and cytoscape, i have about 2 million nodes with about 3 million edges, it's hard to visualise at once. however i've reduced the graphs to only sub networks with more than 5 connections, the remainder is about 600k, which is still painfully slow to visualise
- maliker 8y ago
  My team has had luck rendering an SVG from the graph and sending it to a browser. It works well for about 10k vertices and edges. Above that scale we use datashader, and we're investigating a potential to move to QGIS. We tried Gephi a few years ago and it had trouble at these scales.
- lmeyerov 8y ago
  Yep -- at https://github.com/graphistry/pygraphistry https://github.com/graphistry/pygraphistry, we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc. We've been curious about server-side static tile rendering for larger graphs, but has been on the back-burner. (We already connect to GPUs on the server, so not rocket science.) Currently, we're actively increasing how much can be ingested + computed on, such as for finding influencers, communities & rings, etc. However, visualizing that hasn't been an operational priority for our users. More useful to generate the communities, and then either inspect individual ones, or see how communities stitch together: quickly run out of pixels otherwise due to too many edges. Likewise, we're building connectors to gigascale-petascale graph DBs: titan, janus, aws neptune, tigergraph, spark graphx, etc. We still are interested, but more for when we start supporting geographic maps: you can see that is the primary use for datashader. Also, because data art is fun :)
mhalle 8y ago
Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or make the results look misleading or bad. Even better that they built a tool that works with existing libraries, rather than replacing them. Good work!
- BubRoss 8y ago
  > anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
  - detaro 8y ago
    Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.
    - BubRoss 8y ago
      Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
      - el_dev_hell 8y ago
        > Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. 1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.
        8y ago
itodd 8y ago
I've used datashader for plotting NGS (Next Generation Sequencing) enrichments. At the time I had to hack together the ability to use the polygon select tool on the data, but it worked and blew my mind. Very elegant solution to a difficult problem (overplotting).
- abcc8 8y ago
  Do you have any examples of this you could point to online? I am looking at different visualization tools for various NGS-based analyses currently.
  - itodd 8y ago
    I do. I remember posting this to the mailing list. I don't have an example calculating enrichments though. We simply group by read and divide using the frequencies and then plot one enrichment vs another. This way we can see how one sequence enriches between conditions. There's more to it than that but this will produce a plot similar to the one in the attachment in the thread linked below. https://groups.google.com/a/continuum.io/forum/m/#!msg/bokeh/BdQV54W0WnY/OHjlMAtqBAAJ https://groups.google.com/a/continuum.io/forum/m/#!msg/bokeh... Edit: here's a link to the plot I refer to: https://10826817673355204906.googlegroups.com/attach/e6e58ade6248/Screen%20Shot%202017-05-04%20at%202.47.07%20PM.png?part=0.1&view=1&vt=ANaJVrHL6xR_HvljfCodWVnBwALwADyL8wIpWga5D7Q5DzAHYPOsTTpqg193EuAC81xxXb-toP7Nv8y724yMogPSNlLXeU6lZXJ-Trm3r5-zpIVHEe2fpUI https://10826817673355204906.googlegroups.com/attach/e6e58ad...
- deleted 8y ago
  [deleted]
24gttghh 8y ago
https://anaconda.org/jbednar/gerrymandering/notebook https://anaconda.org/jbednar/gerrymandering/notebook I wish more people were outraged at this kind of election tampering. Great visualizations though! Zoom in on some of those tight masses of black outlines. The shapes are ridiculous. Maryland 3rd? Come on.
- deleted 8y ago
  [deleted]
burtonator 8y ago
This actually gave me an interesting idea regarding bitcoin passphrase mnemonics. Instead of text we could use the same algorithm to generate images. So you could have an index of images and generate them. I'm actually wondering if you could use nouns and verbs to maybe make stories if you could mutate the nouns reliably. Like 'bird flying' vs 'bird sleeping' ... This could help to remember long passphrases visually which people seem to be better at.
whoisjuan 8y ago
What license is this project using? The repo has a license but besides the provisions listed there I don't see any standard license.
- tnvaught 8y ago
  The repo has a standard 3-clause BSD license.
tokyodude 8y ago
> Turns even the largest data into images, accurately The first image, the image of the USA, seems really mis-representative to me. LA and NYC should be way way way more bright in relation to everything else than the entire area east of the Mississippi. At least to my eyes that map makes it look like parts of Denver, Kansas City, Salt Lake City, Atlanta, and the San Joaquin Valley are just as dense as Manhattan. Atlanta's population density 630 per square mile Manhattan's population density 70826 per square mile It seems like an accurate data image would have Atlanta's brightness 1/100th of Manhattan's. Basically it looks like they saturated out at around 250 people do anything over 250 people is the same brightness.
- jbednar 8y ago
  By default, Datashader accurately conveys the shape of the distribution in a way that the human visual system can process. If you want a linear representation, you can do that easily; see the first plot in http://datashader.org/topics/census.html http://datashader.org/topics/census.html , but you'll quickly see that the resulting plot completely fails to show that there are any patterns anywhere besides the top few population hotspots, which is highly unrepresentative of the actual patterns in this data. There is no saturation here; what it's doing in the homepage image is basically a rank-order encoding, where the top brightness value is indeed shared by several high-population pixes, the next brightness value is shared by the next batch of populations, etc. Given only 256 possible values, there has to be some grouping, but it's not saturating.
- pwang 8y ago
  Yes, datashader actually gives you the ability to dial-in as much gamma compensation as you want, to account for the human visual system's nonlinear response to luminance.

237 points by luu 8y ago

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These too have been problems solved a long time ago with a lot less resources. The book advanced renderman are is a good place to start, and of course, opengl uses basic z-buffer rendering. Unless you need sorted transparency (and accumulating points for brightness definitely does not), then you can rasterize as much as you want in any order, without holding anything in memory except for the image buffer. It doesn't matter how many point/particles you have, you don't have to do anything to make it scale.

Eli_P 8y ago
What are those fractals on the 3rd picture? They remind me of Lissajous curves used in oscilloscopes.
- gilleain 8y ago
  I think they are "symmetric icons" : http://www.alunw.freeuk.com/symiconroom.html http://www.alunw.freeuk.com/symiconroom.html Can't remember where I've seen them before - possibly in a book on Symmetry.
  - jbednar 8y ago
    Yes, those are symmetric icons, a type of attractor. See http://datashader.org/topics/strange_attractors.html http://datashader.org/topics/strange_attractors.html for all the details.
jameskilton 8y ago
And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly know everything. To the Datashader team: I apologize for the above comment. Good job in building and launching a tool for others to use, and great choices for examples!
- RodericDay 8y ago
  As a newcomer to the field, parent post is far more welcoming than the bevy of trash Ninja flavor-of-the-week bullshit that makes it seem impossible to catch up. I only actually got my bearings and self-confidence as a programmer when I realized that most of the people pushing blogs with subscriptions about "cutting-edge" tech were literally snake-oil salesmen and shovel merchants. That coding wasn't actually different from anything else I had learned in my life, and that there were some fundamentals I could latch onto, and grow from there upwards. All this nonsense about the field experiencing a revolution that upends all existing knowledge year-after-year is far more mentally taxing.
- corysama 8y ago
  “Dropbox is just rsync on a cron job” and “OP’s project is not completely novel” are unfortunately common tropes on HN... Like you said. Starting with “Great job launching. What are the benefits of using Dropbox over, say just rsync + cron?” would go a long way towards improving the environment around here.
- BubRoss 8y ago
  My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets into a territory of reinventing the wheel but naming it 'the flattened infinite curvature hypersphere'. So, before you get too self righteous at least realize that it is the delivery and lack of context and precedent, not the actual work that is the problem.
douglaswlance 8y ago
It doesn't matter if something has been done before if the new way hooks into new systems. Tools do not exist in a vacuum. All tools are apart of a system of tools that should always be considered when evaluating any component piece.
sevensor 8y ago
I also was expecting something new, but in their defense they've made a very appealing version of something old. I'm sure there are a lot of people out there who haven't thought about saturation with "large" data sets before.
IanCal 8y ago
> thinking up fancy names for reinventing the wheel They're not claiming to have reinvented the wheel, they're just explaining what it is. > 'Turning data into images' isn't exactly a new concept. No, but doing so on large data accurately (the last word is important that you cut off) is not something I know I can easily achieve in a different python library faster. I'd like to know if I could.
- CyberDildonics 8y ago
  > but doing so on large data accurately What do you mean by 'large' or 'accurate' ? Where would accuracy be lost in any approach?
  - IanCal 8y ago
    If you want to discuss this properly, can you please finish the sentence? What library should I use to do this faster in python without subsampling the data for a start?
  - pwang 8y ago
    This paper on our original ideas of "Abstract Rendering" talks about the kinds of accuracy problems that plague the visualization of large datasets: https://www.semanticscholar.org/paper/Abstract-rendering%3A-out-of-core-rendering-for-Cottam-Lumsdaine/41a157f01b27bcf20143fc578a0db49f4e7b90f3 https://www.semanticscholar.org/paper/Abstract-rendering%3A-... We renamed from Abstract Rendering to Datashader for affordances of human cognition. This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization, as part of an effort to come up with an algebraic process for visual design: https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20tvcg12-kindlmann-2346325.pdf https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20... Using their metrics around "confusers" and "hallucinators", Datashader came out as one of the few things that doesn't suffer from such intrinsic limitations.
    - CyberDildonics 8y ago
      There are a lot of red flags in the abstract of that paper alone. > Rendering techniques are currently a major limiter since they tend to be builtaround central processing with all of the geometric data present. This is completely untrue - OpenGL and virtually all real time rendering is done using z-buffer techniques that were originally used because they don't need all the geometry present. These techniques date back to the 70s and were some of the first hidden surface rendering algorithms. > This paper presents Abstract Rendering (AR), a technique for eliminating the cen-tralization requirement while preserving some forms of interactivity. Interactivity might be novel here so that is what should really be focused on, if anything. I don't think coining a new term and acronym that don't seem to relate to what is happening is a going to be a good choice to communicate the techniques. > AR is based on the observation that pixelsare fundamentally bins, and that rendering is essentially a binning process on a lattice of bins. This observation was made in the early 80s and has been the backbone of renderman renderers for almost 40 years. Renderman calls them 'buckets'. > This approach enables: (1) rendering onlarge datasets without requiring large amounts of working memory, Renderman originally rendered film resolution images with high resolution textures with only 10MB of memory. > (3) a direct means of distributing the rendering task across processes, Giving different threads their own buckets is standard for any non-toy renderer. Distributing buckets across multiple computers is part of many toolsets. > high-performanceinteraction techniques on large datasets This is the only part that has a chance of being novel, but paper only shows basic accumulation of density for adjacency matrices. The visualization are timed in the multiple seconds but look extremely simple, and for some reason are rendered 'out-of-core' on a computer with 144GB of memory even though it seems very unclear that these images couldn't be made with z-buffer rendering in opengl. > This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization It looks like that paper is about the transformations of visualizations for higher dimensional data, not rendering accuracy, so these two things are being conflated even though they are completely separate concepts.
candu 8y ago
Ah, the good old "MapReduce is basically functional programming 101" trope, usually resulting from a fundamental misunderstanding of the problem the framework / tool in question solves.
- TeMPOraL 8y ago
  Well, in that particular case, Google has brought this on themselves by naming their sort-of-a-product after a common FP idiom, and then hyping the hell out of it.
pwang 8y ago
You're missing the point of this project. It's not about the feasibility of throwing a billion points at a pile of software, to get an image. I can do that with a simple Python script. It's about doing so to create a meaningful and accurate data visualization, and not just a picture of, say, shiny spheres or a scene from Avatar. I actually have a background in 3D computer graphics, and it's precisely because of my detailed knowledge of raytracing, rasterization, OpenGL, BMRT, photon maps, computational radiometry, BDRFs, computational geometry, and statistical sampling, etc... that when I came to the field of data science & specifically the problem of visualizing large datasets, I realized the total lack of tooling in this space. The field of information visualization lags behind general "computer-generated imagery" by decades. When I first presented my ideas around Abstract Rendering (which became Datashader) to my DARPA collaborators, even to famous visualization people like Bill Cleveland or Jeff Heer, it was clear that I was thinking about the problem in an entirely different way. I recall our DARPA PM asking Hanspeter Pfister how he would visualize a million points, and he said, "I wouldn't. I'd subsample, or aggregate the data." Datashader eats a million points for breakfast. Since you're clearly a computer graphics guy, the way to think about this problem is not one of naive rendering, but rather one of dynamically generating correct primitives & aesthetics at every image scale, so that the viewer has the most accurate understanding of what's actually in the dataset. So it's not just a particle cloud, nor is it nurbs with a normal & texture map; rather, it's a bunch of abstract values from which a data scientist may want to synthesize any combination of geometry and textures. I chose the name "datashader" for a very specific and intentional reason: we are dynamically invoking a shader - usually a bunch of Python code for mathematical transformation - at every point, within a sampling volume (typically a square, but it doesn't have to be). One can imagine drawing a map of the rivers of the US, with the shading based on some function of all industrial plants in its watershed. Both the domain of integration and the function to evaluate are dynamic for each point in the view frustum.
- BubRoss 8y ago
  > Datashader eats a million points for breakfast. So does opengl on a decade old computer.

johnmarinelli 8y ago

Looks like a really cool project. One thing that I would be interested in see would be using Datashader as a dynamic visualisation library - for example, generative art projects. Probably not the main interest of data visualisation practitioners but hey, if you've got a sweet pipeline to render all those points, why not?

jbednar 8y ago
The attractors at http://datashader.org/topics/strange_attractors.html http://datashader.org/topics/strange_attractors.html are probably closer to art than science...

IanCal 8y ago

Datashader is a great project. Very fast, very easy to use. You can throw a lot of data at it in a notebook and get back a zoomable interactive pane. Here's a 2016 talk on it: https://www.youtube.com/watch?v=fB3cUrwxMVY https://www.youtube.com/watch?v=fB3cUrwxMVY There's likely a lot of improvements since then, but that should help show some of the core parts and explain why it's a useful tool.

simplyinfinity 8y ago

Is there anything similar for network graphs?

jointpdf 8y ago
Perhaps Gephi? It was used for those Graphcore neural network visualizations (millions of edges and nodes) and they look stunning. [1] https://gephi.org/features/ https://gephi.org/features/ [2] https://www.graphcore.ai/posts/what-does-machine-learning-look-like https://www.graphcore.ai/posts/what-does-machine-learning-lo...
- simplyinfinity 8y ago
  I've used both gephi and cytoscape, i have about 2 million nodes with about 3 million edges, it's hard to visualise at once. however i've reduced the graphs to only sub networks with more than 5 connections, the remainder is about 600k, which is still painfully slow to visualise
maliker 8y ago
My team has had luck rendering an SVG from the graph and sending it to a browser. It works well for about 10k vertices and edges. Above that scale we use datashader, and we're investigating a potential to move to QGIS. We tried Gephi a few years ago and it had trouble at these scales.
lmeyerov 8y ago
Yep -- at https://github.com/graphistry/pygraphistry https://github.com/graphistry/pygraphistry, we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc. We've been curious about server-side static tile rendering for larger graphs, but has been on the back-burner. (We already connect to GPUs on the server, so not rocket science.) Currently, we're actively increasing how much can be ingested + computed on, such as for finding influencers, communities & rings, etc. However, visualizing that hasn't been an operational priority for our users. More useful to generate the communities, and then either inspect individual ones, or see how communities stitch together: quickly run out of pixels otherwise due to too many edges. Likewise, we're building connectors to gigascale-petascale graph DBs: titan, janus, aws neptune, tigergraph, spark graphx, etc. We still are interested, but more for when we start supporting geographic maps: you can see that is the primary use for datashader. Also, because data art is fun :)

mhalle 8y ago

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or make the results look misleading or bad. Even better that they built a tool that works with existing libraries, rather than replacing them. Good work!

BubRoss 8y ago
> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
- detaro 8y ago
  Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.
  - BubRoss 8y ago
    Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
    - el_dev_hell 8y ago
      > Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. 1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.
      - 8y ago

itodd 8y ago

I've used datashader for plotting NGS (Next Generation Sequencing) enrichments. At the time I had to hack together the ability to use the polygon select tool on the data, but it worked and blew my mind. Very elegant solution to a difficult problem (overplotting).

abcc8 8y ago
Do you have any examples of this you could point to online? I am looking at different visualization tools for various NGS-based analyses currently.
- itodd 8y ago
  I do. I remember posting this to the mailing list. I don't have an example calculating enrichments though. We simply group by read and divide using the frequencies and then plot one enrichment vs another. This way we can see how one sequence enriches between conditions. There's more to it than that but this will produce a plot similar to the one in the attachment in the thread linked below. https://groups.google.com/a/continuum.io/forum/m/#!msg/bokeh/BdQV54W0WnY/OHjlMAtqBAAJ https://groups.google.com/a/continuum.io/forum/m/#!msg/bokeh... Edit: here's a link to the plot I refer to: https://10826817673355204906.googlegroups.com/attach/e6e58ade6248/Screen%20Shot%202017-05-04%20at%202.47.07%20PM.png?part=0.1&view=1&vt=ANaJVrHL6xR_HvljfCodWVnBwALwADyL8wIpWga5D7Q5DzAHYPOsTTpqg193EuAC81xxXb-toP7Nv8y724yMogPSNlLXeU6lZXJ-Trm3r5-zpIVHEe2fpUI https://10826817673355204906.googlegroups.com/attach/e6e58ad...
deleted 8y ago
[deleted]

24gttghh 8y ago

https://anaconda.org/jbednar/gerrymandering/notebook https://anaconda.org/jbednar/gerrymandering/notebook I wish more people were outraged at this kind of election tampering. Great visualizations though! Zoom in on some of those tight masses of black outlines. The shapes are ridiculous. Maryland 3rd? Come on.

deleted 8y ago
[deleted]

burtonator 8y ago

This actually gave me an interesting idea regarding bitcoin passphrase mnemonics. Instead of text we could use the same algorithm to generate images. So you could have an index of images and generate them. I'm actually wondering if you could use nouns and verbs to maybe make stories if you could mutate the nouns reliably. Like 'bird flying' vs 'bird sleeping' ... This could help to remember long passphrases visually which people seem to be better at.

whoisjuan 8y ago

What license is this project using? The repo has a license but besides the provisions listed there I don't see any standard license.

tnvaught 8y ago
The repo has a standard 3-clause BSD license.

tokyodude 8y ago

> Turns even the largest data into images, accurately The first image, the image of the USA, seems really mis-representative to me. LA and NYC should be way way way more bright in relation to everything else than the entire area east of the Mississippi. At least to my eyes that map makes it look like parts of Denver, Kansas City, Salt Lake City, Atlanta, and the San Joaquin Valley are just as dense as Manhattan. Atlanta's population density 630 per square mile Manhattan's population density 70826 per square mile It seems like an accurate data image would have Atlanta's brightness 1/100th of Manhattan's. Basically it looks like they saturated out at around 250 people do anything over 250 people is the same brightness.

jbednar 8y ago
By default, Datashader accurately conveys the shape of the distribution in a way that the human visual system can process. If you want a linear representation, you can do that easily; see the first plot in http://datashader.org/topics/census.html http://datashader.org/topics/census.html , but you'll quickly see that the resulting plot completely fails to show that there are any patterns anywhere besides the top few population hotspots, which is highly unrepresentative of the actual patterns in this data. There is no saturation here; what it's doing in the homepage image is basically a rank-order encoding, where the top brightness value is indeed shared by several high-population pixes, the next brightness value is shared by the next batch of populations, etc. Given only 256 possible values, there has to be some grouping, but it's not saturating.
pwang 8y ago
Yes, datashader actually gives you the ability to dial-in as much gamma compensation as you want, to account for the human visual system's nonlinear response to luminance.