5 ms·

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying thing

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets into a territory of reinventing the wheel but naming it 'the flattened infinite curvature hypersphere'. So, before you get too self righteous at least realize that it is the delivery and lack of context and precedent, not the actual work that is the problem.

detaro 8y ago
I can't see a single reference to this being somehow something you couldn't do before in the linked page. It describes what it does, it doesn't make claims of superiority over other approaches, ...
johnmarinelli 8y ago
I suspect marketing Datashader as an "accumulation buffer" wouldn't have the same effect on its target audience (data visualisation developers) as presenting it simply as a way to "Turn data into images". I'm also curious (as a fledgling graphics programmer) - what leads you to believe that Datashader uses an accumulation buffer internally? I would think that they use some magic to draw all the points in a single draw call using instanced rendering, but I am very naive :)
- mhalle 8y ago
  If you watch the video, one of the creators explains this point, which makes datashader different from, say, Bokeh. (Pipeline explained around 7:20 in the video) https://www.youtube.com/watch?v=6m3CFbKmK_c https://www.youtube.com/watch?v=6m3CFbKmK_c D3 and Bokeh and other web-based visualization tools, in general, plot HTML or CSS primitives to the browser. This approach works great for smaller datasets, but doesn't scale to millions/billions. Datashader aggregates (accumulates) graphical representations of data into images, then provides a way to get those to the browser and work well with the other libraries. That high level description leaves out 95% of the critical practical details of visualization, which the creators of datashader handle.
- jbednar 8y ago
  Datashader's approach is a bit different from an accumulation buffer, though similar in principle. It's not 3D rendering, and has no need for a z ordering; instead it's essentially 2D histogramming. For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. The key benefit over something like SciPy's histogram2d functions are in how it is implemented and used -- highly optimized, and highly integrated with viz tools so that it can let you just interact with your data naturally as if you had infinite resolution. Try it and see!
  - BubRoss 8y ago
    > For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. That is literally what opengl does. If you mean a histogram per pixel in depth, that's literally voxels in perspective space. If there are usability benefits here, that's great, but everything seems to be centered around there being new rendering techniques here, when not only are they not new, they're completely trivial, with solidified names and formalized math.
- pwang 8y ago
  It's actually not merely an accumulation buffer. It's a shader pipeline that allows for arbitrary Python code to be executed at each stage of data processing. It's actually very much like "renderman for data", but with Python (via Numba, Dask for performance). The pipeline is also built in such a way that it permits front-end JS viewers like Bokeh to drive a very dynamic experience.
p49k 8y ago
“Turn data into images” is a much better phrase than “rendering.” Anyone who is not intentionally being pedantic for dramatic effect would understand the purpose of the product based on reading that phrase, whereas “Datashader: a renderer” is less clear and could refer to products with an entirely different scope/purpose. Their product page is well-written and accurate. It sounds like you want them to purposely describe their product as something that is inferior to what it actually is.
- jbednar 8y ago
  If you want one word, Datashader is a rasterizer. It takes data of many types (points, lines, grids, meshes) and creates a regular grid where each grid cell's value is a well defined function of the incoming data. Not sure anyone would be any happier with "rasterizer" than "renderer" or "shader" or any other single word...
mikepurvis 8y ago
It is novel in the sense that it combines the interactivity of a D3/Bokeh/whatever JS based visualization (typically limited to a few thousand points) with the massive data display capability of offline rendering.

detaro 8y ago

I can't see a single reference to this being somehow something you couldn't do before in the linked page. It describes what it does, it doesn't make claims of superiority over other approaches, ...

johnmarinelli 8y ago

I suspect marketing Datashader as an "accumulation buffer" wouldn't have the same effect on its target audience (data visualisation developers) as presenting it simply as a way to "Turn data into images". I'm also curious (as a fledgling graphics programmer) - what leads you to believe that Datashader uses an accumulation buffer internally? I would think that they use some magic to draw all the points in a single draw call using instanced rendering, but I am very naive :)

mhalle 8y ago
If you watch the video, one of the creators explains this point, which makes datashader different from, say, Bokeh. (Pipeline explained around 7:20 in the video) https://www.youtube.com/watch?v=6m3CFbKmK_c https://www.youtube.com/watch?v=6m3CFbKmK_c D3 and Bokeh and other web-based visualization tools, in general, plot HTML or CSS primitives to the browser. This approach works great for smaller datasets, but doesn't scale to millions/billions. Datashader aggregates (accumulates) graphical representations of data into images, then provides a way to get those to the browser and work well with the other libraries. That high level description leaves out 95% of the critical practical details of visualization, which the creators of datashader handle.
jbednar 8y ago
Datashader's approach is a bit different from an accumulation buffer, though similar in principle. It's not 3D rendering, and has no need for a z ordering; instead it's essentially 2D histogramming. For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. The key benefit over something like SciPy's histogram2d functions are in how it is implemented and used -- highly optimized, and highly integrated with viz tools so that it can let you just interact with your data naturally as if you had infinite resolution. Try it and see!
- BubRoss 8y ago
  > For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. That is literally what opengl does. If you mean a histogram per pixel in depth, that's literally voxels in perspective space. If there are usability benefits here, that's great, but everything seems to be centered around there being new rendering techniques here, when not only are they not new, they're completely trivial, with solidified names and formalized math.
pwang 8y ago
It's actually not merely an accumulation buffer. It's a shader pipeline that allows for arbitrary Python code to be executed at each stage of data processing. It's actually very much like "renderman for data", but with Python (via Numba, Dask for performance). The pipeline is also built in such a way that it permits front-end JS viewers like Bokeh to drive a very dynamic experience.

p49k 8y ago

“Turn data into images” is a much better phrase than “rendering.” Anyone who is not intentionally being pedantic for dramatic effect would understand the purpose of the product based on reading that phrase, whereas “Datashader: a renderer” is less clear and could refer to products with an entirely different scope/purpose. Their product page is well-written and accurate. It sounds like you want them to purposely describe their product as something that is inferior to what it actually is.

jbednar 8y ago
If you want one word, Datashader is a rasterizer. It takes data of many types (points, lines, grids, meshes) and creates a regular grid where each grid cell's value is a well defined function of the incoming data. Not sure anyone would be any happier with "rasterizer" than "renderer" or "shader" or any other single word...

mikepurvis 8y ago

It is novel in the sense that it combines the interactivity of a D3/Bokeh/whatever JS based visualization (typically limited to a few thousand points) with the massive data display capability of offline rendering.