4 ms·

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little bu

by mhalle 8y ago

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or make the results look misleading or bad. Even better that they built a tool that works with existing libraries, rather than replacing them. Good work!

BubRoss 8y ago
> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
- detaro 8y ago
  Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.
  - BubRoss 8y ago
    Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
    - el_dev_hell 8y ago
      > Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. 1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.
      - BubRoss 8y ago
        Once again, there is no need for very much to be in memory at one time. This is accumulation and buffers can be reused. I'm not sure how much more I can simplify this other than to say that addition doesn't care about ordering, you can just add continually without needing access to any of the other data that has been rasterized.
        jbednar 8y ago
        I think this was already said above, but it still seems to be getting confused, so to repeat: Datashader renders everything out of core, in the server. So it doesn't matter whether a client could successfully accumulate results for a large data incrementally; to use WebGL directly one still has to send all of the data to the client eventually. With Datashader the dataset is never sent to the client in the first place; it stays on the server, which could be a remote HPC system with thousands of cores processing petabytes. Datashader renders the data into an image-shaped array on the server, then sends that (much smaller) array to the client, so that the client never sees any data larger than the available screen resolution. This is no claim that doing so is unprecedented or some crazy new idea, just that Datashader lets you render datasets regardless of their size, completely independently of any client (browser) limitations, and without having to serialize the data over an internet connection.
- mhalle 8y ago
  No, it can't. WebGL is at a completely different level of the stack than Datashader. WebGL is even lower level than what most people use for 3D graphics (hence threejs and babylonjs). Data visualization is at a higher semantic level than rendering; ideally you don't want to deal with pixels and polygons. D3, for instance, binds graphical primitives (usually in SVG) with data representations but requires more programming to do actual data visualization (and that's why a bunch of software layers on top of D3). Bokeh deals with still higher level primitives closer to the data set level (plotting and charts). And Datashader carves out a niche where there's too much data to have a 1:1 ratio of data element to graphical primitive on the screen. It does that by rasterizing, but then also handling the hard part of mapping backwards from image to data for selection and interactivity (I hope that's right; I got it from watching the 2016 video). Anyone who has had to do this stuff for a living knows it is hard to do right, and that good modular tools are always welcome. I don't see how this repeated "it's just like X" line of responses is benefiting the discussion. Datashader is not just like WebGL or basic low-level rendering, any more than D3 is just SVG or web apps are just TCP connections. Completely different levels of abstraction with lots of value add in between (and hard work, I'm sure).
  - BubRoss 8y ago
    I didn't say that webgl would do everything this can do, I said it would do the things I copied from the post I replied to. Again, my issue is that they seem to put a lot of focus on this having some sort of sophisticated new rendering when it seems to be marketing of trivial techniques. People seem to like what this library does, but they didn't invent new rendering algorithms and their buzzwords and clever names just show a lack of awareness of what they are doing in the rendering department.
    - jbednar 8y ago
      I'm not sure if you're objecting to the name "Datashader", but surely every library needs a name, and this one is accurate in that it allows the sort of shading that one does for 3D rendering to be applied to 2D data plotting. Or are there other buzzwords used in the docs you find objectionable?
- jbednar 8y ago
  Datashader is server-side rendering, and thus not in any way comparable to WebGL in its usage. With Datashader, only the final rendered/rasterized image-like object is sent to the client, which lets it handle arbitrarily large datasets (anything your remote servers can process). With WebGL the dataset is sent to the browser for rendering, which has some advantages but is a very different process than what Datashader does.

by mhalle 8y ago

BubRoss 8y ago

> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.

detaro 8y ago
Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.
- BubRoss 8y ago
  Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
  - el_dev_hell 8y ago
    > Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. 1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.
    - BubRoss 8y ago
      Once again, there is no need for very much to be in memory at one time. This is accumulation and buffers can be reused. I'm not sure how much more I can simplify this other than to say that addition doesn't care about ordering, you can just add continually without needing access to any of the other data that has been rasterized.
      - jbednar 8y ago
        I think this was already said above, but it still seems to be getting confused, so to repeat: Datashader renders everything out of core, in the server. So it doesn't matter whether a client could successfully accumulate results for a large data incrementally; to use WebGL directly one still has to send all of the data to the client eventually. With Datashader the dataset is never sent to the client in the first place; it stays on the server, which could be a remote HPC system with thousands of cores processing petabytes. Datashader renders the data into an image-shaped array on the server, then sends that (much smaller) array to the client, so that the client never sees any data larger than the available screen resolution. This is no claim that doing so is unprecedented or some crazy new idea, just that Datashader lets you render datasets regardless of their size, completely independently of any client (browser) limitations, and without having to serialize the data over an internet connection.
mhalle 8y ago
No, it can't. WebGL is at a completely different level of the stack than Datashader. WebGL is even lower level than what most people use for 3D graphics (hence threejs and babylonjs). Data visualization is at a higher semantic level than rendering; ideally you don't want to deal with pixels and polygons. D3, for instance, binds graphical primitives (usually in SVG) with data representations but requires more programming to do actual data visualization (and that's why a bunch of software layers on top of D3). Bokeh deals with still higher level primitives closer to the data set level (plotting and charts). And Datashader carves out a niche where there's too much data to have a 1:1 ratio of data element to graphical primitive on the screen. It does that by rasterizing, but then also handling the hard part of mapping backwards from image to data for selection and interactivity (I hope that's right; I got it from watching the 2016 video). Anyone who has had to do this stuff for a living knows it is hard to do right, and that good modular tools are always welcome. I don't see how this repeated "it's just like X" line of responses is benefiting the discussion. Datashader is not just like WebGL or basic low-level rendering, any more than D3 is just SVG or web apps are just TCP connections. Completely different levels of abstraction with lots of value add in between (and hard work, I'm sure).
- BubRoss 8y ago
  I didn't say that webgl would do everything this can do, I said it would do the things I copied from the post I replied to. Again, my issue is that they seem to put a lot of focus on this having some sort of sophisticated new rendering when it seems to be marketing of trivial techniques. People seem to like what this library does, but they didn't invent new rendering algorithms and their buzzwords and clever names just show a lack of awareness of what they are doing in the rendering department.
  - jbednar 8y ago
    I'm not sure if you're objecting to the name "Datashader", but surely every library needs a name, and this one is accurate in that it allows the sort of shading that one does for 3D rendering to be applied to 2D data plotting. Or are there other buzzwords used in the docs you find objectionable?
jbednar 8y ago
Datashader is server-side rendering, and thus not in any way comparable to WebGL in its usage. With Datashader, only the final rendered/rasterized image-like object is sent to the client, which lets it handle arbitrarily large datasets (anything your remote servers can process). With WebGL the dataset is sent to the browser for rendering, which has some advantages but is a very different process than what Datashader does.