6 ms·

As someone involved in computational geoscientific research, finding new ways to visualize and present data is a persistent challenge. We don't use Javascript m

As someone involved in computational geoscientific research, finding new ways to visualize and present data is a persistent challenge. We don't use Javascript much (read: ever) in day-to-day work, but I think there is enormous potential there for viz of post-processed data. This article gives a great intro to D3.js, and I'll be looking into it more. A colleague also recently showed me Dash by Plotly [0] which may be of interest to folks in a similar situation. [0]: https://dash-gallery.plotly.host/Portal/ https://dash-gallery.plotly.host/Portal/

psv1 7y ago
> We don't use Javascript much Not even Leaflet? I've been using its R interface and I think it's fairly easy to pick up.
- dlivingston 7y ago
  I'll take a look at it. For scripting, I'm mainly Python and Julia based, but R is definitely pretty common around here. Thanks for the info.
  - qorrect 7y ago
    What does Julia have that Python or R lack ?
    - dagw 7y ago
      Speed.
      - psv1 7y ago
        I like Julia as a language but the speed argument doesn't really apply vs Python. Yes, benchmarks against pure Python implementations look good but in reality no one uses naive pure-python implementations. Python (with all the bells and whistles that interop with C/C++ like numpy, pytorch, tensorflow, numba, cython...) is very performant.
        dagw 7y ago
        Python (with all the bells and whistles that interop with C/C++ like numpy, pytorch, tensorflow, numba, cython...) is very performant I know. I spend much of my days writing 'high performance' Python, and Python is still my go to tool for just about everything. Julia's big advantage is, kind of like Fortran vs C back in the day, that it's fast even if you're just writing things in the most obvious and natural way for your domain.
mikepurvis 7y ago
My issue with D3 and other DOM-oriented systems is that it's rare I have a small enough amount data that it would work for them. There's a certain amount preparation and filtering that can be done, but particularly with time-series or geographic data, if you're in exploratory mode, you want to be able to slice and dice as part of the interactive browser presentation.
- k__ 7y ago
  As far as I know, D3 can also render into a canvas, which should eliminate the DOM/SVG nodes overhead.
  - MurrayHill1980 7y ago
    Yes, in my experience this can get to considerable scale (hundreds of thousands and beyond) but at a price. You need to do all your own management for things DOM/SVG were doing (e.g. hit detection, styling). You still need to send the data to the client and probably store it locally (customers will enjoy learning how to start Chrome from the command line with flags like --args --js-flags="--max_old_space_size=8192"). The client environment still doesn't have much support for data management or processing unless you bring in some other JS libraries. I'm a big fan of JS development but the criticism is on target.
    - k__ 7y ago
      True story. There is no endless RAM, even for browsers :) DOM-nodes are rather heavy compared to simple JS objects, so getting rid of them can bring some visualizations that are too heavy for the browser back into the realm of doable. After that you have to optimize your other data-structures. If you have objects with many boolean fields or strings that are essentially used as enums, you can try to replace them with bit-fields, I read some library creators (BlueBird promise) got much performance gains with them.
- wattenberger 7y ago
  k is right that d3 has some helper functions for drawing on canvas, so most of the methods here will work: https://wattenberger.com/blog/d3#drawing-svg-shapes https://wattenberger.com/blog/d3#drawing-svg-shapes Canvas is way more performant than SVG for a lot of shapes, since they don't have the overhead of a DOM node. But in general, you want to work outside of the browser if you have that much data. What I'll usually do is either: - write a python script to process my data and use d3 for the final visualization, or - have an API that returns a sliced portion of the data and create an interface that will request different sliced datasets when interacting with filters, controls, etc. This election dashboard is a good example: https://currents.parsely.com/election https://currents.parsely.com/election it hits two endpoints, which return basically the maximum amount of data you can expect a browser to handle (the massive dataset is processed elsewhere). You could create an endpoint to return different views when you click on, eg, a different date, although then the user would have to wait for the data to load.
Micoloth 7y ago
I wanted to ask the same question: how does it compare to Dash? As a non-javascript user who does data visualization, I would love to have some advice about which technology(es) to invest time into

psv1 7y ago

> We don't use Javascript much Not even Leaflet? I've been using its R interface and I think it's fairly easy to pick up.

dlivingston 7y ago
I'll take a look at it. For scripting, I'm mainly Python and Julia based, but R is definitely pretty common around here. Thanks for the info.
- qorrect 7y ago
  What does Julia have that Python or R lack ?
  - dagw 7y ago
    Speed.
    - psv1 7y ago
      I like Julia as a language but the speed argument doesn't really apply vs Python. Yes, benchmarks against pure Python implementations look good but in reality no one uses naive pure-python implementations. Python (with all the bells and whistles that interop with C/C++ like numpy, pytorch, tensorflow, numba, cython...) is very performant.
      - dagw 7y ago
        Python (with all the bells and whistles that interop with C/C++ like numpy, pytorch, tensorflow, numba, cython...) is very performant I know. I spend much of my days writing 'high performance' Python, and Python is still my go to tool for just about everything. Julia's big advantage is, kind of like Fortran vs C back in the day, that it's fast even if you're just writing things in the most obvious and natural way for your domain.

mikepurvis 7y ago

My issue with D3 and other DOM-oriented systems is that it's rare I have a small enough amount data that it would work for them. There's a certain amount preparation and filtering that can be done, but particularly with time-series or geographic data, if you're in exploratory mode, you want to be able to slice and dice as part of the interactive browser presentation.

k__ 7y ago
As far as I know, D3 can also render into a canvas, which should eliminate the DOM/SVG nodes overhead.
- MurrayHill1980 7y ago
  Yes, in my experience this can get to considerable scale (hundreds of thousands and beyond) but at a price. You need to do all your own management for things DOM/SVG were doing (e.g. hit detection, styling). You still need to send the data to the client and probably store it locally (customers will enjoy learning how to start Chrome from the command line with flags like --args --js-flags="--max_old_space_size=8192"). The client environment still doesn't have much support for data management or processing unless you bring in some other JS libraries. I'm a big fan of JS development but the criticism is on target.
  - k__ 7y ago
    True story. There is no endless RAM, even for browsers :) DOM-nodes are rather heavy compared to simple JS objects, so getting rid of them can bring some visualizations that are too heavy for the browser back into the realm of doable. After that you have to optimize your other data-structures. If you have objects with many boolean fields or strings that are essentially used as enums, you can try to replace them with bit-fields, I read some library creators (BlueBird promise) got much performance gains with them.
wattenberger 7y ago
k is right that d3 has some helper functions for drawing on canvas, so most of the methods here will work: https://wattenberger.com/blog/d3#drawing-svg-shapes https://wattenberger.com/blog/d3#drawing-svg-shapes Canvas is way more performant than SVG for a lot of shapes, since they don't have the overhead of a DOM node. But in general, you want to work outside of the browser if you have that much data. What I'll usually do is either: - write a python script to process my data and use d3 for the final visualization, or - have an API that returns a sliced portion of the data and create an interface that will request different sliced datasets when interacting with filters, controls, etc. This election dashboard is a good example: https://currents.parsely.com/election https://currents.parsely.com/election it hits two endpoints, which return basically the maximum amount of data you can expect a browser to handle (the massive dataset is processed elsewhere). You could create an endpoint to return different views when you click on, eg, a different date, although then the user would have to wait for the data to load.

Micoloth 7y ago

I wanted to ask the same question: how does it compare to Dash? As a non-javascript user who does data visualization, I would love to have some advice about which technology(es) to invest time into