3 ms·

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebAS

I'd also be interested in some performance comparisons with a library like CrossFilter [1]. Does the improvement outweigh the penalties of crossing the JS/WebASM boundary? [1] http://square.github.io/crossfilter/ http://square.github.io/crossfilter/

texodus 9y ago
The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.
- BenGosub 9y ago
  bringing Apache Arrow in the browser alongside wasm is exciting to say the least! Amazing capabilities are coming to browsers...
  - lmeyerov 9y ago
    We (Graphistry) recently contributed a native JS reader/writer into the Apache Arrow project, so may help both teams! We did it as legwork for our beyond-native efforts (GPU cloud streaming) and taming our JS datatypes, so similar needs here I'm guessing! Funny enough: was in NYC today talking with banking teams about related tech. Too bad we didn't know about this effort, would have loved to meet!
    - dman 9y ago
      Drop me a note the next time you are in NY, would love to meet up.
    - texodus 9y ago
      Yes, this is the library we use. We have met before actually, you did a demo at JPM in midtown several years back. Graphistry has come a very long way since then - impressive work!
      - lmeyerov 9y ago
        Ah, small world. That was probably when we first started on client<>server GPU streaming. Looking forward to digging into the Perspective source!
- polskibus 9y ago
  Can you elaborate on how that could work? Does arrow really allow for abstracting away the need for serialization even in JS - server scenarios? I though it was more of a shared memory data frame utility ?
  - texodus 9y ago
    There is an early arrow example in the examples package - superstore-arrow.html. The idea is that, instead of converting your data to internet->text->JSON->ArrayBuffer, you just keep the data in binary and write it directly into the C++ heap ArrayBuffer as-is. We currently do not read this in C++ directly for various reasons related to how emscripten allocates memory, but the general idea is the same.
infinite8s 9y ago
In addition to what texodus has said below, Crossfilter only implements a small subset of what perspective provides. For example, no streaming (although there is a related project from the Heer lab that supports incremental updates - https://github.com/jheer/datavore https://github.com/jheer/datavore), only a single level of grouping, only in 1 dimension, and you can only support 16 dimension fields at once (without increasing a constant in the codebase).

texodus 9y ago

The boundary-crossing is definitely the bottleneck right now. We are currently putting alot of work into the Apache Arrow support specifically to avoid this crossover, which will allow us to send data from the server in binary and avoid parsing in the browser.

BenGosub 9y ago
bringing Apache Arrow in the browser alongside wasm is exciting to say the least! Amazing capabilities are coming to browsers...
- lmeyerov 9y ago
  We (Graphistry) recently contributed a native JS reader/writer into the Apache Arrow project, so may help both teams! We did it as legwork for our beyond-native efforts (GPU cloud streaming) and taming our JS datatypes, so similar needs here I'm guessing! Funny enough: was in NYC today talking with banking teams about related tech. Too bad we didn't know about this effort, would have loved to meet!
  - dman 9y ago
    Drop me a note the next time you are in NY, would love to meet up.
  - texodus 9y ago
    Yes, this is the library we use. We have met before actually, you did a demo at JPM in midtown several years back. Graphistry has come a very long way since then - impressive work!
    - lmeyerov 9y ago
      Ah, small world. That was probably when we first started on client<>server GPU streaming. Looking forward to digging into the Perspective source!
polskibus 9y ago
Can you elaborate on how that could work? Does arrow really allow for abstracting away the need for serialization even in JS - server scenarios? I though it was more of a shared memory data frame utility ?
- texodus 9y ago
  There is an early arrow example in the examples package - superstore-arrow.html. The idea is that, instead of converting your data to internet->text->JSON->ArrayBuffer, you just keep the data in binary and write it directly into the C++ heap ArrayBuffer as-is. We currently do not read this in C++ directly for various reasons related to how emscripten allocates memory, but the general idea is the same.

infinite8s 9y ago

In addition to what texodus has said below, Crossfilter only implements a small subset of what perspective provides. For example, no streaming (although there is a related project from the Heer lab that supports incremental updates - https://github.com/jheer/datavore https://github.com/jheer/datavore), only a single level of grouping, only in 1 dimension, and you can only support 16 dimension fields at once (without increasing a constant in the codebase).