4 ms·

Perspective: Streaming Analytics via WebAssembly

36 points by ArtWomb 6y ago

TrueDuality 6y ago
Seems neat but doesn't seem to work in the latest Firefox (though I do have a hardened user config, I don't think its relevant here). The error showing up in the console is "SyntaxError: invalid regexp group".
- rpastuszak 6y ago
  Ugh, went back to HN to post the same comment. On FF this usually means that the regexp uses `lookbehind`: https://caniuse.com/#feat=js-regexp-lookbehind https://caniuse.com/#feat=js-regexp-lookbehind
- texodus 6y ago
  Apologies, this was fixed a few days ago but hasn't made it into a release yet. https://github.com/finos/perspective/pull/1065 https://github.com/finos/perspective/pull/1065
XCSme 6y ago
Are you able to avoid going through HTTP requests and use directly UDP for streaming when using WebAssembly? Or the only advantages of WebAssembly are the improved client-side rendering and processing performance? In that case, if client-side processing speed is important, wouldn't it be a lot more important and faster to render the output using OffscreenCanvas than by frequendly updating DOM elements?
- glass_of_water 6y ago
  Currently, WebAssembly doesn't can't directly call any of the browser APIs; it has to call back into javascript to make any network requests, make WebGL calls, read to/write from the DOM. Projects like wasm-bindgen make it look like you're directly calling browser APIs, but it's actually calling out to a Javascript shim. I think this project's use case is that you have a fixed query, but the dataset over which you're running that query is changing over time (getting streaming updates). The wasm part is doing the work of running the query over the dataset and updating the query results as new data streams in. So the assumption is that rendering isn't the bottleneck, but running the query is.
  - XCSme 6y ago
    So the dataset would be a local dataset (that gets frequently updated from the network) and the query is a local query that runs over that dataset? I still am a bit confused about the use-case.
    - XCSme 6y ago
      Oh, I still I'm starting to get it. Is it if you have a large dataset locally stored (eg. in JS memory) that receives constant updates from a server? Are there any example apps that have this architecture?
      - infinite8s 6y ago
        Any kind of pivoting/roll-up over streaming updates (financial trading, IOT, transportation). A lot of streaming dashboards across a number of industries are effectively just pivot tables over ticking data.
- texodus 6y ago
  WebAssembly cannot utilize UDP, or any other APIs not exposed by the Browser - just HTTP and Websocket in this case. However, Perspective can read Apache Arrow using the C++ client library without decoding the data structure to an intermediate JS representation first, which in our benchmarks has a substantial impact on overall update throughput.
- PaulWaldman 6y ago
  I think WebAssembly System Interface will address using other protocols. https://github.com/WebAssembly/WASI https://github.com/WebAssembly/WASI
deleted 6y ago
[deleted]

36 points by ArtWomb 6y ago

Seems neat but doesn't seem to work in the latest Firefox (though I do have a hardened user config, I don't think its relevant here). The error showing up in the console is "SyntaxError: invalid regexp group".

rpastuszak 6y ago
Ugh, went back to HN to post the same comment. On FF this usually means that the regexp uses `lookbehind`: https://caniuse.com/#feat=js-regexp-lookbehind https://caniuse.com/#feat=js-regexp-lookbehind
texodus 6y ago
Apologies, this was fixed a few days ago but hasn't made it into a release yet. https://github.com/finos/perspective/pull/1065 https://github.com/finos/perspective/pull/1065

XCSme 6y ago

Are you able to avoid going through HTTP requests and use directly UDP for streaming when using WebAssembly? Or the only advantages of WebAssembly are the improved client-side rendering and processing performance? In that case, if client-side processing speed is important, wouldn't it be a lot more important and faster to render the output using OffscreenCanvas than by frequendly updating DOM elements?

glass_of_water 6y ago
Currently, WebAssembly doesn't can't directly call any of the browser APIs; it has to call back into javascript to make any network requests, make WebGL calls, read to/write from the DOM. Projects like wasm-bindgen make it look like you're directly calling browser APIs, but it's actually calling out to a Javascript shim. I think this project's use case is that you have a fixed query, but the dataset over which you're running that query is changing over time (getting streaming updates). The wasm part is doing the work of running the query over the dataset and updating the query results as new data streams in. So the assumption is that rendering isn't the bottleneck, but running the query is.
- XCSme 6y ago
  So the dataset would be a local dataset (that gets frequently updated from the network) and the query is a local query that runs over that dataset? I still am a bit confused about the use-case.
  - XCSme 6y ago
    Oh, I still I'm starting to get it. Is it if you have a large dataset locally stored (eg. in JS memory) that receives constant updates from a server? Are there any example apps that have this architecture?
    - infinite8s 6y ago
      Any kind of pivoting/roll-up over streaming updates (financial trading, IOT, transportation). A lot of streaming dashboards across a number of industries are effectively just pivot tables over ticking data.
texodus 6y ago
WebAssembly cannot utilize UDP, or any other APIs not exposed by the Browser - just HTTP and Websocket in this case. However, Perspective can read Apache Arrow using the C++ client library without decoding the data structure to an intermediate JS representation first, which in our benchmarks has a substantial impact on overall update throughput.
PaulWaldman 6y ago
I think WebAssembly System Interface will address using other protocols. https://github.com/WebAssembly/WASI https://github.com/WebAssembly/WASI

deleted 6y ago

[deleted]