15 ms·
Perspective: Streaming pivot visualization via WebAssembly
- dman 9y agoI wrote the C++ part of this codebase, would be happy to answer any questions about it here.
- timkpaine 9y ago[]
- dman 9y agoOfftopic :)
- deleted 9y ago[deleted]
- icefox 9y agoWhat was the best/worst part about getting it to work in WebAssembly?
- dman 9y agoBest part was the validation of a design decision of having minimal dependencies. It made the web assembly port almost trivial. Worst part (which has improved dramatically) was the rough edges in the web assembly toolchains early on.
- nickthemagicman 9y agoThanks for taking questions. For a back end developer that's watching the front end landscape change from afar.. Do you feel like web assembly could potentially replace JS in the future with, say, my favorite back end language of choice?
- jameslk 9y agoNot the op, but there's more to frontend development than JS. You need to account for the API of browsers, the DOM, CSS and the myraid of frameworks and libraries built for the specific UI requirements of frontend development (e.g. React). So even if your favorite language of choice becomes compilable to WebAssembly, you'll still be learning an entirely different way of development that will be completely unfamiliar regardless. The only thing I see languages compiled to WebAssembly taking over is development of highly resource intensive applications, algorithms or libraries. Or if there's a quantum leap with some specific language + framework that eliminates a lot of work with current frontend development. Otherwise, there will be too much fragmentation and you'll see the regression to the mean effect as has been the case with predecessors such as CoffeeScript, Elm, PureScript, ReasonML, and every other compile-to-JS language du jour.
- ChicagoDave 9y agoA paradigm shift could happen and what's cool is that it could be entirely something new. This reminds me of the early days of the WWW when the first programming paradigm was Perl/CGI, then Cold Fusion, then Legacy ASP, and then ActiveX, and so on. The possibility of some company creating, from scratch, a new web development paradigm, given everything we know today, is exciting.
- jameslk 9y agoSure, and based on historical precedent, that's almost guaranteed to happen. But I doubt it will be whatever-my-favorite-backend-language-of-choice-is and more likely some-domain-specific-language-and-new-framework. In other words, we'll all be relearning everything again 5 years from now.
- goatlover 9y agoJava applets were supposed to provide that in the 90s. Netscape was going to actually embed Java in the browser as an alternative to JavaScript, but there wasn't enough time, so they just shipped with JS, and Java was provided via a plugin.
- zimablue 9y agoI see it's possible to embed it in jupyterlab, it seems to do a lot: * grid format * graph format * actual pivoting of data Is that right? It's using webassembly so in effect to do the pivot you must have created functions that exist in pandas? Has anyone ported pandas to webassembly? Is the grid editable? What are you using to create your output, html or canvas or something? I realise I could look this stuff up but since you asked..
- texodus 9y agoI believe you'd need to port Python to WebAssembly to make much use of pandas - unaware of any projects attempting the former. The grid is just a plugin for the excellent [Hypergrid](https://github.com/openfin/fin-hypergrid https://github.com/openfin/fin-hypergrid), which is editable, but you'd need to manually push those edits back to the engine for now.
- zimablue 9y agowhich answers my other question, so it's canvas?
- texodus 9y agoThe frontend is actually pure JS/HTML - the WebAssembly part is just the data engine, and runs in a WebWorker for CPU isolation. Hypergrid is a canvas-based renderer, yes.
- Laaas 9y agoNuitka can compile Python to LLVM IR, which can be compiled into WASM, I think.
- fulafel 9y agoAs historical background: Python was one of the early emscripten asm.js experiments: https://github.com/kripken/emscripten/tree/master/tests/python https://github.com/kripken/emscripten/tree/master/tests/pyth...
- 9y ago
- pavlov 9y agoIs Emscripten the only game in town for WebAssembly C/C++ toolchains, or are there other contenders?
- dman 9y agoAs far as I know yes for C/C++ this was the only viable toolchain that is publicly available.
- Klathmon 9y agoEven outside of C/C++, Emscripten seems to be required in some fashion for just about every "compile X to webasm". Which i'm not that happy about as it's quite heavy and takes a long time to compile, which makes it harder for newcomers to try it out because it requires a fairly large time investment to even compile hello world.
- bbatha 9y agoRust’s wasm32-unknown-unknown target does not use emscripten at all. The tool cargo-web will handle the js module shims for you.
- Game_Ender 9y agocargo-web uses emscripten under the hood.
- a_humean 9y agoNo, he is right. cargo-web supports three backends, of which only two include emscripten and the third using a native rust backend. https://github.com/koute/cargo-web#features https://github.com/koute/cargo-web#features Rust can be compiled to wasm without emscripten using wasm32-unknown-unknown.
- bringtheaction 9y agoDoes that mean I can install cargo-web without emscripten or do I still need it even though I am not going to use it just because the two other targets use it? In other words, is emscripten an optional or a hard dependency?
- jarpineh 9y agoBriefly in the animation I see there's hierachical (or multi-dimensional) data used. Could you tell me about what sort of data (statistics, monitoring) this supports? Also, what methods are available for getting data streamed [into Perspective]? I'm skimming through the docs, but am not quite getting it... [edited for clarity]
- texodus 9y agoRight now, you can call a load() or update() method on an engine or widget to pass JSON, CSV or Apache Arrow data (as an ArrayBuffer) to the engine. Apologize for the documentation clarity - this is our first release and very much under active construction!
- jarpineh 9y agoOk, thank you for this. I'll have to see some time about that Arrow support. JSON and CSV is already plenty enough. I didn't have time for a proper look yet, so no worries. If at all possible post the code and data (or something like it) that is in those animations you have. That would perhaps allow users to dive in by tinkering here and there.
- stupidcar 9y agoDid you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?
- dman 9y agoNo, when I first prototyped it I had written a Python version initially. From memory the first C++ build was ~80x faster than the Python prototype.
- carapace 9y agoI'm curious if you did any profiling? If the intent was always to rewrite the whole prototype in C++ then it kinda wouldn't matter, eh?
- texodus 9y agoWe do some benchmarking just for catching performance regressions and validating assumptions against the non-WebAssembly version, but definitely need more in this regard. [benchmark for wasm build](https://jpmorganchase.github.io/perspective/examples/benchmark.html https://jpmorganchase.github.io/perspective/examples/benchma...)
- goatlover 9y ago80x faster than using Pandas? What if you just used the Numpy values array?
- dman 9y agoYou will get me into trouble! I said 80x faster than the Python prototype. I did benchmark this vs Pandas over the years and depending on the usecase the two codebases trade blows with each other for static data. However for streaming datasets Perspective has a large advantage over Pandas since pandas does a full group by on every update, while perspective does its work incrementally using deltas.
- 9y ago
- polskibus 9y agoDoes it support: virtualization (showing only a view of full dataset that is updated from server on scroll) Pivoting (multiple levels/hierarchies on rows and on columns) Apart from features what part did you consider was the biggest hurdle? What browsers did you target?
- texodus 9y agoIt supports virtualization, and it is utilized by both plugins (Hypergrid and Highcharts). N levels of pivoting on both axes. Should work on all browsers - if not, please open an issue!
- polskibus 9y agoIs there a way to drill down a node dynamically, by clicking on row header during pivoting, and then see its children?
- deleted 9y ago[deleted]
- texodus 9y agoNot through the UI yet, only in the engine API - but the pivoting itself is quite fast, so should still be suitable for drilling down "on the fly" so to stpeak
- polskibus 9y agoBy virtualization I meant also not loading entire dataset to memory, but being able to work only on a constrained view of the data. For example, database has 1TB dataset, I only view 100 MB, further columns/sections are loaded when I scroll.
- texodus 9y agoIt is virtual in the sense that it does not realize the entire dataset in the JS side of the WebAssembly bridge - this is a performance optimization but entirely in-memory. While you can run the engine in node.js and use it to efficiently stream updates to a symmetric engine in the browser, we do not currently implement server-virtualized views. Though, we have quite alot of experience doing this in the past - and the design of Perspective is very much a reaction to what we learned, at lesat in regards to the typical financial dataset which is much smaller than 1TB.
- LukeB42 9y agoI see it's possible to convert to Numpy arrays. Is `perspective` also a Python library?
- dman 9y agoYes for much of its life perspective has been a python library. It offers a streaming dataframe abstraction in Python. I believe the Python bindings have not been open sourced yet.
- polskibus 9y agoCan you say how old is this codebase and what was its original business case?
- dman 9y agoCodebase started in 2013. Original business case was to allow business users to create filtered/aggregated views on top of streaming data.
- polskibus 9y agoWhat's on the roadmap?
- fitzoh 9y agoJust stalked you on linkedin with a similar question, but I was wondering if you were involved with the actual process of getting the library open sourced, and how that works at a large security focused company like JPMorgan Chase. (former Chase employee)
- hackcasual 9y agoFor those interested, keep in mind the only way to use WebAssembly with CSP in Chrome is by turning on 'unsafe-eval'. FF/Edge/Safari all at least support compilation from URLs with more locked down policies
- texodus 9y agoTrue - worth pointing out that this library gracefully falls back to ASM.js if your browser does not support WebAssembly, though.
- hackcasual 9y agoDoes it fallback if Window.WebAssembly doesn't exist, or does it fallback if WebAssembly.instantiateStreaming fails?
- texodus 9y agoThe former. In practice, it seems some runtimes are still a bit buggy - so we also revert to ASM.js if e.g. you are on an iPhone
- bpicolo 9y agoThis is great. Finance people especially love this sort of thing - they do want to browse through entire datasets vs seeing higher level metrics in a lot of cases.
- JumpCrisscross 9y ago> Finance people especially love this sort of thing - they do want to browse through entire datasets vs seeing higher level metrics in a lot of cases To expand on this, a lot of finance is searching out and understanding, normalizing or exploiting edge cases.
- andrewmcwatters 9y ago@dman Cool, but I'm curious why WebAssembly was chosen for this project. What was the value of using it here versus just using the Web API?
- infinite8s 9y agoWhat do you mean when you say the Web API?
- lucozade 9y agoThe C++ probably wasn't written for a browser. More likely it was written for a native frontend and has been ported to the browser via WebAssembly.
- dman 9y agoThe following features made wasm compelling here a. Filter / Pivot / Aggregate moving clientside means a lot of load off the servers. b. Performance in wasm is surprisingly close to same C++ code running natively on the desktop. c. Ability to reuse existing C++ codebase and use it to build both native and web apps.
- ChicagoDave 9y agoPoor naming choice. Google Perspective has been around for a year or two I believe.
- timkpaine 9y agoin that case, its actually a good naming choice, as this existed (in a closed source form) for much longer than that.
- ChicagoDave 9y agoI'm not knocking the code...I think it's brilliant. I just think you're going to cross wires with Google here. Will perspective mean "filtering troll comments" or will it mean "displaying analytics in web assembly" is an unfortunate set of choices.
- timkpaine 9y agoyou'd be surprised how many naming collisions we have, I assume most teams that code-name their software hit the same issue when they go to open source
- dman 9y agoThe name had been in place internally since 2013, so the naming ship had sailed a long time ago.
- whatok 9y agoGiven appropriate context, I find it hard to see someone confusing the two.
- gaius 9y agoGo was already a programming language when Google steamrollered the name!
- rrdharan 9y agoInteresting, I had no idea: https://en.wikipedia.org/wiki/Go!_(programming_language) https://en.wikipedia.org/wiki/Go!_(programming_language)
- mi100hael 9y agoReally cool to see non-software companies putting cool projects like this out as open source
- ashtonian 9y agothe stock market is digital, they are a software company.
- vthallam 9y agoThey consider themselves a banking company. It's about the culture, so it is indeed refreshing to see non-traditional tech companies open sourcing things.
- lotyrin 9y agoHaving worked at a branch off a university devoted to its online school and content marketing where the inputs and outputs of production are entirely digital, I can say they sure as hell aren't a software company; nobody above on-the-ground grunts think much about even the big pictures of domain modeling / workflows, data ingestion / transformation, computing / efficiency / performance, software development lifecycle / product portfolio management / QA, growth hacking / optimization / testing etc. because they simply aren't interested in the slightest. I'm sure plenty of organizations with at least one foot in pre-internet industries are in the same boat, and it's exciting to see counterexamples.
- jnordwick 9y agoI think it is more correct to say that a large subunit of JPM works as a software company -- for its internal use. All modern financial companies are heavily tech oriented, even those you wouldn't normally think of that way. The best tech firms I've ever been involved with financial firms from front desk trading on back. One of the best tech teams I've ever met with has been one of Bloomberg's teams dealing with their internal core services and bond platform (I was totally floored at how amazing these guys were). These companies are too large to say they are entirely software, but there are definitely Directors and CTOs that live and breath all those things you mention.
- ChicagoDave 9y agoI love seeing new UX on Web Assembly. Someone needs to port or create a standard layout engine like WPF to build on this. If the tools were there, C# + VS Code + solid layout engine...I'd dump JS + HTML + CSS in a "flash".
- pjmlp 9y agoThere are some attempts on that direction. https://www.hanselman.com/blog/NETAndWebAssemblyIsThisTheFutureOfTheFrontend.aspx https://www.hanselman.com/blog/NETAndWebAssemblyIsThisTheFut...
- ChicagoDave 9y agoSure. I've played with Blazor. It's very raw at this point and just a side project. I'd like to see MS Research or even a full blown R&D team at MS go full force at this.
- pjmlp 9y agoMicrosoft just announced that they will start investing more into it, depending how the community takes it. https://blogs.msdn.microsoft.com/webdev/2018/02/06/blazor-experimental-project/ https://blogs.msdn.microsoft.com/webdev/2018/02/06/blazor-ex...
- h2g3yhogw8 9y agomy question is how do game engines do styling ?
- elwell 9y ago"J.P. Morgan open sources..." -- that's so cool
- ashtonian 9y agoGlad to see the new bitcoin department is hard at work ..
- deleted 9y ago[deleted]
- texodus 9y agoIf anyone finds this project exciting and is interested in learning more about working on Open Source at J.P.Morgan, feel free to send me a message - we are always looking to hire experienced, passionate talent!
- dman 9y ago+1 - Even though I do not work there anymore, I can vouch for the fact that the team has amazing engineers and works on fundamental CS problems.
- feel_the_need 9y agoI would like to work on cakeshop-- how do I get on touch with you?
- deleted 9y ago[deleted]
- mortdeus 9y agoHey man, two thumbs up on this. Wall street as a contributor to open source. Who do I thank? Trump?
- deleted 9y ago[deleted]
- kchoudhu 9y agoIt's open sourced from an investment bank, so you'll never see it updated again. Beware of integrating it into your projects. Good on whoever took on legal to get this out the door though. Hope you make it to MD or whatever it is you were gunning for.
- testnet1 9y agoyou can see the stats for yourself. its under active development
- texodus 9y agoWe very much plan to continue developing this entirely in the open! It is reputationally important we do not just dump our dead projects on the internet as Open Source, which is one of the reason why we chose Perspective for this project in the first place.
- adamsea 9y agoFirst, let me say this looks awesome! I’m curious if your considered React.js for the presentation layer (i.e. the js/html5 parts) - on the surface it seems like it would be a good fit - and if so, what you saw as the benefits or drawbacks of it vs the approach you went with.
- dman 9y agoWill defer to texodus on that - he is the front end expert here.
- texodus 9y agoWe chose to go with a Web Components based interface for compatibility across frameworks, but alot of where we go in the future will be determined by the expertise of the developers we hire to work on it, and the community if there is interest there.
- drglitch 9y agoFirst of all, great work everyone involved! I know first-hand how hard it is to push something like this past compliance ;) Going with WC is a sound decision since its trivial to wrap it into ember, react, or any other framework-du-jour... especially so since more and more of them are adopting WC patterns and design practices. I might try to wrap this into react and will see if bosses permit opensourcing it!
- adamsea 9y agoI hadn’t realized you were using Web Components. That makes sense. Thank you.
- flight21 9y agoWhat are you using for UI, any framework (Polymer, D3) or just plain WebComponent?
- texodus 9y agoPlain web components. The UI is quite light in terms of functionality so far.
- baybal2 9y agoSo, what there is about visualisation? I read your docs, and found nothing about how to actually render stuff to the page.
- polskibus 9y agoAmazing, despite the fact that some heavy-lifting is done by highcharts and hypergriid. What's missing for this to effectively kill all typical BI tools like PowerBI, Tableau?
- deleted 9y ago[deleted]
- infinite8s 9y agoExpensive sales teams. Also neither of those handles streaming data.
- tgtweak 9y agoThe backend.
- bcherny 9y agoPresumably, the point of using WASM and service workers is for performance, but I see no benchmarks. I also have trouble imagining that this actually improves performance, unless you’re doing all of your compute in the browser and it’s CPU bound (this seems like bad design). What is the performance of doing it this way vs. without WASM and workers? Until we see numbers, this smells like a recruiting play.
- texodus 9y agoWe use Web Workers principally to separate data CPU load from rendering load, as the datasets we deal with update very frequently and are quite large, and e.g. Highcharts can take on teh order of 100ms+ on detailed charts. We have some light benchmarks we use for regression testing, but definitely need more work in this area [benchmarks](https://jpmorganchase.github.io/perspective/examples/benchmark.html https://jpmorganchase.github.io/perspective/examples/benchma...)
- lucozade 9y agoFor ticking data being pushed into bespoke pivots, running this on the backend and, say, pushing renders to the frontend isn't generally better. You're not likely to be sharing much of the grunt. Also, latency will be an issue so you need to be pretty careful with GC runtimes that aren't optimised for latency. > What is the performance of doing it this way vs. without WASM and workers? @dman says in another thread that the original was in Python and this was 10x faster. If this met their performance goals, it does the job. It may be possible that another way is better, for some definition of better, but who cares if it works as required? > Until we see numbers, this smells like a recruiting play. It's presumably OK not to use it if you don't want to. And what if they think this may help recruiting? Seems like a reasonable trade to me.
- devscreen 9y agoThis is seriously cool. Unfortunately I have no use-case for it in the product I currently work on, but still fascinating to flick through the source code and play with it.
- mjansen 9y agoreally cool. would love to see a Rust version.
- shamaku 9y agobuild it
- polskibus 9y agoDoes the C++ code use vectorization? If not, is it planned?
- wsgeek 9y agoI followed the directions to the letter, and everything installed correctly on MacOS. When I tried the same on Linux, though, I got tons of the following errors on the build step (keep in mind I do have boost development libraries installed in /usr/include): (Sorry about the formatting.... I tried <code>). Anyway, TL/DR it cannot find boost. $ ./node_modules/.bin/lerna run start --stream lerna info version 2.8.0 @jpmorganchase/perspective: > @jpmorganchase/perspective@0.1.1 start /home/dj/usr/src/perspective-clone/packages/perspective @jpmorganchase/perspective: > npm run compile && (npm run perspective & npm run compile_test & npm run compile_node & wait) @jpmorganchase/perspective: > @jpmorganchase/perspective@0.1.1 compile /home/dj/usr/src/perspective-clone/packages/perspective @jpmorganchase/perspective: > mkdir -p build build/wasm_async build/wasm_sync build/asmjs && (cd build/; emcmake cmake ../; emmake make -j8; cd ..) @jpmorganchase/perspective: -- Configuring done @jpmorganchase/perspective: -- Generating done @jpmorganchase/perspective: -- Build files have been written to: /home/dj/usr/src/perspective-clone/packages/perspective/build @jpmorganchase/perspective: Scanning dependencies of target psp @jpmorganchase/perspective: [ 1%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_win.cpp.o @jpmorganchase/perspective: [ 2%] Building CXX object CMakeFiles/psp.dir/src/cpp/arg_sort.cpp.o @jpmorganchase/perspective: [ 4%] Building CXX object CMakeFiles/psp.dir/src/cpp/calc_agg_dtype.cpp.o @jpmorganchase/perspective: [ 5%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggspec.cpp.o @jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggregate.cpp.o @jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/base.cpp.o @jpmorganchase/perspective: [ 9%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_linux.cpp.o @jpmorganchase/perspective: [ 10%] Building CXX object CMakeFiles/psp.dir/src/cpp/build_filter.cpp.o @jpmorganchase/perspective: [ 12%] Building CXX object CMakeFiles/psp.dir/src/cpp/column.cpp.o @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/cpp/calc_agg_dtype.cpp:11: @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/calc_agg_dtype.h:12: @jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/schema.h:13: @jpmorganchase/perspective: /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/base.h:29:10: fatal error: 'boost/unordered_map.hpp' file not found @jpmorganchase/perspective: #include <boost/unordered_map.hpp>\n ^~~~~~~~~~~~~~~~~~~~~~~~~ </code>