6 ms·
I wrote the C++ part of this codebase, would be happy to answer any questions about it here.
by dman 9y ago
I wrote the C++ part of this codebase, would be happy to answer any questions about it here.
- timkpaine 9y ago[]
- dman 9y agoOfftopic :)
- deleted 9y ago[deleted]
- icefox 9y agoWhat was the best/worst part about getting it to work in WebAssembly?
- dman 9y agoBest part was the validation of a design decision of having minimal dependencies. It made the web assembly port almost trivial. Worst part (which has improved dramatically) was the rough edges in the web assembly toolchains early on.
- nickthemagicman 9y agoThanks for taking questions. For a back end developer that's watching the front end landscape change from afar.. Do you feel like web assembly could potentially replace JS in the future with, say, my favorite back end language of choice?
- jameslk 9y agoNot the op, but there's more to frontend development than JS. You need to account for the API of browsers, the DOM, CSS and the myraid of frameworks and libraries built for the specific UI requirements of frontend development (e.g. React). So even if your favorite language of choice becomes compilable to WebAssembly, you'll still be learning an entirely different way of development that will be completely unfamiliar regardless. The only thing I see languages compiled to WebAssembly taking over is development of highly resource intensive applications, algorithms or libraries. Or if there's a quantum leap with some specific language + framework that eliminates a lot of work with current frontend development. Otherwise, there will be too much fragmentation and you'll see the regression to the mean effect as has been the case with predecessors such as CoffeeScript, Elm, PureScript, ReasonML, and every other compile-to-JS language du jour.
- ChicagoDave 9y agoA paradigm shift could happen and what's cool is that it could be entirely something new. This reminds me of the early days of the WWW when the first programming paradigm was Perl/CGI, then Cold Fusion, then Legacy ASP, and then ActiveX, and so on. The possibility of some company creating, from scratch, a new web development paradigm, given everything we know today, is exciting.
- jameslk 9y agoSure, and based on historical precedent, that's almost guaranteed to happen. But I doubt it will be whatever-my-favorite-backend-language-of-choice-is and more likely some-domain-specific-language-and-new-framework. In other words, we'll all be relearning everything again 5 years from now.
- goatlover 9y agoJava applets were supposed to provide that in the 90s. Netscape was going to actually embed Java in the browser as an alternative to JavaScript, but there wasn't enough time, so they just shipped with JS, and Java was provided via a plugin.
- dman 9y agoAs a fellow backend developer I certainly hope so. I think web assembly gets us very close. Only two things I missed were direct canvas access from C++ without going through a JS shim and access to some kind of blob storage api in the browser.
- zimablue 9y agoI see it's possible to embed it in jupyterlab, it seems to do a lot: * grid format * graph format * actual pivoting of data Is that right? It's using webassembly so in effect to do the pivot you must have created functions that exist in pandas? Has anyone ported pandas to webassembly? Is the grid editable? What are you using to create your output, html or canvas or something? I realise I could look this stuff up but since you asked..
- texodus 9y agoI believe you'd need to port Python to WebAssembly to make much use of pandas - unaware of any projects attempting the former. The grid is just a plugin for the excellent [Hypergrid](https://github.com/openfin/fin-hypergrid https://github.com/openfin/fin-hypergrid), which is editable, but you'd need to manually push those edits back to the engine for now.
- zimablue 9y agowhich answers my other question, so it's canvas?
- texodus 9y agoThe frontend is actually pure JS/HTML - the WebAssembly part is just the data engine, and runs in a WebWorker for CPU isolation. Hypergrid is a canvas-based renderer, yes.
- Laaas 9y agoNuitka can compile Python to LLVM IR, which can be compiled into WASM, I think.
- fulafel 9y agoAs historical background: Python was one of the early emscripten asm.js experiments: https://github.com/kripken/emscripten/tree/master/tests/python https://github.com/kripken/emscripten/tree/master/tests/pyth...
- 9y ago
- pavlov 9y agoIs Emscripten the only game in town for WebAssembly C/C++ toolchains, or are there other contenders?
- dman 9y agoAs far as I know yes for C/C++ this was the only viable toolchain that is publicly available.
- Klathmon 9y agoEven outside of C/C++, Emscripten seems to be required in some fashion for just about every "compile X to webasm". Which i'm not that happy about as it's quite heavy and takes a long time to compile, which makes it harder for newcomers to try it out because it requires a fairly large time investment to even compile hello world.
- bbatha 9y agoRust’s wasm32-unknown-unknown target does not use emscripten at all. The tool cargo-web will handle the js module shims for you.
- Game_Ender 9y agocargo-web uses emscripten under the hood.
- a_humean 9y agoNo, he is right. cargo-web supports three backends, of which only two include emscripten and the third using a native rust backend. https://github.com/koute/cargo-web#features https://github.com/koute/cargo-web#features Rust can be compiled to wasm without emscripten using wasm32-unknown-unknown.
- bringtheaction 9y agoDoes that mean I can install cargo-web without emscripten or do I still need it even though I am not going to use it just because the two other targets use it? In other words, is emscripten an optional or a hard dependency?
- jarpineh 9y agoBriefly in the animation I see there's hierachical (or multi-dimensional) data used. Could you tell me about what sort of data (statistics, monitoring) this supports? Also, what methods are available for getting data streamed [into Perspective]? I'm skimming through the docs, but am not quite getting it... [edited for clarity]
- texodus 9y agoRight now, you can call a load() or update() method on an engine or widget to pass JSON, CSV or Apache Arrow data (as an ArrayBuffer) to the engine. Apologize for the documentation clarity - this is our first release and very much under active construction!
- jarpineh 9y agoOk, thank you for this. I'll have to see some time about that Arrow support. JSON and CSV is already plenty enough. I didn't have time for a proper look yet, so no worries. If at all possible post the code and data (or something like it) that is in those animations you have. That would perhaps allow users to dive in by tinkering here and there.
- stupidcar 9y agoDid you experiment with doing it JS-only prior to the C++ version? If so, what kind of performance increase did you see with WebAssembly?
- dman 9y agoNo, when I first prototyped it I had written a Python version initially. From memory the first C++ build was ~80x faster than the Python prototype.
- carapace 9y agoI'm curious if you did any profiling? If the intent was always to rewrite the whole prototype in C++ then it kinda wouldn't matter, eh?
- texodus 9y agoWe do some benchmarking just for catching performance regressions and validating assumptions against the non-WebAssembly version, but definitely need more in this regard. [benchmark for wasm build](https://jpmorganchase.github.io/perspective/examples/benchmark.html https://jpmorganchase.github.io/perspective/examples/benchma...)
- goatlover 9y ago80x faster than using Pandas? What if you just used the Numpy values array?
- dman 9y agoYou will get me into trouble! I said 80x faster than the Python prototype. I did benchmark this vs Pandas over the years and depending on the usecase the two codebases trade blows with each other for static data. However for streaming datasets Perspective has a large advantage over Pandas since pandas does a full group by on every update, while perspective does its work incrementally using deltas.
- 9y ago
- polskibus 9y agoDoes it support: virtualization (showing only a view of full dataset that is updated from server on scroll) Pivoting (multiple levels/hierarchies on rows and on columns) Apart from features what part did you consider was the biggest hurdle? What browsers did you target?
- texodus 9y agoIt supports virtualization, and it is utilized by both plugins (Hypergrid and Highcharts). N levels of pivoting on both axes. Should work on all browsers - if not, please open an issue!
- polskibus 9y agoIs there a way to drill down a node dynamically, by clicking on row header during pivoting, and then see its children?
- deleted 9y ago[deleted]
- texodus 9y agoNot through the UI yet, only in the engine API - but the pivoting itself is quite fast, so should still be suitable for drilling down "on the fly" so to stpeak
- polskibus 9y agoBy virtualization I meant also not loading entire dataset to memory, but being able to work only on a constrained view of the data. For example, database has 1TB dataset, I only view 100 MB, further columns/sections are loaded when I scroll.
- texodus 9y agoIt is virtual in the sense that it does not realize the entire dataset in the JS side of the WebAssembly bridge - this is a performance optimization but entirely in-memory. While you can run the engine in node.js and use it to efficiently stream updates to a symmetric engine in the browser, we do not currently implement server-virtualized views. Though, we have quite alot of experience doing this in the past - and the design of Perspective is very much a reaction to what we learned, at lesat in regards to the typical financial dataset which is much smaller than 1TB.
- LukeB42 9y agoI see it's possible to convert to Numpy arrays. Is `perspective` also a Python library?
- dman 9y agoYes for much of its life perspective has been a python library. It offers a streaming dataframe abstraction in Python. I believe the Python bindings have not been open sourced yet.
- polskibus 9y agoCan you say how old is this codebase and what was its original business case?
- dman 9y agoCodebase started in 2013. Original business case was to allow business users to create filtered/aggregated views on top of streaming data.
- polskibus 9y agoWhat's on the roadmap?
- fitzoh 9y agoJust stalked you on linkedin with a similar question, but I was wondering if you were involved with the actual process of getting the library open sourced, and how that works at a large security focused company like JPMorgan Chase. (former Chase employee)