3 ms·
Somewhat related, who has played with performant JS dataframe libraries? I mostly want to serialize from pandas/polars to JS in a fast way, with some very minor
by paddy_m 3y ago
Somewhat related, who has played with performant JS dataframe libraries? I mostly want to serialize from pandas/polars to JS in a fast way, with some very minor selection and indexing operations JS side. Ideally I could find a library that does dataframe like operations from reading base64 encoded Buffers/Arrays into their JS objects, then doing the right things.
I'm investigating
https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/
https://github.com/vega/falcon https://github.com/vega/falcon
https://github.com/pola-rs/nodejs-polars https://github.com/pola-rs/nodejs-polars
https://github.com/uwdata/arquero https://github.com/uwdata/arquero
- paddy_m 3y agoI added a github issue to my project where this matters literally this morning. I'd love to talk to someone dealing with the same stuff. I have baked dataframe -> JS serialization utils so many times over the past decade. I never want to do much in JS, just simple selection and filtering. But you end up needing both sides of the serialization to do it right.
- kylebarron 3y agonodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box. As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)