3 ms·

This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implemen

This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implementation to help us bridge GPU visual analytics components in the browser with real-time GPU clusters in the data center, and our backend is DSLs like dataframes to make it easier to do that. An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing. Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off. It's exciting to see it all come together -- imo, still early days for what's possible!

tracyhenry 6y ago
Thanks for the thoughts. Graphistry looks really slick and interesting. On thing I'm curious about: how do you compare Graphistry with OmniSci and similar GPU-based solutions? You seem to focus on a niche - graph visualizations. That's one differentiator I can tell. Do other solutions support graphs too? If so, how do you compare Graphistry with them? Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there. You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.
- lmeyerov 6y ago
  Fundamentally, omnisci locks you into: - omnisci SQL. in contrast, rapids.ai opens layers below (cudf arrow, dask, ...) that enable cooperating solns on top (blazingsql, cugraph, custreams, prefect, ...) that are faster + easier for their domains, w fallback to general dataframes/sql. -- omnisci is governance by a VC co, while rapids.ai is by nvidia (who wants to sell hw, not sw) and more OSS partners Omnisci did good engineering, so it does have strengths. ex: its geospatial visual analytics means it's a good esri alternative consideration, as it is more polished than manually stitching together cuspatial + blazingsql + leaflet etc. Likewise, commercially polished for hostile enterprise environments (procurement, ...). re:scale, see rapids tpcx-bb numbers ('big data'), I think on 10TB datasets. it shows scale + cost effectiveness wins vs others. less obvious, out-of-core so can do TBs even on one GPU, and full tpcx-bb needed the above versatility where sql is a kludge. re:graph vs table, if you do just points and no edges, the node table is just a regular table you can do regular tabular data analysis + viz in. ex: load in samples scored by some ml model (x/y plot w lots of data columns for each point), then connect nearest neighbors to make it into an interactive graph. we are doing more and more here in practice, it's fun :)

tracyhenry 6y ago

Thanks for the thoughts. Graphistry looks really slick and interesting. On thing I'm curious about: how do you compare Graphistry with OmniSci and similar GPU-based solutions? You seem to focus on a niche - graph visualizations. That's one differentiator I can tell. Do other solutions support graphs too? If so, how do you compare Graphistry with them? Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there. You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.

lmeyerov 6y ago
Fundamentally, omnisci locks you into: - omnisci SQL. in contrast, rapids.ai opens layers below (cudf arrow, dask, ...) that enable cooperating solns on top (blazingsql, cugraph, custreams, prefect, ...) that are faster + easier for their domains, w fallback to general dataframes/sql. -- omnisci is governance by a VC co, while rapids.ai is by nvidia (who wants to sell hw, not sw) and more OSS partners Omnisci did good engineering, so it does have strengths. ex: its geospatial visual analytics means it's a good esri alternative consideration, as it is more polished than manually stitching together cuspatial + blazingsql + leaflet etc. Likewise, commercially polished for hostile enterprise environments (procurement, ...). re:scale, see rapids tpcx-bb numbers ('big data'), I think on 10TB datasets. it shows scale + cost effectiveness wins vs others. less obvious, out-of-core so can do TBs even on one GPU, and full tpcx-bb needed the above versatility where sql is a kludge. re:graph vs table, if you do just points and no edges, the node table is just a regular table you can do regular tabular data analysis + viz in. ex: load in samples scored by some ml model (x/y plot w lots of data columns for each point), then connect nearest neighbors to make it into an interactive graph. we are doing more and more here in practice, it's fun :)