5 ms·

We are building a data vis tool (https://github.com/tracyhenry/kyrix https://github.com/tracyhenry/kyrix) to solve this exact problem many web vis tools have.

We are building a data vis tool (https://github.com/tracyhenry/kyrix https://github.com/tracyhenry/kyrix) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/ http://wildfire.kyrixdemo.live/ Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful: Falcon: https://github.com/uwdata/falcon https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/ https://nanocubes.net/ Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.

domoritz 6y ago
Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as https://github.com/omnisci/jupyterlab-omnisci https://github.com/omnisci/jupyterlab-omnisci.
- carls 6y ago
  I've used Vega-Lite and d3 before and have appreciated both. Vega-Lite seems to be great for rapid prototyping and d3 for really refined, intricate and more complex plots. I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries. Do you mind clarifying?
breck 6y ago
I saw you mention Falcon and NanoCubes so had to check our kyrix. I think this is a really important space. I used to assume Palantir had solved this but now that they're public it seems to be more of a smoke and mirrors startup. 3 questions: 1. Why isn't there a demo I can just click and visit? 2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade) 3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).
- tracyhenry 6y ago
  Thanks for the feedback! 1. In the original comment there is a link to a demo created by Kyrix: http://wildfire.kyrixdemo.live/ http://wildfire.kyrixdemo.live/ Are you looking for a different type of demo? 2. Honest answer: we are very understaffed academics who also need to write papers and theses. We want to convert to TS, that is just one item on the wish list. 3. We want everyone to be able to spin up a Kyrix app on their laptop using 3 CLI commands (you can try, instructions are in the README). With a backend and a database comes the cost of complex installation. Docker helps make everyone's dev environment consistent so it's easier to troubleshoot.
  - breck 6y ago
    1. Ah! Sorry I missed that and went to the GitHub. Might be helpful to also add that link to the "website" field of the GitHub? 2. I get it. I never had the "pleasure" of being an academic, but got the chance to be a software engineer working alongside grad students for a couple of years and it felt like I imagine what taking a tour of a sweatshop factory floor would be like. (a huge exaggeration, but I saw a lot of low pay and lots of time on pdfs and not enough on code) 3. Yeah, I get it, but maybe allowing everyone to run it on their own VM would be better (and providing a one click "click here to get your own droplet on digital ocean running kyrix" sort of thing). I've found that Docker suffers from the XKCD problem (10 different environments to support—what if there was just 1....11 different environments to support!). I've almost never had a pleasant experience using Docker (and anything that requires downloading a 1GB+ image I don't consider pleasant). But again, I'm probably just being a grump and the other grads I worked with all seems to like it. Another idea is could use do this with SQLite? Have a more slimmed down version that didn't require Docker + those dependencies Thanks for the answers! Very cool stuff and it's a very interesting problem.
lmeyerov 6y ago
This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implementation to help us bridge GPU visual analytics components in the browser with real-time GPU clusters in the data center, and our backend is DSLs like dataframes to make it easier to do that. An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing. Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off. It's exciting to see it all come together -- imo, still early days for what's possible!
- tracyhenry 6y ago
  Thanks for the thoughts. Graphistry looks really slick and interesting. On thing I'm curious about: how do you compare Graphistry with OmniSci and similar GPU-based solutions? You seem to focus on a niche - graph visualizations. That's one differentiator I can tell. Do other solutions support graphs too? If so, how do you compare Graphistry with them? Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there. You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.

domoritz 6y ago

Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as https://github.com/omnisci/jupyterlab-omnisci https://github.com/omnisci/jupyterlab-omnisci.

carls 6y ago
I've used Vega-Lite and d3 before and have appreciated both. Vega-Lite seems to be great for rapid prototyping and d3 for really refined, intricate and more complex plots. I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries. Do you mind clarifying?

breck 6y ago

I saw you mention Falcon and NanoCubes so had to check our kyrix. I think this is a really important space. I used to assume Palantir had solved this but now that they're public it seems to be more of a smoke and mirrors startup. 3 questions: 1. Why isn't there a demo I can just click and visit? 2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade) 3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).

tracyhenry 6y ago
Thanks for the feedback! 1. In the original comment there is a link to a demo created by Kyrix: http://wildfire.kyrixdemo.live/ http://wildfire.kyrixdemo.live/ Are you looking for a different type of demo? 2. Honest answer: we are very understaffed academics who also need to write papers and theses. We want to convert to TS, that is just one item on the wish list. 3. We want everyone to be able to spin up a Kyrix app on their laptop using 3 CLI commands (you can try, instructions are in the README). With a backend and a database comes the cost of complex installation. Docker helps make everyone's dev environment consistent so it's easier to troubleshoot.
- breck 6y ago
  1. Ah! Sorry I missed that and went to the GitHub. Might be helpful to also add that link to the "website" field of the GitHub? 2. I get it. I never had the "pleasure" of being an academic, but got the chance to be a software engineer working alongside grad students for a couple of years and it felt like I imagine what taking a tour of a sweatshop factory floor would be like. (a huge exaggeration, but I saw a lot of low pay and lots of time on pdfs and not enough on code) 3. Yeah, I get it, but maybe allowing everyone to run it on their own VM would be better (and providing a one click "click here to get your own droplet on digital ocean running kyrix" sort of thing). I've found that Docker suffers from the XKCD problem (10 different environments to support—what if there was just 1....11 different environments to support!). I've almost never had a pleasant experience using Docker (and anything that requires downloading a 1GB+ image I don't consider pleasant). But again, I'm probably just being a grump and the other grads I worked with all seems to like it. Another idea is could use do this with SQLite? Have a more slimmed down version that didn't require Docker + those dependencies Thanks for the answers! Very cool stuff and it's a very interesting problem.

lmeyerov 6y ago

This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implementation to help us bridge GPU visual analytics components in the browser with real-time GPU clusters in the data center, and our backend is DSLs like dataframes to make it easier to do that. An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing. Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off. It's exciting to see it all come together -- imo, still early days for what's possible!

tracyhenry 6y ago
Thanks for the thoughts. Graphistry looks really slick and interesting. On thing I'm curious about: how do you compare Graphistry with OmniSci and similar GPU-based solutions? You seem to focus on a niche - graph visualizations. That's one differentiator I can tell. Do other solutions support graphs too? If so, how do you compare Graphistry with them? Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there. You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.