6 ms·

An issue of visualization on the web I don’t know how to solve is how to load large data.

chatmasta 6y ago
Observable supports direct DB connections [0], but these only work in private notebooks at the moment. At Splitgraph [1] we're building a "data delivery network" (DDN, like a CDN but for databases) that looks like a big Postgres database. It works really well with the Observable Postgres client in private notebooks -- you configure it like you would for any Postgres connection. For public notebooks, you can use our HTTP API for sending SQL queries directly to the DDN. Here's an example [2] using that SQL-over-HTTP API to plot some Covid data. [0] https://observablehq.com/@observablehq/connecting-to-databases https://observablehq.com/@observablehq/connecting-to-databas... [1] https://www.splitgraph.com https://www.splitgraph.com [2] https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-other-dataset-join-and-visualiza https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-...
- breck 6y ago
  I've been watching splitgraph for a while. Very interesting what you are doing!
  - chatmasta 6y ago
    Thanks! That means a lot :) Much more to come.
RobinL 6y ago
Very much depends on how large you're talking, but check out Apache Arrow: https://observablehq.com/@theneuralbit/introduction-to-apache-arrow https://observablehq.com/@theneuralbit/introduction-to-apach... https://observablehq.com/@uwdata/arquero-and-apache-arrow https://observablehq.com/@uwdata/arquero-and-apache-arrow
breck 6y ago
I've found 10MB (gzipped) to be a good rule of them. Anything bigger, query a backend. Storing bigger things in ephemeral tab memory can be painful if you refresh browser a lot.
catacombs 6y ago
That depends on the job. Is the data constantly updating?
tracyhenry 6y ago
We are building a data vis tool (https://github.com/tracyhenry/kyrix https://github.com/tracyhenry/kyrix) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/ http://wildfire.kyrixdemo.live/ Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful: Falcon: https://github.com/uwdata/falcon https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/ https://nanocubes.net/ Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.
- domoritz 6y ago
  Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as https://github.com/omnisci/jupyterlab-omnisci https://github.com/omnisci/jupyterlab-omnisci.
  - carls 6y ago
    I've used Vega-Lite and d3 before and have appreciated both. Vega-Lite seems to be great for rapid prototyping and d3 for really refined, intricate and more complex plots. I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries. Do you mind clarifying?
- breck 6y ago
  I saw you mention Falcon and NanoCubes so had to check our kyrix. I think this is a really important space. I used to assume Palantir had solved this but now that they're public it seems to be more of a smoke and mirrors startup. 3 questions: 1. Why isn't there a demo I can just click and visit? 2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade) 3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).

Observable supports direct DB connections [0], but these only work in private notebooks at the moment. At Splitgraph [1] we're building a "data delivery network" (DDN, like a CDN but for databases) that looks like a big Postgres database. It works really well with the Observable Postgres client in private notebooks -- you configure it like you would for any Postgres connection. For public notebooks, you can use our HTTP API for sending SQL queries directly to the DDN. Here's an example [2] using that SQL-over-HTTP API to plot some Covid data. [0] https://observablehq.com/@observablehq/connecting-to-databases https://observablehq.com/@observablehq/connecting-to-databas... [1] https://www.splitgraph.com https://www.splitgraph.com [2] https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-other-dataset-join-and-visualiza https://observablehq.com/@mildbyte/splitgraph-ddn-oxcovid19-...

breck 6y ago
I've been watching splitgraph for a while. Very interesting what you are doing!
- chatmasta 6y ago
  Thanks! That means a lot :) Much more to come.

RobinL 6y ago

Very much depends on how large you're talking, but check out Apache Arrow: https://observablehq.com/@theneuralbit/introduction-to-apache-arrow https://observablehq.com/@theneuralbit/introduction-to-apach... https://observablehq.com/@uwdata/arquero-and-apache-arrow https://observablehq.com/@uwdata/arquero-and-apache-arrow

breck 6y ago

I've found 10MB (gzipped) to be a good rule of them. Anything bigger, query a backend. Storing bigger things in ephemeral tab memory can be painful if you refresh browser a lot.

catacombs 6y ago

That depends on the job. Is the data constantly updating?

tracyhenry 6y ago

We are building a data vis tool (https://github.com/tracyhenry/kyrix https://github.com/tracyhenry/kyrix) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand. DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives that enable creation of pan/zoom like visualizations very easily (e.g. 10s of lines of JSON for a 15-zoom-level vis). You can check out an interactive demo here, which visualizes 1.88 million wildfires: http://wildfire.kyrixdemo.live/ http://wildfire.kyrixdemo.live/ Since big data vis is very use case dependent, I'd like to reference two other tools that might be useful: Falcon: https://github.com/uwdata/falcon https://github.com/uwdata/falcon NanoCubes: https://nanocubes.net/ https://nanocubes.net/ Disclaimer: I'm still in academia so I'm apparently biased towards academic works. But fortunately they are all open source.

domoritz 6y ago
Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as https://github.com/omnisci/jupyterlab-omnisci https://github.com/omnisci/jupyterlab-omnisci.
- carls 6y ago
  I've used Vega-Lite and d3 before and have appreciated both. Vega-Lite seems to be great for rapid prototyping and d3 for really refined, intricate and more complex plots. I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries. Do you mind clarifying?
breck 6y ago
I saw you mention Falcon and NanoCubes so had to check our kyrix. I think this is a really important space. I used to assume Palantir had solved this but now that they're public it seems to be more of a smoke and mirrors startup. 3 questions: 1. Why isn't there a demo I can just click and visit? 2. Why not TypeScript? (I think this would be a worthy short term and long term investment —for this project and beyond— to make the upgrade) 3. Why Docker? (This one I'm probably wrong about and am just a curmudgeonly old gray beard). Every time I see Docker I say "nope". I get it's use for massive clusters, but for running on one machine I dislike it (at that point I'd prefer just to run an image on a cloud VM).