4 ms·

Zoomable, animated scatterplots in the browser that scales over a billion points

148 points by samwillis 4y ago

paulgb 4y ago
Nomic are also the ones behind gpt4all. Such a powerhouse team. https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all
mLuby 4y ago
Nice. At least 7 years ago I saw a similar demo: https://demo-taxis.heavy.ai/ https://demo-taxis.heavy.ai/ for an example. Is the difference that nomic's billion points are rendered entirely client-side?
- VHRanger 4y ago
  yes, it's using webGPU to render the whole thing
wirthjason 4y ago
Github issues, even low quality ones, are welcom here. This a refreshing line to read. I wonder if the missing “e” (welcom vs welcome) is intentional. It sounds like and easy (low quality?) PR.
- coding123 4y ago
  I do find it funny when you have this glaringly bad bug that effects everyone and the bug template wants you to report everything down to your DNA. Sometimes it is nice to just report: latest released version, here's my stack trace...
larsrc 4y ago
Is the name a reference to the democracy game from Scientific American back in the 80ies (or so)? Also, is there a good equivalent for node-edge graphs?
- VHRanger 4y ago
  Ideally, you'd embed the graph into 2 or 3d first, then visualize it as a scatterplot. Visualizing the edges at scale doesnt yield nice results in general. It's simply too noisy. The way to do it is to reduce the graph to some 300d or 500d embeddings, then use TSNE/UMAP/PACMAP to reduce that to 3d. Then visualize. My prefered way is to use some first order embedding method like GGVec in this library [1] (disclaimer I wrote it). Node2Vec and ProNE don't yield great embeddings for visualization (the first is too filamented, the second too close to the unit ball). Another great library to do this work is GRAPE [2]. Try first-order embedding methods, or short walks on second order methods to avoid the embeddings being too filamented by long random walk sampling. [1] https://github.com/VHRanger/nodevectors https://github.com/VHRanger/nodevectors [2] https://github.com/AnacletoLAB/grape/ https://github.com/AnacletoLAB/grape/
  - larsrc 4y ago
    Thanks!
- lordwarnut 4y ago
  I don't suppose you have a link to that game? It sounds interesting and I'd love to learn more.
  - larsrc 4y ago
    https://wiki.c2.com/?NomicGame https://wiki.c2.com/?NomicGame has a bunch of information around it, but the original article is behind a paywall (https://www.jstor.org/stable/24966607 https://www.jstor.org/stable/24966607).
tudorw 4y ago
nice, https://github.com/laurolins/nanocube https://github.com/laurolins/nanocube is/was another interesting approach to the many points issue, though it's seen no updates since 21 afaik
kristjansson 4y ago
Great tool to add to the visualization toolbox! It does seem to lack the visual fidelity at large scales that it resovles at finer ones. I wonder if it could extend to the render + raster approach of Datashaper, while still running everything in browser?
pphysch 4y ago
It seems I don't have the necessary graphics extensions available on my mobile device (mainstream device using mainstream browser). Is this supposed to work on mobile?
coppsilgold 4y ago
Reminds me of a more ambitious project where you can view and zoom 1000s of images: <https://gitlab.com/galapix/galapix https://gitlab.com/galapix/galapix> There is also a simpler Rust imitation of that project: <https://github.com/google/pix-image-viewer https://github.com/google/pix-image-viewer> Demos: <https://www.youtube.com/watch?v=OJz0dz3oG-Q https://www.youtube.com/watch?v=OJz0dz3oG-Q> <https://www.youtube.com/watch?v=9-Mg6MkE-tg https://www.youtube.com/watch?v=9-Mg6MkE-tg>
doodlebugging 4y ago
I'm pretty sure I have seen this or something very similar somewhere before but I can't remember where. It is pretty impressive. The background map they use though for this is definitely wishful thinking in my area. I clicked on points that looked to be near me and discovered that the census data used for those points has people living on a couple of roads that haven't been built and won't be built for at least two years if things go like they have on the other road construction projects within a few miles of me. Today the roads only exist on planning documents posted by the city.
- oplav 4y ago
  Highcharts has had a "boost" module for a long time (at least back since 2015/2016) that switched the rendering over to WebGL for large data sets, but I don't think it ever got up to 1B points. https://www.highcharts.com/docs/advanced-chart-features/boost-module https://www.highcharts.com/docs/advanced-chart-features/boos...

148 points by samwillis 4y ago

Nomic are also the ones behind gpt4all. Such a powerhouse team. https://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all

mLuby 4y ago

Nice. At least 7 years ago I saw a similar demo: https://demo-taxis.heavy.ai/ https://demo-taxis.heavy.ai/ for an example. Is the difference that nomic's billion points are rendered entirely client-side?

VHRanger 4y ago
yes, it's using webGPU to render the whole thing

wirthjason 4y ago

Github issues, even low quality ones, are welcom here. This a refreshing line to read. I wonder if the missing “e” (welcom vs welcome) is intentional. It sounds like and easy (low quality?) PR.

coding123 4y ago
I do find it funny when you have this glaringly bad bug that effects everyone and the bug template wants you to report everything down to your DNA. Sometimes it is nice to just report: latest released version, here's my stack trace...

larsrc 4y ago

Is the name a reference to the democracy game from Scientific American back in the 80ies (or so)? Also, is there a good equivalent for node-edge graphs?

VHRanger 4y ago
Ideally, you'd embed the graph into 2 or 3d first, then visualize it as a scatterplot. Visualizing the edges at scale doesnt yield nice results in general. It's simply too noisy. The way to do it is to reduce the graph to some 300d or 500d embeddings, then use TSNE/UMAP/PACMAP to reduce that to 3d. Then visualize. My prefered way is to use some first order embedding method like GGVec in this library [1] (disclaimer I wrote it). Node2Vec and ProNE don't yield great embeddings for visualization (the first is too filamented, the second too close to the unit ball). Another great library to do this work is GRAPE [2]. Try first-order embedding methods, or short walks on second order methods to avoid the embeddings being too filamented by long random walk sampling. [1] https://github.com/VHRanger/nodevectors https://github.com/VHRanger/nodevectors [2] https://github.com/AnacletoLAB/grape/ https://github.com/AnacletoLAB/grape/
- larsrc 4y ago
  Thanks!
lordwarnut 4y ago
I don't suppose you have a link to that game? It sounds interesting and I'd love to learn more.
- larsrc 4y ago
  https://wiki.c2.com/?NomicGame https://wiki.c2.com/?NomicGame has a bunch of information around it, but the original article is behind a paywall (https://www.jstor.org/stable/24966607 https://www.jstor.org/stable/24966607).

tudorw 4y ago

nice, https://github.com/laurolins/nanocube https://github.com/laurolins/nanocube is/was another interesting approach to the many points issue, though it's seen no updates since 21 afaik

kristjansson 4y ago

Great tool to add to the visualization toolbox! It does seem to lack the visual fidelity at large scales that it resovles at finer ones. I wonder if it could extend to the render + raster approach of Datashaper, while still running everything in browser?

pphysch 4y ago

It seems I don't have the necessary graphics extensions available on my mobile device (mainstream device using mainstream browser). Is this supposed to work on mobile?

coppsilgold 4y ago

Reminds me of a more ambitious project where you can view and zoom 1000s of images: <https://gitlab.com/galapix/galapix https://gitlab.com/galapix/galapix> There is also a simpler Rust imitation of that project: <https://github.com/google/pix-image-viewer https://github.com/google/pix-image-viewer> Demos: <https://www.youtube.com/watch?v=OJz0dz3oG-Q https://www.youtube.com/watch?v=OJz0dz3oG-Q> <https://www.youtube.com/watch?v=9-Mg6MkE-tg https://www.youtube.com/watch?v=9-Mg6MkE-tg>

doodlebugging 4y ago

I'm pretty sure I have seen this or something very similar somewhere before but I can't remember where. It is pretty impressive. The background map they use though for this is definitely wishful thinking in my area. I clicked on points that looked to be near me and discovered that the census data used for those points has people living on a couple of roads that haven't been built and won't be built for at least two years if things go like they have on the other road construction projects within a few miles of me. Today the roads only exist on planning documents posted by the city.

oplav 4y ago
Highcharts has had a "boost" module for a long time (at least back since 2015/2016) that switched the rendering over to WebGL for large data sets, but I don't think it ever got up to 1B points. https://www.highcharts.com/docs/advanced-chart-features/boost-module https://www.highcharts.com/docs/advanced-chart-features/boos...