6 ms·

Show HN: Rendering One Million Datapoints with D3 and WebGL

200 points by ColinEberhardt 6y ago

zylepe 6y ago
Very cool! I’ve done visualizations on similar size datasets by mapping tsne output to fake latitude/longitude coordinates and rendering it with mapbox gl. It feels like a hack, but then you get all of the panning/zooming interaction, styling, and rendering labels with collision avoidance for free. For smaller datasets I pass in geojson directly, but for larger ones I precompute vector tiles so you can lazy-load more of the data when you zoom in. There are a lot of things built for mapping that are relevant for other domains, I wish that there was a common base that could be extracted that mapping libraries adapt to geographic data but data visualizations could adapt to non-geographic data. I’ll have to play around with your technique more, encouraging to see you get good performance eagerly loading a million data points!
- andybak 6y ago
  Deck.GL has some similar examples. I think the reason this makes sense is that geo data viz had to pioneer huge datasets - which are less common in other fields.
- 7373737373 6y ago
  I can recommend LargeVis[0][1] which I used to layout and Polymaps[2] to visualize tiles of the https://wikiscape.org https://wikiscape.org article point cloud. [0] https://github.com/lferry007/LargeVis https://github.com/lferry007/LargeVis [1] https://github.com/elbamos/largeVis https://github.com/elbamos/largeVis [2] http://polymaps.org/ http://polymaps.org/
  - 7373737373 6y ago
    Scrap polymaps, use leaflet instead: https://leafletjs.com https://leafletjs.com
paulgb 6y ago
Scott Logic is probably the highest-quality company technology blog I've encountered in a while. I keep returning to it for WebAssembly and WebGL topics and it is consistently high quality. I'm not even sure what Scott Logic does, but they sure know their stuff :) Kudos, Colin and team.
- onion2k 6y ago
  Their head office is round the corner from my office in Newcastle, UK, and I occasionally bump in to their devs in the local pub and at the JS meetup. They're very good at what they do and lovely people too.
- luhem7 6y ago
  https://www.scottlogic.com/what-we-do/ https://www.scottlogic.com/what-we-do/ ;)
- ColinEberhardt 6y ago
  Thanks Paul, really appreciate that :-) Just FYI, we're a medium sized software consultancy based in the UK. We do quite a lot of work in financial services, which is why you'll find quite a few posts on performance and cutting-edge web technology. I started the company blog 12 years ago and now we have ~50 authors creating some amazing content. Unfortunately I don't have the time to write quite as much as I used to!
gdubs 6y ago
The scatterplot of the library data is fascinating — looks like an organism.
1wheel 6y ago
Nice write up Colin! fwiw quadtrees don't make much sense when you're showing a tooltip for ~1 million points: https://roadtolarissa.com/scan-sorted/ https://roadtolarissa.com/scan-sorted/
- ColinEberhardt 6y ago
  That's really interesting, thanks for sharing those results. So easy to go down the route of premature optimisation!

Very cool! I’ve done visualizations on similar size datasets by mapping tsne output to fake latitude/longitude coordinates and rendering it with mapbox gl. It feels like a hack, but then you get all of the panning/zooming interaction, styling, and rendering labels with collision avoidance for free. For smaller datasets I pass in geojson directly, but for larger ones I precompute vector tiles so you can lazy-load more of the data when you zoom in. There are a lot of things built for mapping that are relevant for other domains, I wish that there was a common base that could be extracted that mapping libraries adapt to geographic data but data visualizations could adapt to non-geographic data. I’ll have to play around with your technique more, encouraging to see you get good performance eagerly loading a million data points!

andybak 6y ago
Deck.GL has some similar examples. I think the reason this makes sense is that geo data viz had to pioneer huge datasets - which are less common in other fields.
7373737373 6y ago
I can recommend LargeVis[0][1] which I used to layout and Polymaps[2] to visualize tiles of the https://wikiscape.org https://wikiscape.org article point cloud. [0] https://github.com/lferry007/LargeVis https://github.com/lferry007/LargeVis [1] https://github.com/elbamos/largeVis https://github.com/elbamos/largeVis [2] http://polymaps.org/ http://polymaps.org/
- 7373737373 6y ago
  Scrap polymaps, use leaflet instead: https://leafletjs.com https://leafletjs.com

paulgb 6y ago

Scott Logic is probably the highest-quality company technology blog I've encountered in a while. I keep returning to it for WebAssembly and WebGL topics and it is consistently high quality. I'm not even sure what Scott Logic does, but they sure know their stuff :) Kudos, Colin and team.

onion2k 6y ago
Their head office is round the corner from my office in Newcastle, UK, and I occasionally bump in to their devs in the local pub and at the JS meetup. They're very good at what they do and lovely people too.
luhem7 6y ago
https://www.scottlogic.com/what-we-do/ https://www.scottlogic.com/what-we-do/ ;)
ColinEberhardt 6y ago
Thanks Paul, really appreciate that :-) Just FYI, we're a medium sized software consultancy based in the UK. We do quite a lot of work in financial services, which is why you'll find quite a few posts on performance and cutting-edge web technology. I started the company blog 12 years ago and now we have ~50 authors creating some amazing content. Unfortunately I don't have the time to write quite as much as I used to!

gdubs 6y ago

The scatterplot of the library data is fascinating — looks like an organism.

1wheel 6y ago

Nice write up Colin! fwiw quadtrees don't make much sense when you're showing a tooltip for ~1 million points: https://roadtolarissa.com/scan-sorted/ https://roadtolarissa.com/scan-sorted/

ColinEberhardt 6y ago
That's really interesting, thanks for sharing those results. So easy to go down the route of premature optimisation!