4 ms·
Thanks for kind words all :) Scale - backend: As we're rapids.ai-native (helped start early days of both Apache Arrow + Nvidia RAPIDS.ai), we work with custome
by lmeyerov 4y ago
Thanks for kind words all :)
Scale - backend: As we're rapids.ai-native (helped start early days of both Apache Arrow + Nvidia RAPIDS.ai), we work with customers doing billion-level nodes/edges in interactive time on GPU servers. Mostly for fast ingest, ETL, + graph neural nets / manifold learning, and we're slowly pushing that into the visual stack.
Scale - frontend: We normally recommend reducing down to about ~2M edges or less. For sensible visual experiences, add in auto algorithms that cut to more like 500K. We've planned a way we think we can do another 100X, just not (yet) an engineering priority. Fun fact: your browser's JS VM is limited to ~1GB of RAM, so we're already at that limit in practice.
RE:Scaling graph visuals as an engineering practice, it looks like memgraph is starting where neo4j reached a few years ago, and makes sense. That approach doesn't really work well for the use cases memgraph advertises for, because as soon as a bunch of user/customer/IT/etc events happen and get visualized, the browser crashes. Optimization approaches like wasm and workers are clever -- a v0 prototype of graphistry did that! -- but we found is too unreliable to be the path for good performance across users of most operational teams ("Works on my machine" syndrome). Do it, but that shouldn't be the main source of 100X performance, just a 2X boost. We end up connecting GPUs in the browser to GPUs in the datacenter not just for 100X'ing this kind of stuff, but for a predictable performance way that limits how often your user's browser crash on real datasets.
Also maybe not obvious, this article focuses on interactive rendering, but a lot of the challenge after they figure out how to solve it is interactive analytics too. Most layout algorithms have non-linear complexity, so O(500K) is actually a challenge. A lot of our GPU offloading work nowadays isn't just rendering but layout, ETL, ML/AI clustering, etc. This ends up overwhelming the browser (why we do distributed GPU), and OLTP graph DB's aren't good at that either -- Neo4j basically had to write a V2 DB-in-a-DB to make their Graph Data Science module perform.
And no worries: There's no competition because Graphistry isn't a graph database :) Most of our users will do something like databricks dashboard / jupyter notebook / powerbi / etc query <> graphistry visual. I bet pairing a great streaming db like memgraph with Graphistry would combine respective engineering strengths quite well!