3 ms·
Extra fun: We find most enterprise/gov graph analytics work only requires 1-2 attributes to go along with the graph index, and those attributes often are alread
by lmeyerov 3y ago
Extra fun: We find most enterprise/gov graph analytics work only requires 1-2 attributes to go along with the graph index, and those attributes often are already numeric (time, $, ...) or can be dictionary-encoded as discussed here (categorical, ID, ...). The result is that even 'tough' billion scale graphs are fine on 1 gpu.
Ex: Most of our users are happy to do 100K-100M edges quickly on a single node, imagine a quick parquet read, so goes surprisingly far. For $0.5/hr, you can get 16GB of GPU RAM on aws, stuff in 100M+ edges comfortably, and have room to compute all sorts of things. With a more typical one, 1B is easy.
Early, but that's been the basic thinking into our new GFQL system: slice into the columns you want, and then do all the in-GPU traversals you want. In our V1, we keep things dataframe-native, including the in-GPU data representation, and we are now working on adding the first extensions to support on-the-fly switching to more graph-native indexing as needed.
Ex: https://github.com/graphistry/pygraphistry/blob/master/demos/gfql/benchmark_hops_cpu_gpu.ipynb https://github.com/graphistry/pygraphistry/blob/master/demos...
- gpderetta 3y agoHence the typical observation that a lot of Big Data jobs actually run comfortably on a developer laptop.
- fulafel 3y agoObligatory reference: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... wherein various big data benchmarks (eg PageRank) get this treatment.