3 ms·
We have been rewriting our stack for multi/many-gpu scale out via python GPU dataframes, but it's clear that smaller workloads and some others would be fine on
by lmeyerov 6y ago
We have been rewriting our stack for multi/many-gpu scale out via python GPU dataframes, but it's clear that smaller workloads and some others would be fine on CPU (and thus free up the GPUs for our other tenants), so having a good CPU impl is exciting, esp if they achieve API compatibility w pandas/dask as RAPIDS and others do. I've been eyeing vauex here (I think the other rust arrow project isn't DF's?), so good to have a contender!
I'd love to see a comparison to RAPIDS dataframes for the single GPU case (ex: 2 GB), single GPU bigger-than-memory (ex: 100 GB), and then the same for multi-GPU. We have started to measure as things like "200 GB/s in-memory and 60 GB / s when bigger than memory", to give perspective.