3 ms·
cuDF – GPU DataFrame Library
- xrd 2y agoDoes this accelerate on an M1? I know this says it is for cuda and that obviously means Nvidia GPU, but lots of ML projects have a port to Apple silicon. I would love to try this on my Mac and see what kind of acceleration my pandas tools get for free.
- zamalek 2y agoI wish we could commit to not conflating NVIDIA with GPU. It wouldn't hurt a soul to call it "cuDF - NVIDIA DataFrame Library." To answer your question, it will probably run on the CPU.
- xs83 2y agoIt requires NVIDIA
- killingtime74 2y agoFrom the readme, no. Says NVIDIA drivers required
- xrd 2y agoI should have been more precise to say that I wondered if there was anyone trying to port it to the M1. It seems like every other ML project has some pull request trying that. Or, perhaps there is another project that is attempting to do the same thing for pandas on the M1. It's a noble goal.
- kiratp 2y agoJax is the closest thing at the moment. https://developer.apple.com/metal/jax/ https://developer.apple.com/metal/jax/ And MLX https://github.com/ml-explore/mlx https://github.com/ml-explore/mlx
- mvanaltvorst 2y agoJax-metal on Apple M-series GPUs is barely useable in my opinion. It's not possible to invert a matrix, for example, because Apple has not implemented the necessary triangular solve operator. It's also not possible to sample points from a normal distribution, because the Cholesky decomposition operator is not yet implemented. Apple hasn't responsed to both of these issues for the past year. It's difficult to take a numerical computing framework seriously if one cannot invert a matrix. [1]: https://github.com/google/jax/issues/16321 https://github.com/google/jax/issues/16321 [2]: https://github.com/google/jax/issues/17490 https://github.com/google/jax/issues/17490
- ashvardanian 2y agocuDF is the most impressive DataFrame implementation I've seen and have been recommending for years. The API is exceptionally close to Pandas (just a couple of different function arguments here and there), much more so than PyArrow or Modin. Throughput and energy efficiency were often 10x that of PyArrow running on a comparable TDP SotA CPU 2 years ago [1]. [1]: https://www.unum.cloud/blog/2022-09-20-pandas https://www.unum.cloud/blog/2022-09-20-pandas
- fbdab103 2y agoLimited by VRAM is a huge constraint for me. Even if it is slower, being able to load 100GB+ into RAM without any batching headaches is worth a lot. Unless cudf has implemented some clever dask+cudf kind of situation which can intelligently push data in/out of GPU as required?
- hack_ml 2y agoThere is dask cudf which gets a lot of the way there. https://docs.rapids.ai/api/dask-cudf/stable/ https://docs.rapids.ai/api/dask-cudf/stable/
- 3abiton 2y agoThis is huge, this was my only gripe with cudf!
- xs83 2y agoI did a conversion of 500GB of data using dask_cudf on a GTX 1060 with 6GB of VRAM and was able to do it faster than a 20 node m3.xlarge Cluster. What you can do on even consumer GPU's is mind blowing.
- ashvardanian 2y agoLast time I’ve used it, Dask was a lot worse than simple manual batching.
- 2y ago
- xs83 2y agoThis and Rapids.ai is the single reason that NVIDIA is the leader in AI. They made GPU processing at scale accessible to everyone, I have been a long term user of Rapids and found that even as a data engineer I can do things on an old consumer GPU that would otherwise require a 20+ node cluster to do in the same time.
- __mharrison__ 2y agoEven though other tools might be "better" than Pandas, it's ubiquity is why I suggest it. The ability to run coffee 100-1000x faster with this is just icing on the cake. (I've run through this with most of my Pandas training material and it just works with no code changes.)
- mafro 2y agoOut of interest, what other tools might be "better" than Pandas? I like pandas, and python.
- Carbocarde 2y agoI’m not who you were replying to but I’ve had positive experiences working with polars: https://pola.rs/ https://pola.rs/
- __mharrison__ 2y agoPolars is one (I'm a huge fan, I even wrote a book about it), SQL, R, Excel...
- internet101010 2y agoPolars is great. Most of my Python work involves wrangling data with pandas and polars was integral in getting me out of rust tutorial city and into rust production.
- CarRamrod 2y agoIs there something comparable to this for Matplotlib?
- mwexler 2y agoThis is actually a good callout. While pipeline speedups of transforms is hugely important, lots of other fundamental python tools for viz, model examination, etc are not built on a different foundation and not optimized by pandas improvements. I get it: some of these are legacy, others are hand optimized python since default pandas is so slow. But I'm hoping that, over time, we'll improve the runtime of the other stages of analysis too.
- hack_ml 2y agoThere are some integrations for stuff like https://docs.rapids.ai/visualization https://docs.rapids.ai/visualization : HoloViews hvPlot Datashader Plotly Bokeh Seaborn Panel PyDeck cuxfilter node RAPIDS
- mgt19937 2y agoHow does this compare to duckdb/polars? I wonder if GPU based compute engine is a good idea. GPU memory is expensive and limited. The bandwidth between GPU and main memory isn't very much either.
- pbib 2y agoThe same group (Nvidia/Rapids) is working on a similar project but with Polars API compatibility instead of Pandas. It seems to be quite far from completion, though. See discussion: https://news.ycombinator.com/item?id=39930846 https://news.ycombinator.com/item?id=39930846
- davidthewatson 2y agoThanks for the heads up. This is amazing. I've been watching cuda since it's introduction and Polars since I had an intern porting our Pandas code there a couple years ago but I had no idea Polars would go this far, this fast!
- skenderbeu 2y agoIs this GPU thing some kind of magic? We have like 12 different types of it in the wild. I think it's time we came up with a 1 or 2 GPU HW standards similar to how we have for CPUs.