10 ms·
Rust Dataframe: Update 1
- nevi-me 7y agoTL:DR; I've been writing a Rust dataframe library for a while (on and off when I have time). This is my first update, to motivate why I'm writing it.
- kzrdude 7y agoExciting, looking forward to future updates. In relation to pandas and xarray, this would be more inspired by the former?
- nevi-me 7y agoYes, the former. I haven't seen xarray before, it seems interesting (still reading up on it). The ecosystem around Pandas is amazingly enormous :)
- kzrdude 7y agoxarray is more geared towards physical sciences data, data formats like HDF5 and so on, but the general concepts overlap a lot with a dataframe.
- nestorD 7y agoNice, there are definitely people trying to push more of their datascience/ML stack to Rust and a good dataframe implementation would be useful. As a side note, a small usage example in the readme would be good.
- nevi-me 7y agoThanks for the feedback. I was thinking that Update 2 would be in the form of examples of what can currently be done. I'll also add that to the README
- FridgeSeal 7y agoOh this is really cool! I could definitely make use of something like this.
- lmeyerov 7y agoGetting a fast & safe Rust UDF layer that targets SPIR/CUDA/PTX would be quite interesting wrt enabling RAPIDS.ai (libcudf + python bindings) as well. It'd enable getting rid of slow & quirky numba etc - I remember Mozilla had GPGPU RUST codegen experiments here awhile back...
- peterhj 7y agorustc does have a working nvptx target today, though it’s not supported nearly as well as the mainstream cpu targets, and some things you would really want for gpu programming (e.g. shared memory address space) are not currently exposed in the rust language. But kernels written in rust can compile to ptx; you’ll still need to write glue code.
- lmeyerov 7y agoYeah this would be about extending it to columnar analytics funcs, like `df['x'].apply(f)`, `df.query("x > 10 && y < 10"). I realized I may be wrong about the compiler speed part, not sure if it'd be faster than numba for codegen nowadays :)
- forrestthewoods 7y agoWhat is a dataframe? I wish libraries would define their key terminology. Especially when the term is rather generic.
- macawfish 7y agoIt's kinda like a spreadsheet with a programmatic interface. Check out pandas for a nice introduction!
- CameronNemo 7y agoAm I the only one who finds pandas to be terrible compared to dplyr?
- yumyai 7y agoMaybe? I hate how dplyr handle the rowname/index in general. In pandas, I could put 1-2 columns that I wouldn't use for the calculation in the index/multindex while dplyr NEED to rely on mutate/summarise_if() or gather function, and hope for the best.
- bandage 7y agoNo, pity that a lot of new build libraries and platform gets inspired by pandas workflow in search for something better, instead of looking at R ecosystem where people actually enjoy the available tools and workflows.
- Icathian 7y agoEveryone does. On the other hand I, and many others I assume, aren't willing to move to R for the sake of the one superior tool over python. Two if you count ggplot2, I suppose.
- CameronNemo 7y agoYeah but for giving someone an intro to what dataframes are, I think pandas might leave a sour taste where dplyr would allow someone to learn comfortably. Although, one might run into a "true level" situation. This is when Morty feels what true level is like, then says "everything is crooked, reality is poison" when he has to live in the "fake level" world.
- quotemstr 7y agoOne of the things that most pisses me off about numpy, pandas, and the whole scipy ecosystem is that everything is "immediate mode": all operations evaluate _instantly_ to new arrays or dataframes. There's no opportunity for any component to evaluate a whole expression tree and optimize it, e.g., by loop hoisting. The right way to design a data analysis DSL is to do it the way Python's dask does it: build an operation graph and execute it as the last step. The trouble is that Dask doesn't get it right either because, as part of its graph formation, it computes sizes of operands, and computing operand sizes can involve huge amounts of computation, and so dask, in effect, is also an "immediate mode" system. What I really want to see is something lazy that can do sane query planning and that can work within limited system resources. Maybe one day I'll open source the work I've done in this space. Query languages are infinitely nicer for analytics than data processing libraries.
- bobbylarrybobby 7y agoIt really is remarkable that Python's numerical computing libraries have such poor performance. When doing chains of elementwise operations on large arrays (such as toy example `elementwise cos(sqrt(sin(pow(array, 2))))`), Julia appears to outperform Python by a factor 2! Numpy cannot avoid computing each intermediate array, which means it has to allocate a ton of wasteful memory. Meanwhile Julia does the smart thing and coalesces all operations into one and applies that single operation elementwise, allocating only a single new array. Pandas does not also defer computations, which means computing Boolean functions that include the same data multiple times must make multiple passes over said data. Absurd.
- amelius 7y agoWhat bothers me is: why can't Rust developers just import a C++ library that does the job? What novelty would a Rust version of the same thing bring really? Why not focus on real innovation, and use wrappers for things that were already built in another language a decade ago?
- chungy 7y agoRust can't import C++ libraries, for many of the same reasons that C programs can't import C++ libraries.
- bluejekyll 7y agoRight. The C++ lib would need a C FFI interface for Rust to consume.
- alvarelle 7y agoOr using this crate: https://docs.rs/cpp/0.5.4/cpp/ https://docs.rs/cpp/0.5.4/cpp/
- WilliamEdward 7y agoDo you think all rust developers are connected together like some sort of hivemind? What an open source developer chooses to spend their time on doesn't require the consideration and consolidation of the entire rust team, nor does it speak for all of them. That said there are plenty of reasons why they shouldn't just import c++ libraries, one of them being that c++ is not the same language as rust.
- j88439h84 7y agoSome thoughts on dataframe. - Don't put methods on it like pandas.DataFrame. You won't get the API right the first many times and you'll end up with a million methods. - Make chaining easy. - Use pure functions. No mutation. - Get inspired by R's dplyr and data.table.
- nightcracker 7y ago> Don't put methods on it like pandas.DataFrame. You won't get the API right the first many times and you'll end up with a million methods. I was actually thinking about this for a bit. Since the data layout seems unlikely to ever backwards-incompatibly change, can you not define the API entirely in traits on the struct, and package them into one super trait (e.g. DataFrameAPIVersion1)? Then you can simply import the DataFrame struct as well as the version of the API you'll use, and be 100% compatible with any other code regardless of API version they use to manipulate the dataframes.
- j88439h84 7y agoThen application-specific functions would need a different syntax from the built in methods. Like df.pipe(f). I think that is not desirable. Better to have them all be detached from the frame
- nightcracker 7y agoThat's not true. You can implement your own traits for foreign types.
- aldanor 7y ago"Pure functions" is usually a bad idea with large numeric data structures if they are non-lazy - because you'll end up copying the data. One of the common pure alternatives is to make them lazy - i.e., all those pure methods don't do anything but rather just collect the information on what to do with your data. But then you need to write an execution engine for arbitrary computation graph, which ain't easy.
- Icathian 7y agoA good Rust equivalent to Pandas is really the main thing keeping me from switching to Rust for about half of my day job. This is incredibly promising, and I am following your work with great interest. Thank you for working on this!
- BubRoss 7y agoWhy would you want to use rust here in the first place? Won't it just slow down your iteration time?
- lumost 7y agoNot the OP, but also interested in similar applications of rust. The two main benefits I see are 1) Speed and memory efficiency, I've often worked in python/pandas and been frustrated by its performance. If I have a job that takes 12 hours to churn through all of the data, flipping over to rust may accelerate my iteration time to ~7.2-72 minutes. 2) Straightforward FFI bindings into other languages. Like C you can trivially bind rust code into python. Language choice can then be a bit of a two-way door back to python.
- Icathian 7y agoYou pretty much nailed my reasons as well. Trying to explain that updating model results for this month's data will take 12 hours usually does not go over well. The learning curve to move stuff to Rust would be considered well worth it.
- BubRoss 7y agoI thought pandas called into C libraries, does it not run close to native speeds?
- lumost 7y agoIt depends on what type of object you're using in pandas as well as whether you ever need to use a User Defined function e.g. pd.Series.map. Crossing between python objects and C objects is incredibly slow, and python defined UDFs will usually run at python speed. In most of my projects I'm unable to avoid using a udf/python data manipulation at some point.