3 ms·

A good Rust equivalent to Pandas is really the main thing keeping me from switching to Rust for about half of my day job. This is incredibly promising, and I a

A good Rust equivalent to Pandas is really the main thing keeping me from switching to Rust for about half of my day job. This is incredibly promising, and I am following your work with great interest. Thank you for working on this!

BubRoss 7y ago
Why would you want to use rust here in the first place? Won't it just slow down your iteration time?
- lumost 7y ago
  Not the OP, but also interested in similar applications of rust. The two main benefits I see are 1) Speed and memory efficiency, I've often worked in python/pandas and been frustrated by its performance. If I have a job that takes 12 hours to churn through all of the data, flipping over to rust may accelerate my iteration time to ~7.2-72 minutes. 2) Straightforward FFI bindings into other languages. Like C you can trivially bind rust code into python. Language choice can then be a bit of a two-way door back to python.
  - Icathian 7y ago
    You pretty much nailed my reasons as well. Trying to explain that updating model results for this month's data will take 12 hours usually does not go over well. The learning curve to move stuff to Rust would be considered well worth it.
    - BubRoss 7y ago
      I thought pandas called into C libraries, does it not run close to native speeds?
      - lumost 7y ago
        It depends on what type of object you're using in pandas as well as whether you ever need to use a User Defined function e.g. pd.Series.map. Crossing between python objects and C objects is incredibly slow, and python defined UDFs will usually run at python speed. In most of my projects I'm unable to avoid using a udf/python data manipulation at some point.
nevi-me 7y ago
Out of interest, what are the common tasks that you'd be looking to achieve? We can't replace Pandas and its ecosystem in the short-term, so for me a Rust-backed backend to a dataframe that is compatible with Pandas would be a win.
- Icathian 7y ago
  My workflow usually reads something like this: 1. Import data using hand-written SQL from SQL server (read_sql) 2. Perform various filtering, aggregation, math (loc, iterrows/itertuples, groupby, agg) 3. Push results either out to delivery file or back to source server (to_csv, to_excel, to_sql). Really bread and butter stuff, but the relative ease and stability of using pandas to do it is the attraction.
  - deleted 7y ago
    [deleted]
- potatochup 7y ago
  We use pandas for writing assertions on time series data collected from simulated and real firmeare. Eg: - two signals X and Y should always be within 5% of each other while condition Z is true. - signal A should transition from value M to N within time window t1-t2 Would love something in rust to match the rest of the infrastructure. Python also has plenty of warts with regards to packaging, cross compilation and deployment

BubRoss 7y ago

Why would you want to use rust here in the first place? Won't it just slow down your iteration time?

lumost 7y ago
Not the OP, but also interested in similar applications of rust. The two main benefits I see are 1) Speed and memory efficiency, I've often worked in python/pandas and been frustrated by its performance. If I have a job that takes 12 hours to churn through all of the data, flipping over to rust may accelerate my iteration time to ~7.2-72 minutes. 2) Straightforward FFI bindings into other languages. Like C you can trivially bind rust code into python. Language choice can then be a bit of a two-way door back to python.
- Icathian 7y ago
  You pretty much nailed my reasons as well. Trying to explain that updating model results for this month's data will take 12 hours usually does not go over well. The learning curve to move stuff to Rust would be considered well worth it.
  - BubRoss 7y ago
    I thought pandas called into C libraries, does it not run close to native speeds?
    - lumost 7y ago
      It depends on what type of object you're using in pandas as well as whether you ever need to use a User Defined function e.g. pd.Series.map. Crossing between python objects and C objects is incredibly slow, and python defined UDFs will usually run at python speed. In most of my projects I'm unable to avoid using a udf/python data manipulation at some point.

nevi-me 7y ago

Out of interest, what are the common tasks that you'd be looking to achieve? We can't replace Pandas and its ecosystem in the short-term, so for me a Rust-backed backend to a dataframe that is compatible with Pandas would be a win.

Icathian 7y ago
My workflow usually reads something like this: 1. Import data using hand-written SQL from SQL server (read_sql) 2. Perform various filtering, aggregation, math (loc, iterrows/itertuples, groupby, agg) 3. Push results either out to delivery file or back to source server (to_csv, to_excel, to_sql). Really bread and butter stuff, but the relative ease and stability of using pandas to do it is the attraction.
- deleted 7y ago
  [deleted]
potatochup 7y ago
We use pandas for writing assertions on time series data collected from simulated and real firmeare. Eg: - two signals X and Y should always be within 5% of each other while condition Z is true. - signal A should transition from value M to N within time window t1-t2 Would love something in rust to match the rest of the infrastructure. Python also has plenty of warts with regards to packaging, cross compilation and deployment