4 ms·

Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at res

Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at reshaping and cleaning data.

jmartin2683 4y ago
We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?
- Kinrany 4y ago
  What are your requirements, and how do you store and load the data? Are you doing anything to select only the relevant parts of the parquet files?
  - jmartin2683 4y ago
    Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan only the bits of data we need, then predicate and projection push down (thanks to polars) to reduce that further.
    - Kinrany 4y ago
      Thanks. If you don't mind, what's the size of your individual files, and how many of them does it take to serve a single request: one, a few, N?
      - jmartin2683 4y ago
        Variable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).
- Dowwie 4y ago
  Why serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?
  - jmartin2683 4y ago
    We’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.
  - jmartin2683 4y ago
    The big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire universe of data. The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.
  - unoti 4y ago
    Using Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.
- maxFlow 4y ago
  I'm a Python dev, learning Rust out of interest, but also considering its viability for Data Engineering. It would be helpful to know the equivalents of the following libraries: Pandas (Polars), Prefect/Airflow/Orchestration, Flask, SqlAlchemy, psycopg2, requests, BeautifulSoup. Thanks!
  - Kinrany 4y ago
    Flask -> Axum (see also https://www.arewewebyet.org/topics/frameworks/#server https://www.arewewebyet.org/topics/frameworks/#server) SqlAlchemy -> Sqlx or SeaORM psycopg2 -> postgres and tokio-postgres requests -> reqwest BeautifulSoup -> scraper Searching for packages on lib.rs, crates.io or github usually works. There will be differences of course. The ecosystem is not as complete. On the other hand, writing the code and having it work the first time is easier once you know the language.
    - maxFlow 4y ago
      Brilliant, I'll be playing with these once I reach the more advanced material in The Book. Ty.
batterylow 4y ago
A couple of years ago when checking out Rust for scientific work I started looking for familiar ground, e.g. Jupyter notebook support. I was happy to find google/evcxr's Jupyter kernel [1] and started writing an early access book on Rust for data analysis [2]. I still check _Are we learning yet_ [3] to see their recommendation on the state of the ML stack (including data processing and structures) in Rust. It still lists "the ecosystem isn't very complete yet.", which is how I felt at the time! [1] https://github.com/google/evcxr/tree/main/evcxr_jupyter https://github.com/google/evcxr/tree/main/evcxr_jupyter [2] https://datacrayon.com/shop/product/data-analysis-with-rust-notebooks/ https://datacrayon.com/shop/product/data-analysis-with-rust-... [3] https://www.arewelearningyet.com https://www.arewelearningyet.com
62951413 4y ago
Isn't https://arrow.apache.org/ballista/ https://arrow.apache.org/ballista/ supposed to be a kind of next generation Spark written in Rust?
- tormeh 4y ago
  Datafusion is okay. Ballista is not at all ready at the moment.
mateo411 4y ago
I don't know a whole lot about Rust. I think Rust is a great language for writing Database components and systems. For example plugin for PostgreSQL. Previously this would be done in C. You could still do this in C today too. Data Engineers are heavy users of databases and different data stores, but they normally don't design database systems.

Is anyone familiar with any Rust language data libraries for data engineering? My experience with Rust is it's too low level and verbose to be productive at reshaping and cleaning data.

jmartin2683 4y ago

We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?

Kinrany 4y ago
What are your requirements, and how do you store and load the data? Are you doing anything to select only the relevant parts of the parquet files?
- jmartin2683 4y ago
  Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan only the bits of data we need, then predicate and projection push down (thanks to polars) to reduce that further.
  - Kinrany 4y ago
    Thanks. If you don't mind, what's the size of your individual files, and how many of them does it take to serve a single request: one, a few, N?
    - jmartin2683 4y ago
      Variable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).
Dowwie 4y ago
Why serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?
- jmartin2683 4y ago
  We’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.
- jmartin2683 4y ago
  The big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire universe of data. The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.
- unoti 4y ago
  Using Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.
maxFlow 4y ago
I'm a Python dev, learning Rust out of interest, but also considering its viability for Data Engineering. It would be helpful to know the equivalents of the following libraries: Pandas (Polars), Prefect/Airflow/Orchestration, Flask, SqlAlchemy, psycopg2, requests, BeautifulSoup. Thanks!
- Kinrany 4y ago
  Flask -> Axum (see also https://www.arewewebyet.org/topics/frameworks/#server https://www.arewewebyet.org/topics/frameworks/#server) SqlAlchemy -> Sqlx or SeaORM psycopg2 -> postgres and tokio-postgres requests -> reqwest BeautifulSoup -> scraper Searching for packages on lib.rs, crates.io or github usually works. There will be differences of course. The ecosystem is not as complete. On the other hand, writing the code and having it work the first time is easier once you know the language.
  - maxFlow 4y ago
    Brilliant, I'll be playing with these once I reach the more advanced material in The Book. Ty.

batterylow 4y ago

A couple of years ago when checking out Rust for scientific work I started looking for familiar ground, e.g. Jupyter notebook support. I was happy to find google/evcxr's Jupyter kernel [1] and started writing an early access book on Rust for data analysis [2]. I still check _Are we learning yet_ [3] to see their recommendation on the state of the ML stack (including data processing and structures) in Rust. It still lists "the ecosystem isn't very complete yet.", which is how I felt at the time! [1] https://github.com/google/evcxr/tree/main/evcxr_jupyter https://github.com/google/evcxr/tree/main/evcxr_jupyter [2] https://datacrayon.com/shop/product/data-analysis-with-rust-notebooks/ https://datacrayon.com/shop/product/data-analysis-with-rust-... [3] https://www.arewelearningyet.com https://www.arewelearningyet.com

62951413 4y ago

Isn't https://arrow.apache.org/ballista/ https://arrow.apache.org/ballista/ supposed to be a kind of next generation Spark written in Rust?

tormeh 4y ago
Datafusion is okay. Ballista is not at all ready at the moment.

mateo411 4y ago

I don't know a whole lot about Rust. I think Rust is a great language for writing Database components and systems. For example plugin for PostgreSQL. Previously this would be done in C. You could still do this in C today too. Data Engineers are heavy users of databases and different data stores, but they normally don't design database systems.