3 ms·
We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?
by jmartin2683 4y ago
We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?
- Kinrany 4y agoWhat are your requirements, and how do you store and load the data? Are you doing anything to select only the relevant parts of the parquet files?
- jmartin2683 4y agoRequirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan only the bits of data we need, then predicate and projection push down (thanks to polars) to reduce that further.
- Kinrany 4y agoThanks. If you don't mind, what's the size of your individual files, and how many of them does it take to serve a single request: one, a few, N?
- jmartin2683 4y agoVariable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).
- Dowwie 4y agoWhy serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?
- jmartin2683 4y agoWe’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.
- jmartin2683 4y agoThe big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire universe of data. The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.
- unoti 4y agoUsing Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.
- maxFlow 4y agoI'm a Python dev, learning Rust out of interest, but also considering its viability for Data Engineering. It would be helpful to know the equivalents of the following libraries: Pandas (Polars), Prefect/Airflow/Orchestration, Flask, SqlAlchemy, psycopg2, requests, BeautifulSoup. Thanks!
- Kinrany 4y agoFlask -> Axum (see also https://www.arewewebyet.org/topics/frameworks/#server https://www.arewewebyet.org/topics/frameworks/#server) SqlAlchemy -> Sqlx or SeaORM psycopg2 -> postgres and tokio-postgres requests -> reqwest BeautifulSoup -> scraper Searching for packages on lib.rs, crates.io or github usually works. There will be differences of course. The ecosystem is not as complete. On the other hand, writing the code and having it work the first time is easier once you know the language.
- maxFlow 4y agoBrilliant, I'll be playing with these once I reach the more advanced material in The Book. Ty.