4 ms·

Rust! Polars/Datafusion are taking over, and for good reason.

Kinrany 4y ago
How are you using Datafusion? And overall, very interested in hearing about your experience. What does the whole stack/pipeline look like?
- jmartin2683 4y ago
  We've used both DataFusion and Polars for very similar use cases, though recently we've really begun to coalesce around Polars and most further development will use it, barring any complication that would require something else (DataFusion included). Basically what we do amounts to bringing in a ton of json, pulling objects out of that into jsonl, applying a bunch of business logic/transformation etc and storing that as parquet on s3, then querying that directly using a thin HTTP API wrapper around Polars. Everything is written in Rust. Experience thus far has been great. Initially all of the primary steps (json -> jsonl) were done in Python, but this proved an order of magnitude or two too slow. Rust (with a lot of optimizations... explicit SIMD etc) allowed for huge improvements. As far as querying the data, the biggest surprise thus far has been how performant Polars can be with thoughtful application. We're able to do a lot more with a lot less than initially expected. The cost savings for some of the ETL-type stuff vs doing it with a Spark cluster have also been very significant.
  - Kinrany 4y ago
    Thank you! I was planning to try using DataFusion to query large parquet files without retrieving them from s3 with https://github.com/datafusion-contrib/datafusion-objectstore-s3 https://github.com/datafusion-contrib/datafusion-objectstore.... I guess I should expect that that won't be necessary :)
    - jmartin2683 4y ago
      Datafusion has better support for object_store api (you can just register the store and pass it s3 uris). With polars, you have to have the file on the local file system (which I’ve found to be a huge performance win, cacheing them anyway).
- jmartin2683 4y ago
  Also, if you want to help... we're hiring :)
  - Kinrany 4y ago
    Heh, ask me in a few months when I'm done with the current thing :P

How are you using Datafusion? And overall, very interested in hearing about your experience. What does the whole stack/pipeline look like?

jmartin2683 4y ago
We've used both DataFusion and Polars for very similar use cases, though recently we've really begun to coalesce around Polars and most further development will use it, barring any complication that would require something else (DataFusion included). Basically what we do amounts to bringing in a ton of json, pulling objects out of that into jsonl, applying a bunch of business logic/transformation etc and storing that as parquet on s3, then querying that directly using a thin HTTP API wrapper around Polars. Everything is written in Rust. Experience thus far has been great. Initially all of the primary steps (json -> jsonl) were done in Python, but this proved an order of magnitude or two too slow. Rust (with a lot of optimizations... explicit SIMD etc) allowed for huge improvements. As far as querying the data, the biggest surprise thus far has been how performant Polars can be with thoughtful application. We're able to do a lot more with a lot less than initially expected. The cost savings for some of the ETL-type stuff vs doing it with a Spark cluster have also been very significant.
- Kinrany 4y ago
  Thank you! I was planning to try using DataFusion to query large parquet files without retrieving them from s3 with https://github.com/datafusion-contrib/datafusion-objectstore-s3 https://github.com/datafusion-contrib/datafusion-objectstore.... I guess I should expect that that won't be necessary :)
  - jmartin2683 4y ago
    Datafusion has better support for object_store api (you can just register the store and pass it s3 uris). With polars, you have to have the file on the local file system (which I’ve found to be a huge performance win, cacheing them anyway).
jmartin2683 4y ago
Also, if you want to help... we're hiring :)
- Kinrany 4y ago
  Heh, ask me in a few months when I'm done with the current thing :P