There is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega https://github.com/rajasekarv/vega. It looks promising.

There is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega https://github.com/rajasekarv/vega. It looks promising. Databricks rewrote the Spark engine in C++ (called the Delta Engine, see here: https://databricks.com/blog/2020/06/24/introducing-delta-engine.html https://databricks.com/blog/2020/06/24/introducing-delta-eng...). Hopefully there will be some good alternatives that don't consume so much memory soon ;)