3 ms·
Rust will not succeed. Ppl can't debug spakr already and rust, even fewer ppl cna debug. Build a spark killer in Julia. Everyone can read the code.
by xiaodai 6y ago
Rust will not succeed. Ppl can't debug spakr already and rust, even fewer ppl cna debug.
Build a spark killer in Julia. Everyone can read the code.
- mumblemumble 6y agoI can't speak to the relative merits of Julia, but I am honestly interested in anything that seeks to produce a less memory-hungry alternative to Spark. Rust, to me, seems like a natural enough choice. It is easy to mate it to other languages, including all the major data science ones, so it would theoretically work well as the basis for a distributed compute engine that has good support for all of them as client languages. Would the same work for Julia? IIRC, it's a bytecode compiled language, which I imagine would make it difficult to link Julia libraries from other technology stacks.
- MrPowers 6y agoThere is a project called vega that's a Spark implementation in Rust: https://github.com/rajasekarv/vega https://github.com/rajasekarv/vega. It looks promising. Databricks rewrote the Spark engine in C++ (called the Delta Engine, see here: https://databricks.com/blog/2020/06/24/introducing-delta-engine.html https://databricks.com/blog/2020/06/24/introducing-delta-eng...). Hopefully there will be some good alternatives that don't consume so much memory soon ;)
- pjmlp 6y agoIt is the bytecode compiled language like any other that uses LLVM as their backend. The alternative to Spark will be Spark, as JVM slowly improves their memory APIs from release to release. People tend to hand wave how much it costs to reboot an ecosystem.