4 ms·
These are great questions and topics I plan on addressing in a future blog post. SQL is a great convenience for simple analytical queries and it can be nice to
by andygrove 7y ago
These are great questions and topics I plan on addressing in a future blog post. SQL is a great convenience for simple analytical queries and it can be nice to be able to mix and match SQL and other access patterns (this is one thing I like with the Apache Spark DataFrame approach).
It's true that the Rust ecosystem for data science is not really there yet and I am trying to inspire people to start changing that. I think Rust does have some good potential here.
In the nearer term though I am exploring options around support user code in distributed query execution and I don't want to limit it to Rust.
- kyllo 7y agoI like it--I'm a data scientist by day, and I've been following Rust with interest but I haven't found a good use case for it in my job yet. A Rust-based Spark competitor sounds like it could be exactly that excuse to use Rust at work that I've been looking for!
- andygrove 7y agoThis is exactly the situation I am in. We have workloads that I think we could deliver with much lower TCO using Rust/DataFusion/Ballista. We could probably even use a single node for some of our workflows just using DataFusion directly with same performance as a distributed Spark job.