3 ms·

I congratulate the effort, as I always thought that Spark is great, but the fact it was written in Java hinders it quite badly (GC, tons of memory required for

I congratulate the effort, as I always thought that Spark is great, but the fact it was written in Java hinders it quite badly (GC, tons of memory required for the runtime, jar hell (want to use proto3 in your spark job? Good luck)). I do however worry that rust has a high bar of entry.

andygrove 7y ago
I agree. Rust has a very steep learning curve compared to JVM languages. My hope in building this platform is that it can provide value to other languages (especially JVM) by taking query plans and executing them efficiently.
- 0815test 7y ago
  > compared to JVM languages Apache Spark is written in Scala, and I wouldn't describe that as having an 'easy' learning curve, even compared to Rust! If you want something 'easy' on the JVM, Kotlin (and perhaps Eta or Frege) might be more appropriate.
  - pjmlp 7y ago
    Scala is way easier to use, given it uses a tracing GC, has three good IDEs, and more mature ecosystem.
  - twic 7y ago
    Coming from Java, i found Rust easier to learn than Scala.
    - Recurecur 7y ago
      I expect you were exposed using a heavily functional programming approach. That's a whole 'nother (larger) area to pick up along with the different language. If Scala is used as a "better Java" to start with, and then the developers explore FP at their own pace, I think there's a better outcome. Granted Java 8+ has done some to close the gap with Scala as well. I wonder if Scala-Native will absorb some Rustish features as other languages have or are attempting to... (What I said above echoes Odersky's ideas about Scala developer levels: https://www.scala-lang.org/old/node/8610.html https://www.scala-lang.org/old/node/8610.html)
pjmlp 7y ago
The first EA releases for value types are out, even if it takes a couple of releases more, it is much easier to improve existing stacks than restart from zero.
fnord123 7y ago
I'm quite sure as soon as there are performant WASM engines deployable outside a browser (e.g. parity-wasm) then Ballista will converge on it and it would hence support anything that can compile to wasm. In fact, I think the time to begin that type of project is now because by the time you do the ground work for such a project and integrate, etc. a wasm engine will be ready.

andygrove 7y ago

I agree. Rust has a very steep learning curve compared to JVM languages. My hope in building this platform is that it can provide value to other languages (especially JVM) by taking query plans and executing them efficiently.

0815test 7y ago
> compared to JVM languages Apache Spark is written in Scala, and I wouldn't describe that as having an 'easy' learning curve, even compared to Rust! If you want something 'easy' on the JVM, Kotlin (and perhaps Eta or Frege) might be more appropriate.
- pjmlp 7y ago
  Scala is way easier to use, given it uses a tracing GC, has three good IDEs, and more mature ecosystem.
- twic 7y ago
  Coming from Java, i found Rust easier to learn than Scala.
  - Recurecur 7y ago
    I expect you were exposed using a heavily functional programming approach. That's a whole 'nother (larger) area to pick up along with the different language. If Scala is used as a "better Java" to start with, and then the developers explore FP at their own pace, I think there's a better outcome. Granted Java 8+ has done some to close the gap with Scala as well. I wonder if Scala-Native will absorb some Rustish features as other languages have or are attempting to... (What I said above echoes Odersky's ideas about Scala developer levels: https://www.scala-lang.org/old/node/8610.html https://www.scala-lang.org/old/node/8610.html)

pjmlp 7y ago

The first EA releases for value types are out, even if it takes a couple of releases more, it is much easier to improve existing stacks than restart from zero.

fnord123 7y ago

I'm quite sure as soon as there are performant WASM engines deployable outside a browser (e.g. parity-wasm) then Ballista will converge on it and it would hence support anything that can compile to wasm. In fact, I think the time to begin that type of project is now because by the time you do the ground work for such a project and integrate, etc. a wasm engine will be ready.