3 ms·

Anyone have thoughts on nim vs rust for data centric (Scala like) applications

qeternity 6y ago
If they’re data centric, would you not be better off with vanilla python? If you have numpy do the heavy lifting, it’ll be as fast plus all the syntactic/ecosystem benefits
- nkozyra 6y ago
  Yeah, I like nim but it isn't going to offer and advantage over python in this regard.
- ca_parody 6y ago
  The main reasons I see for nim in scientific computing over python are: (a) you need performance for applications outside of numpy / pandas & you want to avoid writing a C extension to do so (nim is also actually a good choice for python extensions via nimpy - if you are still using python as the main language of your project/system) (a2) you want good concurrency (some would say python's async/await is good - but it feels too little too late to me) & you at least want the option of parallelism (like this article takes advantage of) (b) you want / need the powerful macro system for dsls / custom (compile-type-checkable) language extensions. There is a good future for nim in scientific computing (frankly, there is a good present) - but for it to be better, people need to be willing to forgo some of the ecosystem (when possible) and to help build that future. https://github.com/nim-lang/needed-libraries/issues/77 https://github.com/nim-lang/needed-libraries/issues/77
- logicchains 6y ago
  The OP said Scala-like. A big advantage of Scala/Spark over Python is more compile-time checking: the code is much less likely to fail at runtime. Particulary helpful for production code (e.g. pipelines). For people coming from an engineering background (as opposed to data science), Python's lack of any static type checking for data processing code can be a big productivity killer. In that sense, Nim might be able to offer similar advantages to Scala.
- nimmer 6y ago
  Numpy is nice but pure-Nim applications appear to outperform it easily: https://narimiran.github.io/2018/05/10/python-numpy-nim.html https://narimiran.github.io/2018/05/10/python-numpy-nim.html https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer
  - qeternity 6y ago
    Interesting but given the low total execution time, the majority of time in the numpy implementation is going to be spent initializing CPython. I think if this were on the order of minutes/hours you would see numpy outperform.
zamalek 6y ago
Apart from Python, you could look at Julia.
untog 6y ago
I can't speak to data centric but I was looking at both Rust and Nim for a project recently. I love the syntax of Nim and it feels incredibly easy to write in. The power of the macro system is amazing. But the Rust community is so much bigger, pretty much every question I had was answered after a quick Google search.
- steve_adams_86 6y ago
  I agree. I really struggled to get help with certain problems with Nim, but with Rust it has been fairly trivial to find help. The community is very active and helpful. Nim does have a syntax I prefer more, but overall I find I’m quite a bit more productive with Rust.
  - beagle3 6y ago
    The most effective venue for quick Nim answers is the Nim forum - almost everything gets a serious well thought out answer within a few hours to a day. That said, nothing beats “already been answered” and the Rust community is much bigger.
ca_parody 6y ago
its been a couple years - but mratsim (the same person who authored this ray tracing post) wrote this (which is at least tangential to your question) https://medium.com/@MARatsimbazafy/why-rust-fails-hard-at-scientific-computing-9dcb140c8d81 https://medium.com/@MARatsimbazafy/why-rust-fails-hard-at-sc.... Some of the rust problems mentioned (like clone for arrays) have been fixed. Others, like static generics, have not been (AFAIK).
totalperspectiv 6y ago
That depends on why you use Scala? Do you use Scala as a better Java? If so, Nim may be what you want. Do you use Scala for its incredible type system? Rust may be what you want. Do you use Scala for its functional tendencies? Rust may be what you want. As an aside, I've found that once you get over the Rust learning curve, writing 'correct' programs is easier in Rust than any other language I've used. And for data centric applications, being 'correct' is extra important.
- nautilus12 6y ago
  What about scalable data? Like spark applications? I am familiar with weld but seems like more mature distributed computing is still on the horizon

If they’re data centric, would you not be better off with vanilla python? If you have numpy do the heavy lifting, it’ll be as fast plus all the syntactic/ecosystem benefits

nkozyra 6y ago
Yeah, I like nim but it isn't going to offer and advantage over python in this regard.
ca_parody 6y ago
The main reasons I see for nim in scientific computing over python are: (a) you need performance for applications outside of numpy / pandas & you want to avoid writing a C extension to do so (nim is also actually a good choice for python extensions via nimpy - if you are still using python as the main language of your project/system) (a2) you want good concurrency (some would say python's async/await is good - but it feels too little too late to me) & you at least want the option of parallelism (like this article takes advantage of) (b) you want / need the powerful macro system for dsls / custom (compile-type-checkable) language extensions. There is a good future for nim in scientific computing (frankly, there is a good present) - but for it to be better, people need to be willing to forgo some of the ecosystem (when possible) and to help build that future. https://github.com/nim-lang/needed-libraries/issues/77 https://github.com/nim-lang/needed-libraries/issues/77
logicchains 6y ago
The OP said Scala-like. A big advantage of Scala/Spark over Python is more compile-time checking: the code is much less likely to fail at runtime. Particulary helpful for production code (e.g. pipelines). For people coming from an engineering background (as opposed to data science), Python's lack of any static type checking for data processing code can be a big productivity killer. In that sense, Nim might be able to offer similar advantages to Scala.
nimmer 6y ago
Numpy is nice but pure-Nim applications appear to outperform it easily: https://narimiran.github.io/2018/05/10/python-numpy-nim.html https://narimiran.github.io/2018/05/10/python-numpy-nim.html https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer
- qeternity 6y ago
  Interesting but given the low total execution time, the majority of time in the numpy implementation is going to be spent initializing CPython. I think if this were on the order of minutes/hours you would see numpy outperform.

zamalek 6y ago

Apart from Python, you could look at Julia.

untog 6y ago

I can't speak to data centric but I was looking at both Rust and Nim for a project recently. I love the syntax of Nim and it feels incredibly easy to write in. The power of the macro system is amazing. But the Rust community is so much bigger, pretty much every question I had was answered after a quick Google search.

steve_adams_86 6y ago
I agree. I really struggled to get help with certain problems with Nim, but with Rust it has been fairly trivial to find help. The community is very active and helpful. Nim does have a syntax I prefer more, but overall I find I’m quite a bit more productive with Rust.
- beagle3 6y ago
  The most effective venue for quick Nim answers is the Nim forum - almost everything gets a serious well thought out answer within a few hours to a day. That said, nothing beats “already been answered” and the Rust community is much bigger.

ca_parody 6y ago

its been a couple years - but mratsim (the same person who authored this ray tracing post) wrote this (which is at least tangential to your question) https://medium.com/@MARatsimbazafy/why-rust-fails-hard-at-scientific-computing-9dcb140c8d81 https://medium.com/@MARatsimbazafy/why-rust-fails-hard-at-sc.... Some of the rust problems mentioned (like clone for arrays) have been fixed. Others, like static generics, have not been (AFAIK).

totalperspectiv 6y ago

That depends on why you use Scala? Do you use Scala as a better Java? If so, Nim may be what you want. Do you use Scala for its incredible type system? Rust may be what you want. Do you use Scala for its functional tendencies? Rust may be what you want. As an aside, I've found that once you get over the Rust learning curve, writing 'correct' programs is easier in Rust than any other language I've used. And for data centric applications, being 'correct' is extra important.

nautilus12 6y ago
What about scalable data? Like spark applications? I am familiar with weld but seems like more mature distributed computing is still on the horizon