5 ms·

How do data scientists feel about Nim? It feels very pythony, but way faster - could be a really cool language for them, similar to what Julia is trying to be.

How do data scientists feel about Nim? It feels very pythony, but way faster - could be a really cool language for them, similar to what Julia is trying to be. Curious if anyone in a DS role has tried it out. I'm certainly tempted to try it out, I'm sure I'm losing lots of performance in parts of my analytics pipeline due to Python being garbage slow.

thanatropism 7y ago
Re: speed, I'm seeing the Rust ecosystem get better and better; I'm also learning the language on the weekends with small projects. But also: many many moons ago I had Matlab code that was really slow (and I wanted to run 10^LARGENUM simulations for Monte Carlo purposes) and decided to learn Fortran. It was surprisingly easy to learn, and it did cut my running time from afternoons to minutes. But in the process I found out that my Matlab performance problems were mostly due to arrays being dynamically resized all the time -- while static memory preallocation could be hacked easily in straight Matlab. As for Python, numpy is a fast embedded minilanguage written in C; the difference between "pythonic" and "numpythonic" is not emphasized enough in bootcamps, I think.
- galangalalgol 7y ago
  Matlab usually flags cases like that in the editor suggesting you reserve space outside a loop. My experience with numpy was that it was 11x slower than visual basic access in excel. I was deali g with a lot of small matricies instead of a big one and I think that contributed. I want to try julia in production. About to try out rust in production.
pastage 7y ago
The options in python to speed up your code are vast without really having to do much, on a fundamental level. My outdated experience with Nimrod told me that it was a tad too opinionated E.g. Everything is a string in dbs is mentioned in the article. It was an interesting language, for me it felt like a stepping stone to something better. But the docs were really bad,
- refaqtor 7y ago
  "Nimrod" experience is quite outdated. There have been great improvements in documentation in the last year.
- staticassertion 7y ago
  > The options in python to speed up your code are vast without really having to do much This does not match my experience. The #1 way I see to optimize Python is to not write Python at all, and instead to FFI out.
jaco8 7y ago
Nim is used by leading data scientists. Eg.: https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer
longemen3000 7y ago
there is this great discussion about the Nim Language on Julia Discourse: https://discourse.julialang.org/t/version-1-0-released-of-nim-programming-language/29099/125 https://discourse.julialang.org/t/version-1-0-released-of-ni...
freedatoms 7y ago
I work as a Data Scientist/Machine Learning Engineer and I think that biggest issue with Nim are lacking libraries. Working as a Data Scientist is very iterative process, you spend most of the time preprocessing data. It's often try and see approach so interactive repl is a must. While I know Nim has some library to be used with Jupyter, I found it too immature. Other reason to favor Python is it's libraries - they are often written in cython which yields comparable (but of course a bit slower) performance to C. And to speed up custom code you can almost always use numba (llvm-based jit compiler that works with CPython).
ploika 7y ago
The lack of mature libraries in the style of pandas, sklearn or the tidyverse, and the lack of a stable REPL is a major drawback for "production" data science. On the other hand though I've had fun playing with it in a personal capacity precisely because of the rough edges. If I want to fit a particular kind of model I'll probably have to roll my own, meaning I have to be familiar with what the algorithm is doing and why, rather than just importing something straight from sklearn. I also need to pay more attention to the code as I'm writing it instead of lazily debugging by trial and error in the REPL like I can with Python. Basically, I wouldn't recommend it at work, but it's been fun and useful at home.
- V1ndaar 7y ago
  Since you mention tidyverse, I might shamelessly mention that I'm currently working on a sort of port of ggplot2 and some dplyr features in Nim [1]. It's super WIP, but for most plots and simple data frame operations (complicated stuff I do beforehand) I need it has mostly replaced plotly [2] (or in some cases matplotlib). However, don't expect the included data frame to be fast. That in combination with a dynamic nature has been out of scope for me alone. [1] https://github.com/Vindaar/ggplotnim/tree/addDocs https://github.com/Vindaar/ggplotnim/tree/addDocs [2] https://github.com/brentp/nim-plotly https://github.com/brentp/nim-plotly
V1ndaar 7y ago
Not really a data scientist myself, but using Nim for my PhD in physics to analyze the data I took for an axion search experiment, code at [1]. I just love it. Super productive to work with and in my case faster than the C++ framework I'm replacing. The ecosystem of course is pretty small. But for me wrapping C code, porting something or reimplementing some libraries can be a lot of fun, too. In my experience every hole I've found was easily plugged so far. [1] https://github.com/Vindaar/TimepixAnalysis https://github.com/Vindaar/TimepixAnalysis (recently due to ..reasons.. all development happened on the open PR)
mratsim 7y ago
I am trying to improve Nim data science ecosystem with https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer. I think today the best way to try Nim is for slow Python utilities vi nimpy: https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy You can also use Nim in Jupyter the following way: https://github.com/apahl/nim_magic https://github.com/apahl/nim_magic My next step would be to improve interop between Arraymancer tensors and Numpy ndarrays.
- staticassertion 7y ago
  Very cool work! Thanks for the links.

thanatropism 7y ago

Re: speed, I'm seeing the Rust ecosystem get better and better; I'm also learning the language on the weekends with small projects. But also: many many moons ago I had Matlab code that was really slow (and I wanted to run 10^LARGENUM simulations for Monte Carlo purposes) and decided to learn Fortran. It was surprisingly easy to learn, and it did cut my running time from afternoons to minutes. But in the process I found out that my Matlab performance problems were mostly due to arrays being dynamically resized all the time -- while static memory preallocation could be hacked easily in straight Matlab. As for Python, numpy is a fast embedded minilanguage written in C; the difference between "pythonic" and "numpythonic" is not emphasized enough in bootcamps, I think.

galangalalgol 7y ago
Matlab usually flags cases like that in the editor suggesting you reserve space outside a loop. My experience with numpy was that it was 11x slower than visual basic access in excel. I was deali g with a lot of small matricies instead of a big one and I think that contributed. I want to try julia in production. About to try out rust in production.

pastage 7y ago

The options in python to speed up your code are vast without really having to do much, on a fundamental level. My outdated experience with Nimrod told me that it was a tad too opinionated E.g. Everything is a string in dbs is mentioned in the article. It was an interesting language, for me it felt like a stepping stone to something better. But the docs were really bad,

refaqtor 7y ago
"Nimrod" experience is quite outdated. There have been great improvements in documentation in the last year.
staticassertion 7y ago
> The options in python to speed up your code are vast without really having to do much This does not match my experience. The #1 way I see to optimize Python is to not write Python at all, and instead to FFI out.

jaco8 7y ago

Nim is used by leading data scientists. Eg.: https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer

longemen3000 7y ago

there is this great discussion about the Nim Language on Julia Discourse: https://discourse.julialang.org/t/version-1-0-released-of-nim-programming-language/29099/125 https://discourse.julialang.org/t/version-1-0-released-of-ni...

freedatoms 7y ago

I work as a Data Scientist/Machine Learning Engineer and I think that biggest issue with Nim are lacking libraries. Working as a Data Scientist is very iterative process, you spend most of the time preprocessing data. It's often try and see approach so interactive repl is a must. While I know Nim has some library to be used with Jupyter, I found it too immature. Other reason to favor Python is it's libraries - they are often written in cython which yields comparable (but of course a bit slower) performance to C. And to speed up custom code you can almost always use numba (llvm-based jit compiler that works with CPython).

ploika 7y ago

The lack of mature libraries in the style of pandas, sklearn or the tidyverse, and the lack of a stable REPL is a major drawback for "production" data science. On the other hand though I've had fun playing with it in a personal capacity precisely because of the rough edges. If I want to fit a particular kind of model I'll probably have to roll my own, meaning I have to be familiar with what the algorithm is doing and why, rather than just importing something straight from sklearn. I also need to pay more attention to the code as I'm writing it instead of lazily debugging by trial and error in the REPL like I can with Python. Basically, I wouldn't recommend it at work, but it's been fun and useful at home.

V1ndaar 7y ago
Since you mention tidyverse, I might shamelessly mention that I'm currently working on a sort of port of ggplot2 and some dplyr features in Nim [1]. It's super WIP, but for most plots and simple data frame operations (complicated stuff I do beforehand) I need it has mostly replaced plotly [2] (or in some cases matplotlib). However, don't expect the included data frame to be fast. That in combination with a dynamic nature has been out of scope for me alone. [1] https://github.com/Vindaar/ggplotnim/tree/addDocs https://github.com/Vindaar/ggplotnim/tree/addDocs [2] https://github.com/brentp/nim-plotly https://github.com/brentp/nim-plotly

V1ndaar 7y ago

Not really a data scientist myself, but using Nim for my PhD in physics to analyze the data I took for an axion search experiment, code at [1]. I just love it. Super productive to work with and in my case faster than the C++ framework I'm replacing. The ecosystem of course is pretty small. But for me wrapping C code, porting something or reimplementing some libraries can be a lot of fun, too. In my experience every hole I've found was easily plugged so far. [1] https://github.com/Vindaar/TimepixAnalysis https://github.com/Vindaar/TimepixAnalysis (recently due to ..reasons.. all development happened on the open PR)

mratsim 7y ago

I am trying to improve Nim data science ecosystem with https://github.com/mratsim/Arraymancer https://github.com/mratsim/Arraymancer. I think today the best way to try Nim is for slow Python utilities vi nimpy: https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy You can also use Nim in Jupyter the following way: https://github.com/apahl/nim_magic https://github.com/apahl/nim_magic My next step would be to improve interop between Arraymancer tensors and Numpy ndarrays.

staticassertion 7y ago
Very cool work! Thanks for the links.