6 ms·

Nim for scientific computing: Back to the future

116 points by nkurz 11y ago

nanny 11y ago
> there is no FFI to C What? Nim's homepage has the phrase "C FFI is easy in Nim.." along with some example code.
- dom96 11y ago
  I think this is referring to the lack of FFI support in NimScript (http://nim-lang.org/0.11.3/nims.html http://nim-lang.org/0.11.3/nims.html).
jph 11y ago
Nim looks great! I especially appreciate the types and validations. You mention Nim for scripting - can I pique your curiousity to add a Nim script example to the Num statistics project? http://github.com/numcommand/num http://github.com/numcommand/num
- pathsjs 11y ago
  I will have a look, but I cannot promise anything. There are a lot of things that I want to do and only so much time... :-)
ilaksh 11y ago
If only the Nim project had access to a few of the tens of millions being spent by Yahoo in order to more effectively deliver clickbait about the Kardashians and the like. I mean, if the money actually came from a different organization than Yahoo. Its good they do have support from 3DICC though.
dvdplm 11y ago
Lovely write up, thanks! The syntax is so much nicer than Rusts is. :)
antman 11y ago
Would be nice if NIM could be called from pyton passing numpy arrays. That would speed up adaptation until the NIM ecosystem grows.
- jboy 11y ago
  For the last 9 months I've been working (part-time) on a project that does exactly that. The startup where I work makes extensive use of this project to speed up our Python. The basic infrastructure was in place after ~2 months. Since then, our team has been using it extensively, and I've been filling out the functionality & evolving the API (for example, testing different array iteration & access interfaces, to see what feels awkward vs convenient, balancing the trade-off between Nim-ness vs Numpy-ness vs Pythonic-ness, etc.). You write your Nim procs, then annotate them with a special annotation (that's valid Nim, but inert most of the time). When you want to auto-generate Python wrappers around your Nim procs, there's a Python script you run that creates & compiles Python C-API code that exposes the Nim procs in Python as a shared library. This auto-generated code also translates Nim exceptions (including back-traces) to Python exceptions, calls the Nim GC before you return to Python runtime, etc. The auto-generated code also includes auto-generated Python docstrings, extracted from the Nim procs. There's a Nim type that provides a nice Nim interface to Numpy arrays, so you can pass Numpy arrays into your Nim procs and access them natively. We've also recently added support for a few cute features, like the tuple return types & default parameter values. :) The API is not yet solidified, so I'd call the project "alpha" or "beta", but it definitely works! The intention is to open-source it soon (once we're happy with the API). Would you be interested in using something like this? What features would you need? Even better, are you a Python+Nim coder who could contribute? Feel free to reply here or contact me via email (contact info in profile).
  - deleted 11y ago
    [deleted]
  - ngoldbaum 11y ago
    Have you and your employer thought about making this project open source?
ericfrederich 11y ago
You have plenty of options to interface C and Python. You can use Cython to create Python bindings to C libraries or even better you can use cffi which works with PyPy as well
- jboy 11y ago
  My issue with Cython is that (at least as I've understood it from the tutorials I've read) it's a sub-language within Python, where you add Cython elements iteratively+incrementally (diverging from Python in the process) until the code runs "fast enough". I'd rather work directly in a full-featured, internally-self-consistent language (and Nim is a very nice language) from the start. Hence -- for me, at least: Nim. :)
  - malmaud 11y ago
    Have you looked at Julia (http://julialang.org/ http://julialang.org/)?
    - jboy 11y ago
      Yes, I have, and Julia does look nice. It clearly has some very good libraries for scientific computing, and its syntax looks (to me) like an interesting combination of Python (which I like) and Matlab (which I don't like). Aside from an entirely-subjective preference that caused me to fall in love with Nim at first sight, there are practical reasons for choosing Nim over Julia. The overarching reason is that there is much more Python code in the world than Nim or Julia put together, and I think it will remain this way for a long time. (My startup previously worked almost completely in Python, for example, aside from some C or C++ extension modules and the usual Bash scripts for sysadmin.) Python also has some solid webserver frameworks, and is great for a variety of general-purpose tasks. So, I wanted a solution that would enable us to incrementally rewrite code as necessary, without needing to rewrite everything from Python all at once. Also, I have no desire to rewrite our webserver code from Python. Flask & Django are just fine. As I understand it (and please correct me if I'm wrong), Julia doesn't compile to C. So I can't compile snippets of Julia code and call them from my Python main-loop. In contrast, not only is this possible in Nim, but Nim's macro system & well-developed AST spec make it very easy to traverse my Nim procs during compilation, to auto-generate per-proc wrapper code in C.
vezzy-fnord 11y ago
Not sure why Nim keeps getting compared to Python. It's quite evidently influenced by the Wirthian languages, though I suppose contemporary programmers aren't really versed into those.
- TylerE 11y ago
  GC, indentation-significant syntax?
  - vezzy-fnord 11y ago
    Whitespace significance is practically the only resemblance, then. Wirth's languages obviously were garbage collected.
    - beagle3 11y ago
      Algol and Oberon were garbage collected, but Pascal and Modula-2 weren't. Not sure about others (Euler? Modula-3? Oberon-2?) Nim feels a lot more like Python than one would think just reading the descriptions. I can't put my finger on why, exactly - maybe the type inference that gives an almost duck-type experience, maybe the incredibly helpful dogfooding community; I'm not sure.
- lobster_johnson 11y ago
  I believe Nim's creator has a background using Borland's Pascal compilers (Delphi, Turbo Pascal). Nim is basically a modernized ObjectPascal with GC, generics and whitespace sensitivity. He even started out prefixing type names with T and P, just like Borland did, though fortunately that's going away.
- pathsjs 11y ago
  I only compare it to python because of the use case. Actually, if you read the post, I also consider Lua and other languages. Still, I think, the similarity in syntax can help. If there is one thing that python got right, in my opinion, is its straightforward syntax, and that helps saving brain power for something other than parsing
ricklamers 11y ago
What about Numba for Python in order to get the speed? Or is that a stupid suggestion for your requirements?
- pathsjs 11y ago
  Numba is a great technology, but I find it still a little disappointing. The way I see it is this. Python is not a particularly fast language, but has evolved a big ecosystem of scientific libraries. Now people are running into the language limitations, hence the development of various tools to compile parts of Python - either AOT or JIT - with PyPy, Numba, Cython, Numexpr and so on. The problem is that all these tools (save from PyPy, which is just a different interpreter) introduce their ow sublanguage, slightly incompatible from one another. For instance, both Numba and Cython have to infer types, but their type systems are not equal. Nim does not have the reach of Python ecosystem, but it is a solid language, that manages to combine well the speed of C and the simplicity and flexibility of Python. I would rather start from a solid foundation and add libraries on top of it gradually, rather than start from a developed ecosystem and add a host of tools that try to fix the inherent limitations of the platform. Hence, in the short term it is great that the projects created by Continuum exist, but eventually I would like to have something a little more consistent
approbatory 11y ago
In the comparison with torch you are translating x:dot(y) as x * y, but it should be x.t() * y
- pathsjs 11y ago
  No, it is right like that. In linalg, * on vectors is the scalar product. I have designed linalg to agree with mathematical conventions, rather than numpy conventions.
vegabook 11y ago
Can we realistically expect a competent, officially supported REPL for Nim over time? I would be immediately sold if this otherwise very promising looking language could remove this obstacle.
- TylerE 11y ago
  What's wrong with `nim i`? That's a repl, and it's hardly possible to get more official than that
  - perturbation 11y ago
    It's been removed* - provided a bad UX for people coming into Nim since it doesn't support C extensions / importing of modules that use C extensions, which can be a fair bit. If you try to use them regardless, it's easy to crash the REPL. * It's not entirely removed, just obfuscated so that if needed it can be still used as long as you "know what you're doing" and are willing to put up with it not being a general purpose REPL. Use nim secret to run it.
- pathsjs 11y ago
  I certainly hope so. It is of course the biggest missing element in sight, but I do not think we will see anything before 1.0 comes out
progman 11y ago
Nim's garbage collector is indeed very fast. I wrote an application which makes heavy use of string concatenation and a lot of regexp matches in native Perl syntax. I am amazed how performant the code is. Also compilation works pretty fast. However I didn't know yet that the Nim language is that powerful.

116 points by nkurz 11y ago

nanny 11y ago

> there is no FFI to C What? Nim's homepage has the phrase "C FFI is easy in Nim.." along with some example code.

dom96 11y ago
I think this is referring to the lack of FFI support in NimScript (http://nim-lang.org/0.11.3/nims.html http://nim-lang.org/0.11.3/nims.html).

jph 11y ago

Nim looks great! I especially appreciate the types and validations. You mention Nim for scripting - can I pique your curiousity to add a Nim script example to the Num statistics project? http://github.com/numcommand/num http://github.com/numcommand/num

pathsjs 11y ago
I will have a look, but I cannot promise anything. There are a lot of things that I want to do and only so much time... :-)

ilaksh 11y ago

If only the Nim project had access to a few of the tens of millions being spent by Yahoo in order to more effectively deliver clickbait about the Kardashians and the like. I mean, if the money actually came from a different organization than Yahoo. Its good they do have support from 3DICC though.

dvdplm 11y ago

Lovely write up, thanks! The syntax is so much nicer than Rusts is. :)

antman 11y ago

Would be nice if NIM could be called from pyton passing numpy arrays. That would speed up adaptation until the NIM ecosystem grows.

jboy 11y ago
For the last 9 months I've been working (part-time) on a project that does exactly that. The startup where I work makes extensive use of this project to speed up our Python. The basic infrastructure was in place after ~2 months. Since then, our team has been using it extensively, and I've been filling out the functionality & evolving the API (for example, testing different array iteration & access interfaces, to see what feels awkward vs convenient, balancing the trade-off between Nim-ness vs Numpy-ness vs Pythonic-ness, etc.). You write your Nim procs, then annotate them with a special annotation (that's valid Nim, but inert most of the time). When you want to auto-generate Python wrappers around your Nim procs, there's a Python script you run that creates & compiles Python C-API code that exposes the Nim procs in Python as a shared library. This auto-generated code also translates Nim exceptions (including back-traces) to Python exceptions, calls the Nim GC before you return to Python runtime, etc. The auto-generated code also includes auto-generated Python docstrings, extracted from the Nim procs. There's a Nim type that provides a nice Nim interface to Numpy arrays, so you can pass Numpy arrays into your Nim procs and access them natively. We've also recently added support for a few cute features, like the tuple return types & default parameter values. :) The API is not yet solidified, so I'd call the project "alpha" or "beta", but it definitely works! The intention is to open-source it soon (once we're happy with the API). Would you be interested in using something like this? What features would you need? Even better, are you a Python+Nim coder who could contribute? Feel free to reply here or contact me via email (contact info in profile).
- deleted 11y ago
  [deleted]
- ngoldbaum 11y ago
  Have you and your employer thought about making this project open source?

ericfrederich 11y ago

You have plenty of options to interface C and Python. You can use Cython to create Python bindings to C libraries or even better you can use cffi which works with PyPy as well

jboy 11y ago
My issue with Cython is that (at least as I've understood it from the tutorials I've read) it's a sub-language within Python, where you add Cython elements iteratively+incrementally (diverging from Python in the process) until the code runs "fast enough". I'd rather work directly in a full-featured, internally-self-consistent language (and Nim is a very nice language) from the start. Hence -- for me, at least: Nim. :)
- malmaud 11y ago
  Have you looked at Julia (http://julialang.org/ http://julialang.org/)?
  - jboy 11y ago
    Yes, I have, and Julia does look nice. It clearly has some very good libraries for scientific computing, and its syntax looks (to me) like an interesting combination of Python (which I like) and Matlab (which I don't like). Aside from an entirely-subjective preference that caused me to fall in love with Nim at first sight, there are practical reasons for choosing Nim over Julia. The overarching reason is that there is much more Python code in the world than Nim or Julia put together, and I think it will remain this way for a long time. (My startup previously worked almost completely in Python, for example, aside from some C or C++ extension modules and the usual Bash scripts for sysadmin.) Python also has some solid webserver frameworks, and is great for a variety of general-purpose tasks. So, I wanted a solution that would enable us to incrementally rewrite code as necessary, without needing to rewrite everything from Python all at once. Also, I have no desire to rewrite our webserver code from Python. Flask & Django are just fine. As I understand it (and please correct me if I'm wrong), Julia doesn't compile to C. So I can't compile snippets of Julia code and call them from my Python main-loop. In contrast, not only is this possible in Nim, but Nim's macro system & well-developed AST spec make it very easy to traverse my Nim procs during compilation, to auto-generate per-proc wrapper code in C.

vezzy-fnord 11y ago

Not sure why Nim keeps getting compared to Python. It's quite evidently influenced by the Wirthian languages, though I suppose contemporary programmers aren't really versed into those.

TylerE 11y ago
GC, indentation-significant syntax?
- vezzy-fnord 11y ago
  Whitespace significance is practically the only resemblance, then. Wirth's languages obviously were garbage collected.
  - beagle3 11y ago
    Algol and Oberon were garbage collected, but Pascal and Modula-2 weren't. Not sure about others (Euler? Modula-3? Oberon-2?) Nim feels a lot more like Python than one would think just reading the descriptions. I can't put my finger on why, exactly - maybe the type inference that gives an almost duck-type experience, maybe the incredibly helpful dogfooding community; I'm not sure.
lobster_johnson 11y ago
I believe Nim's creator has a background using Borland's Pascal compilers (Delphi, Turbo Pascal). Nim is basically a modernized ObjectPascal with GC, generics and whitespace sensitivity. He even started out prefixing type names with T and P, just like Borland did, though fortunately that's going away.
pathsjs 11y ago
I only compare it to python because of the use case. Actually, if you read the post, I also consider Lua and other languages. Still, I think, the similarity in syntax can help. If there is one thing that python got right, in my opinion, is its straightforward syntax, and that helps saving brain power for something other than parsing

ricklamers 11y ago

What about Numba for Python in order to get the speed? Or is that a stupid suggestion for your requirements?

pathsjs 11y ago
Numba is a great technology, but I find it still a little disappointing. The way I see it is this. Python is not a particularly fast language, but has evolved a big ecosystem of scientific libraries. Now people are running into the language limitations, hence the development of various tools to compile parts of Python - either AOT or JIT - with PyPy, Numba, Cython, Numexpr and so on. The problem is that all these tools (save from PyPy, which is just a different interpreter) introduce their ow sublanguage, slightly incompatible from one another. For instance, both Numba and Cython have to infer types, but their type systems are not equal. Nim does not have the reach of Python ecosystem, but it is a solid language, that manages to combine well the speed of C and the simplicity and flexibility of Python. I would rather start from a solid foundation and add libraries on top of it gradually, rather than start from a developed ecosystem and add a host of tools that try to fix the inherent limitations of the platform. Hence, in the short term it is great that the projects created by Continuum exist, but eventually I would like to have something a little more consistent

approbatory 11y ago

In the comparison with torch you are translating x:dot(y) as x * y, but it should be x.t() * y

pathsjs 11y ago
No, it is right like that. In linalg, * on vectors is the scalar product. I have designed linalg to agree with mathematical conventions, rather than numpy conventions.

vegabook 11y ago

Can we realistically expect a competent, officially supported REPL for Nim over time? I would be immediately sold if this otherwise very promising looking language could remove this obstacle.

TylerE 11y ago
What's wrong with `nim i`? That's a repl, and it's hardly possible to get more official than that
- perturbation 11y ago
  It's been removed* - provided a bad UX for people coming into Nim since it doesn't support C extensions / importing of modules that use C extensions, which can be a fair bit. If you try to use them regardless, it's easy to crash the REPL. * It's not entirely removed, just obfuscated so that if needed it can be still used as long as you "know what you're doing" and are willing to put up with it not being a general purpose REPL. Use nim secret to run it.
pathsjs 11y ago
I certainly hope so. It is of course the biggest missing element in sight, but I do not think we will see anything before 1.0 comes out

progman 11y ago

Nim's garbage collector is indeed very fast. I wrote an application which makes heavy use of string concatenation and a lot of regexp matches in native Perl syntax. I am amazed how performant the code is. Also compilation works pretty fast. However I didn't know yet that the Nim language is that powerful.