4 ms·
For the last 9 months I've been working (part-time) on a project that does exactly that. The startup where I work makes extensive use of this project to speed
by jboy 11y ago
For the last 9 months I've been working (part-time) on a project that does exactly that. The startup where I work makes extensive use of this project to speed up our Python.
The basic infrastructure was in place after ~2 months. Since then, our team has been using it extensively, and I've been filling out the functionality & evolving the API (for example, testing different array iteration & access interfaces, to see what feels awkward vs convenient, balancing the trade-off between Nim-ness vs Numpy-ness vs Pythonic-ness, etc.).
You write your Nim procs, then annotate them with a special annotation (that's valid Nim, but inert most of the time). When you want to auto-generate Python wrappers around your Nim procs, there's a Python script you run that creates & compiles Python C-API code that exposes the Nim procs in Python as a shared library. This auto-generated code also translates Nim exceptions (including back-traces) to Python exceptions, calls the Nim GC before you return to Python runtime, etc. The auto-generated code also includes auto-generated Python docstrings, extracted from the Nim procs.
There's a Nim type that provides a nice Nim interface to Numpy arrays, so you can pass Numpy arrays into your Nim procs and access them natively. We've also recently added support for a few cute features, like the tuple return types & default parameter values. :)
The API is not yet solidified, so I'd call the project "alpha" or "beta", but it definitely works! The intention is to open-source it soon (once we're happy with the API).
Would you be interested in using something like this? What features would you need? Even better, are you a Python+Nim coder who could contribute?
Feel free to reply here or contact me via email (contact info in profile).
- deleted 11y ago[deleted]
- ngoldbaum 11y agoHave you and your employer thought about making this project open source?
- jboy 11y agoWe haven't just thought about it, we intend to do it! :) We're just waiting until we're happy enough with our Numpy array API that we'd be confident in calling it "mostly stable". The (real) Numpy's C API is a perfect example of an API in which gradual accretion of new functions & functionality has given rise to a monstrous (both huge & horrible) API. The naming scheme & parameter patterns are inconsistent; sometimes the names are ambiguous (ie, unclear because they are overly general) or confusing (ie, the function does something other than what you'd expect from the name); and there are several very-similar functions, where it's not always clear which one you should use. I do understand why the Numpy C API has evolved this way: Once you introduce a function in an API, you can't remove it / rename it / change its parameters, or you will break backwards compatibility with your earlier releases and cause headaches for many of your users. (And of course, there are constraints upon an API written in C, that do not affect an API exposed in Python or Nim.) So we're really trying to dogfood our own API as much as possible before we inflict it upon the public. :)
- duerrp 11y agoSounds very interesting... I'd love to have a look.
- jboy 11y agoGreat! I see that you created "pyexperiment" [ https://github.com/duerrp/pyexperiment https://github.com/duerrp/pyexperiment ], so it's safe to assume you know your way around Python & Numpy. Is there a good way to get in contact with you?
- pathsjs 11y agoI certainly would be happy to know about it when it comes out! I would rather avoid the double ecosystem in the long run, but it would be a great thing to have until the Nim ecosystem grows.
- jboy 11y agoI assume you're the article author? If so, I know who you are on the Nim forums, so I'll certainly keep you informed. :) In terms of ecosystems, I think that there is deployed Python code in the world that will never be re-written in Nim. I would prefer that Nim can be used to write drop-in extension modules for this Python code, rather than not at all. I also think that there are situations where Python is a better fit for a problem than Nim. For example, handling JSON files in Nim (or any statically-typed language) will always be awkward (you have the choice between a double-dispatch Visitor Pattern or accessor methods that cast), but in Python, handling JSON is the easiest thing in the world. Likewise for any heterogeneous container. By the way: Great article about Nim, as always. :)