14 ms·

I use Nim instead of Python for data processing (2021)

95 points by archargelod 2y ago

bionsystem 2y ago
> import bioseq # my library, has k-mer iterator and FASTA parsing I'm not a developer, but isn't that a bit unfair ? If one is going to rewrite all libs instead of just importing them from python, one is trading convenience for speed. That said it's also mentioned that nim is compatible with python, does that mean it can import python libraries directly ?
- leetnewb 2y ago
  There is nimpy (https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy)
- Gethsemane 2y ago
  It does seem like a bit of an unfair comparison. That being said, I am not aware of any k-mer python packages that have great performance. If k-mer counting performance was really important, I'd probably start to think about reaching for one of the more performant counters, such as KMC. The real benefit of python in bioinformatics is the number of libraries available that let you interact with a bunch of software, perhaps do some light analysis, maybe make some pretty plots. And if you want to do some more intensive "useful" analysis, then all the ML packages, numpy, polars etc are quite hard to dismiss.
fithisux 2y ago
Is Nim supporting vectorization? Is Nim having x64 intrinsincs?
- nick__m 2y ago
  vectorisation sure: https://forum.nim-lang.org/t/2677 https://forum.nim-lang.org/t/2677 x64 intrinsic are nit in std lib but they exist: https://github.com/bsegovia/x86_simd.nim https://github.com/bsegovia/x86_simd.nim
  - fithisux 2y ago
    I'm downloading it :-)
nick__m 2y ago
while unexplored in this article, the Nim type system is pretty nice too, particularly the subrange type, the enum and the object variants.
- benjamin-lee 2y ago
  Author here. I love the type system. By using distinct strings to represent DNA, RNA, and protein, I can avoid silly errors while still using the optimized implementations under the hood. This is what the `bioseq` library (about two hundred lines) does [1] and I find it incredibly elegant. [1] https://github.com/Benjamin-Lee/vdsearch/blob/2045b29928f7b4ad268dbd8296b0ad3a99942e90/vdsearch/nim/bioseq.nim https://github.com/Benjamin-Lee/vdsearch/blob/2045b29928f7b4...
elashri 2y ago
> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? But I have a problem with talking real life application and using that to claim > it will be impossible for pure Python to beat pure Nim at raw performance Because it maybe be true. I didn't try it but there are many things that can be optimized in the python code example but away from that. In real life application in scientific computing I don't think anyone wouldn't use numpy to deal with that which will make things much better. Also the power of python in data analysis and scientific computing is the ecosystem and community. This will be very hard to beat. And there are more mature alternatives like Julia. Edit: The author code for reading the data reading is creating a new file object for each iteration. I would guess that in nim this would be a similar problem but I am not sure how it actually work or if has the same effect. But anyway you don't do this in real life application with python. Also it would be nice to use a list comprehension to count the occurrences of 'C' and 'G' in each line.
- aargh_aargh 2y ago
  > > so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. > Is this the scientific version of "rich people problems"? It's more like programming to solve many people's recurring problem and selling that as a product vs. programming to solve one man's many different ad hoc problems.
- benjamin-lee 2y ago
  >> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? > Is this the scientific version of "rich people problems"? Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not. However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.
dang 2y ago
Discussed at the time: Why I Use Nim instead of Python for Data Processing - https://news.ycombinator.com/item?id=28626947 https://news.ycombinator.com/item?id=28626947 - Sept 2021 (179 comments)
mg 2y ago
The optimizations the compiler can do because of the "var" the author added in the nim version of the first example should be also possible without it. Because Python defines variables as local or global at compile time. The code looks like they avoided that by putting the code into the global scope? If the code of the first example is what the author really ran, than he got a speed penalty for running the loop in the global scope. One might consider it a bit of a quirk that Python code runs slower in the global scope. But in practice, it rarely matters. As a script with just a loop and no functions (not even a main function) is so rare.
jillesvangurp 2y ago
Python popularity is self re enforcing. It's popular because it's popular. Not because it is fast or particularly good at anything it does. It arguably isn't that great for a lot of things people use it for. A lot of that popularity just stems from people already knowing it and the existence of lots of libraries and tools. It's a bit the Visual Basic (if you are old enough to remember that being popular) of the data science world. Lots of people use it that aren't necessarily very good software engineers or even focused on that. It's not my first choice for a lot of things but I can work with it when needed and it gets the job done. I'm currently doing a project on the side in python for one of my clients and I'm getting familiar with some new things like async and a few other things. It's fine. Nim, Mojo, and a few other python dialects kind of point to the future for python itself as well. The recent decision to get rid of the GIL is a step in the right direction and there has been incremental progress on interpreter and jit internals as well in recent releases. It might never get quite as fast but it can get a lot closer to those. And having some new options to implement stuff in that needs to be fast other than C is probably a good thing. A lot of popular python libraries don't necessarily contain a lot of python code. You see the same in the javascript world where there's a lot of WASM creeping into the space lately.
- wraptile 2y ago
  > Python popularity is self re enforcing. It's popular because it's popular I'm sorry but that's just an incredibly ignorant take. Python revolutionized programming. I was in Euro Python 2016 (or something) and Guido van Rossum (founder of the language) did a surprise visit and a line of people would form around him every now and then of people thanking him for making programming and development so thoughtful and accessible which changed people's lives. I was in the line too. Python's approach to programming and the whole language design, community, enhancement proposals like PEP-8 and Python Foundation to this day is mostly unmatched. Python is absolutely brilliant!
  - deleted 2y ago
    [deleted]
  - moi2388 2y ago
mfld 2y ago
This example will highly benefit from JIT compilation, as it is already possible in cpython with the @jit decorator of numba. I assume the time benefit of Nim will eventually fade away, while you still have the benefit that every dev can understand python.
- 082349872349872 2y ago
  I'm sure any decent dev could understand Nim
in_ab 2y ago
Nim is fast, powerful and has a lightweight syntax. I used it for a lot for hobby projects. I wrote program in Nim that started bringing in some money. But soon as feature requests from customers started coming in, I had to rewrite it. The tooling for Nim and library ecosystem is just not there. It's so much more productive to work with Python, .NET or JVM. I decided to rewrite it in Kotlin because JVM gave me similar performance.
- molticrystal 2y ago
  I am curious and wondering if you could elaborate on what you ended up making and why picked nim over lets say python, was the core usage something that was performance based or did it really fit the nim feature set better than most languages? Would you say nim is good for prototyping or would you say it was a waste of time since you ended up having to port it for tooling?
julianeon 2y ago
I don't think the advantage here is so much that Nim is fast, as Python is slow. If you're willing to dump Python you have many compiled language options, but I'll pick two: C and Rust. For the kind of tasks the author outlines, I'd use AI. It excels at this: these are really simple, well-defined tasks it won't screw up. So what I would do is pick a faster language - I'd pick Rust - then ask AI to script it and then repeat for as many tasks as you need.
- benjamin-lee 2y ago
  Author here. This is basically the approach I am using these days to get maximum multithreaded performance for when it really counts (inner loops) [1]. I draft in Python and use Copilot to convert it to Rust, then optimize from there. However, Nim is still better than Rust in my opinion for simple scripts that I don't want to spend a bunch of time writing. Its only major downside is its relative lack of support for parallelism and bioinformatics (i.e., why I used Rust for a more serious project). [1] https://github.com/Benjamin-Lee/circkit https://github.com/Benjamin-Lee/circkit
big-green-man 2y ago
I like nim, though I agree that the toolset isn't there yet. However I have encountered an interesting quirk with nim that kind of drove me crazy. In a lot of languages, string types are syntactic sugar for arrays or lists of chars. And usually, when you try to parse a string as a list, because it's just syntactic sugar, it works flawlessly. In nim it is also true that strings are just lists of chars, but for some reason the compiler will not allow you to treat it as such! It seems to have all kinds of special corner cases about how you can do one thing or another that behave differently. It doesn't really seem to have a holistic fundamental design or form, it feels to me like just a bunch of stuff slapped together. But if you can get to know the quirks, it's incredibly powerful, if for no reason alone, that tooling exists to transpile nim into anything. Well, almost. A core part of the design philosophy is to leverage as much existing tooling as possible, and so it inherits this property and enables you to compile nim for just about any architecture and into any language. This to me is incredibly powerful.
- elcritch 2y ago
  The primary reason for the separate string type is to interop better with the various backends. The default C backend appends an extra null termination to ease C compatibility.
  - big-green-man 2y ago
    Yeah I think I looked into it and read something like that, I guess when your primary goal is interoperability with already built systems you've got to make trade offs like that and they'll show through in the experience, but it doesn't change the fact that nim feels a little incoherent when you're trying to learn it. It's fine though, I like nim a lot and the value in the language significantly outweighs the downsides to learning those little quirks.
- archargelod 2y ago
  > In nim it is also true that strings are just lists of chars, but for some reason the compiler will not allow you to treat it as such! That's actually a feature - Nim is designed to be a type safe[0] language. Type safety has a number of advantages: - Compile-time checks: Type system catches many potential issues and silly bugs early at compile time. - API discoverability: IDEs and other tools can provide better autocompletion and type hints because they know exactly what operations are available on a string. Nim tries to strike a balance between flexibility and type safety. Instead of having strict restrictions, you have explicit control with slightly more verbose code. For example, if you want to treat a string as an array, you can use the `openarray` type (`openarray` is a special type that acts as interface to all array-like types and is supported by most of stdlib): var hello = "Hello, World" proc foo(s: seq[char]): bool = discard proc bar(s: openarray[char]): bool = discard echo foo(hello) # this doesn't compile because string != seq[char] echo bar(hello) # this compiles because openarray[char] is compatible with string / seq[char] / etc.. [0] - https://en.wikipedia.org/wiki/Type_safety https://en.wikipedia.org/wiki/Type_safety

95 points by archargelod 2y ago

> import bioseq # my library, has k-mer iterator and FASTA parsing I'm not a developer, but isn't that a bit unfair ? If one is going to rewrite all libs instead of just importing them from python, one is trading convenience for speed. That said it's also mentioned that nim is compatible with python, does that mean it can import python libraries directly ?

leetnewb 2y ago
There is nimpy (https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy)
Gethsemane 2y ago
It does seem like a bit of an unfair comparison. That being said, I am not aware of any k-mer python packages that have great performance. If k-mer counting performance was really important, I'd probably start to think about reaching for one of the more performant counters, such as KMC. The real benefit of python in bioinformatics is the number of libraries available that let you interact with a bunch of software, perhaps do some light analysis, maybe make some pretty plots. And if you want to do some more intensive "useful" analysis, then all the ML packages, numpy, polars etc are quite hard to dismiss.

fithisux 2y ago

Is Nim supporting vectorization? Is Nim having x64 intrinsincs?

nick__m 2y ago
vectorisation sure: https://forum.nim-lang.org/t/2677 https://forum.nim-lang.org/t/2677 x64 intrinsic are nit in std lib but they exist: https://github.com/bsegovia/x86_simd.nim https://github.com/bsegovia/x86_simd.nim
- fithisux 2y ago
  I'm downloading it :-)

nick__m 2y ago

while unexplored in this article, the Nim type system is pretty nice too, particularly the subrange type, the enum and the object variants.

benjamin-lee 2y ago
Author here. I love the type system. By using distinct strings to represent DNA, RNA, and protein, I can avoid silly errors while still using the optimized implementations under the hood. This is what the `bioseq` library (about two hundred lines) does [1] and I find it incredibly elegant. [1] https://github.com/Benjamin-Lee/vdsearch/blob/2045b29928f7b4ad268dbd8296b0ad3a99942e90/vdsearch/nim/bioseq.nim https://github.com/Benjamin-Lee/vdsearch/blob/2045b29928f7b4...

elashri 2y ago

> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? But I have a problem with talking real life application and using that to claim > it will be impossible for pure Python to beat pure Nim at raw performance Because it maybe be true. I didn't try it but there are many things that can be optimized in the python code example but away from that. In real life application in scientific computing I don't think anyone wouldn't use numpy to deal with that which will make things much better. Also the power of python in data analysis and scientific computing is the ecosystem and community. This will be very hard to beat. And there are more mature alternatives like Julia. Edit: The author code for reading the data reading is creating a new file object for each iteration. I would guess that in nim this would be a similar problem but I am not sure how it actually work or if has the same effect. But anyway you don't do this in real life application with python. Also it would be nice to use a list comprehension to count the occurrences of 'C' and 'G' in each line.

aargh_aargh 2y ago
> > so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. > Is this the scientific version of "rich people problems"? It's more like programming to solve many people's recurring problem and selling that as a product vs. programming to solve one man's many different ad hoc problems.
benjamin-lee 2y ago
>> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? > Is this the scientific version of "rich people problems"? Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not. However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.

dang 2y ago

Discussed at the time: Why I Use Nim instead of Python for Data Processing - https://news.ycombinator.com/item?id=28626947 https://news.ycombinator.com/item?id=28626947 - Sept 2021 (179 comments)

mg 2y ago

The optimizations the compiler can do because of the "var" the author added in the nim version of the first example should be also possible without it. Because Python defines variables as local or global at compile time. The code looks like they avoided that by putting the code into the global scope? If the code of the first example is what the author really ran, than he got a speed penalty for running the loop in the global scope. One might consider it a bit of a quirk that Python code runs slower in the global scope. But in practice, it rarely matters. As a script with just a loop and no functions (not even a main function) is so rare.

jillesvangurp 2y ago

Python popularity is self re enforcing. It's popular because it's popular. Not because it is fast or particularly good at anything it does. It arguably isn't that great for a lot of things people use it for. A lot of that popularity just stems from people already knowing it and the existence of lots of libraries and tools. It's a bit the Visual Basic (if you are old enough to remember that being popular) of the data science world. Lots of people use it that aren't necessarily very good software engineers or even focused on that. It's not my first choice for a lot of things but I can work with it when needed and it gets the job done. I'm currently doing a project on the side in python for one of my clients and I'm getting familiar with some new things like async and a few other things. It's fine. Nim, Mojo, and a few other python dialects kind of point to the future for python itself as well. The recent decision to get rid of the GIL is a step in the right direction and there has been incremental progress on interpreter and jit internals as well in recent releases. It might never get quite as fast but it can get a lot closer to those. And having some new options to implement stuff in that needs to be fast other than C is probably a good thing. A lot of popular python libraries don't necessarily contain a lot of python code. You see the same in the javascript world where there's a lot of WASM creeping into the space lately.

wraptile 2y ago
> Python popularity is self re enforcing. It's popular because it's popular I'm sorry but that's just an incredibly ignorant take. Python revolutionized programming. I was in Euro Python 2016 (or something) and Guido van Rossum (founder of the language) did a surprise visit and a line of people would form around him every now and then of people thanking him for making programming and development so thoughtful and accessible which changed people's lives. I was in the line too. Python's approach to programming and the whole language design, community, enhancement proposals like PEP-8 and Python Foundation to this day is mostly unmatched. Python is absolutely brilliant!
- deleted 2y ago
  [deleted]
- moi2388 2y ago

mfld 2y ago

This example will highly benefit from JIT compilation, as it is already possible in cpython with the @jit decorator of numba. I assume the time benefit of Nim will eventually fade away, while you still have the benefit that every dev can understand python.

082349872349872 2y ago
I'm sure any decent dev could understand Nim

in_ab 2y ago

Nim is fast, powerful and has a lightweight syntax. I used it for a lot for hobby projects. I wrote program in Nim that started bringing in some money. But soon as feature requests from customers started coming in, I had to rewrite it. The tooling for Nim and library ecosystem is just not there. It's so much more productive to work with Python, .NET or JVM. I decided to rewrite it in Kotlin because JVM gave me similar performance.

molticrystal 2y ago
I am curious and wondering if you could elaborate on what you ended up making and why picked nim over lets say python, was the core usage something that was performance based or did it really fit the nim feature set better than most languages? Would you say nim is good for prototyping or would you say it was a waste of time since you ended up having to port it for tooling?

julianeon 2y ago

I don't think the advantage here is so much that Nim is fast, as Python is slow. If you're willing to dump Python you have many compiled language options, but I'll pick two: C and Rust. For the kind of tasks the author outlines, I'd use AI. It excels at this: these are really simple, well-defined tasks it won't screw up. So what I would do is pick a faster language - I'd pick Rust - then ask AI to script it and then repeat for as many tasks as you need.

benjamin-lee 2y ago
Author here. This is basically the approach I am using these days to get maximum multithreaded performance for when it really counts (inner loops) [1]. I draft in Python and use Copilot to convert it to Rust, then optimize from there. However, Nim is still better than Rust in my opinion for simple scripts that I don't want to spend a bunch of time writing. Its only major downside is its relative lack of support for parallelism and bioinformatics (i.e., why I used Rust for a more serious project). [1] https://github.com/Benjamin-Lee/circkit https://github.com/Benjamin-Lee/circkit

big-green-man 2y ago

I like nim, though I agree that the toolset isn't there yet. However I have encountered an interesting quirk with nim that kind of drove me crazy. In a lot of languages, string types are syntactic sugar for arrays or lists of chars. And usually, when you try to parse a string as a list, because it's just syntactic sugar, it works flawlessly. In nim it is also true that strings are just lists of chars, but for some reason the compiler will not allow you to treat it as such! It seems to have all kinds of special corner cases about how you can do one thing or another that behave differently. It doesn't really seem to have a holistic fundamental design or form, it feels to me like just a bunch of stuff slapped together. But if you can get to know the quirks, it's incredibly powerful, if for no reason alone, that tooling exists to transpile nim into anything. Well, almost. A core part of the design philosophy is to leverage as much existing tooling as possible, and so it inherits this property and enables you to compile nim for just about any architecture and into any language. This to me is incredibly powerful.

elcritch 2y ago
The primary reason for the separate string type is to interop better with the various backends. The default C backend appends an extra null termination to ease C compatibility.
- big-green-man 2y ago
  Yeah I think I looked into it and read something like that, I guess when your primary goal is interoperability with already built systems you've got to make trade offs like that and they'll show through in the experience, but it doesn't change the fact that nim feels a little incoherent when you're trying to learn it. It's fine though, I like nim a lot and the value in the language significantly outweighs the downsides to learning those little quirks.
archargelod 2y ago
> In nim it is also true that strings are just lists of chars, but for some reason the compiler will not allow you to treat it as such! That's actually a feature - Nim is designed to be a type safe[0] language. Type safety has a number of advantages: - Compile-time checks: Type system catches many potential issues and silly bugs early at compile time. - API discoverability: IDEs and other tools can provide better autocompletion and type hints because they know exactly what operations are available on a string. Nim tries to strike a balance between flexibility and type safety. Instead of having strict restrictions, you have explicit control with slightly more verbose code. For example, if you want to treat a string as an array, you can use the `openarray` type (`openarray` is a special type that acts as interface to all array-like types and is supported by most of stdlib): var hello = "Hello, World" proc foo(s: seq[char]): bool = discard proc bar(s: openarray[char]): bool = discard echo foo(hello) # this doesn't compile because string != seq[char] echo bar(hello) # this compiles because openarray[char] is compatible with string / seq[char] / etc.. [0] - https://en.wikipedia.org/wiki/Type_safety https://en.wikipedia.org/wiki/Type_safety