10 ms·

Why I Use Nim instead of Python for Data Processing

256 points by benjamin-lee 5y ago

liamwestray 5y ago
While I trust the author on this, I don’t think DNA datasets and string analysis was a great example. One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do. Not only does it compress your dataset (2 bits vs 16 bits), it allows absurdly faster numerical libraries to be used in lieu of string methods. There’s no real point in showing off that a compiled language is faster at doing something the slow way…
- benjamin-lee 5y ago
  You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucleotide [2]. One could get creative with a 4 bit encoding and still end up saving space (assuming you don't care about the distinction between upper versus lowercase characters in your sequence [3]). Or, if you know in advance your sequence is unambiguous (unlikely in DNA sequencing-derived data) you could use the 2 bit encoding. When dealing with short nucleotide sequences, another approach is to encode the sequence as an integer. I would love to see a library—Python, Nim, or otherwise—that made using the most efficient encoding for a sequence transparent to the developer. [1] https://github.com/biocore/scikit-bio/blob/b470a55a8dfd054ae109d813ef2fd28a246dab6e/skbio/sequence/_genetic_code.py#L536 https://github.com/biocore/scikit-bio/blob/b470a55a8dfd054ae... [2] https://en.wikipedia.org/wiki/Nucleic_acid_notation https://en.wikipedia.org/wiki/Nucleic_acid_notation [3] https://bioinformatics.stackexchange.com/questions/225/uppercase-vs-lowercase-letters-in-reference-genome https://bioinformatics.stackexchange.com/questions/225/upper...
  - liamwestray 5y ago
    Yeah, this is why my comment led with “I trust the author”… I’m surprised you need the full 4 bits to deal with ambiguous bases, but it probably makes sense at some lower level I don’t understand.
    - benjamin-lee 5y ago
      This is because there's four bases and each can either be included or excluded from a given combination. So there are 4*2 = 16 combinations each of which with their own letter. In all honesty, these are pretty rarely used in practice these days except for N (any base) although they do sometimes show up when representing consensus sequences.
      - pdimitar 5y ago
        What do you mean that each base can be included or excluded? Isn't only one extra value needed? Sort of like nil?
        ac29 5y ago
        Because there are notations for any combination of bases. There's a way to indicate "C or G", "A or T", "C, G, or A", etc.
        pdimitar 5y ago
        Oh. So what's the grand total of all possible permutations of single and multiple (connected with an "or") values? I'll also read through your links, thanks for posting them.
- jjtheblunt 5y ago
  Aren't the reads emitting a set of size greater than 4 bases per position, with a wildcard or "?" perhaps one option? (As in GATTACA might be read as is, but might be read as GAT?ACA.) Still that's a minimal of 3 bits versus much longer. [Edit : i see another commenter with the same observation, more thoroughly explained! ]
joshu 5y ago
does nim have anything like scikit yet?
- styluss 5y ago
  Closest I can find is https://nimble.directory/pkg/science https://nimble.directory/pkg/science
zmmmmm 5y ago
It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a whole suite of languages that easily cruise to a 10x performance win over Python. And that isn't counting multicore - if you count that you quickly get to a 100x improvement. Personally I use Groovy for much of what I do for similar reasons (which is somewhat unusual) but its just a placeholder for "use anything except python".
- amyjess 5y ago
  > It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via direct SQL or a REST API to whatever service owns the database). All these tasks were heavily network-bound. The amount of time the CPU was doing any work was minuscule compared to the amount of time it was waiting to get data back from the network. I doubt Nim or any other language would have been a significant performance improvement in this case. For what it's worth, that made these scripts excellent candidates for multithreading. I'd run them with 20+ threads, and it was glorious. At first I did multiprocessing, because of all the GIL horror stories, but multiprocessing made it very difficult to cache data, so eventually I said "well, all this is network-bound so the GIL doesn't even apply" and switched over to multiprocessing.dummy (which implements pools using the same API as multiprocessing but with threads instead of processes), and I never looked back. Edit: For what it's worth, Nim sounds like a really cool language, and it's right up my alley in several ways, I just don't think Python is particularly slow at network-bound tasks that use very little CPU.
  - otherme123 5y ago
    You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again). But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.
TekMol 5y ago
TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.
- ur-whale 5y ago
  Python is indeed slow, but why would anyone ever use php as a comparison point? And for data processing of all things ... Php is also very slow, on top of being many other kinds of unpleasant and broken.
  - pjmlp 5y ago
    PHP has a proper JIT compiler on its reference implementation.
    - ur-whale 5y ago
      > PHP has a proper JIT compiler on its reference implementation. That doesn't make it fast at all. Just faster than if it wasn't jitted. And that certainly does not eradicate the vast ocean of other problems php has, the first of which being that is was never, ever "though out" and instead grew like a cancerous mushroom.
  - DeathArrow 5y ago
    I had a nightmare where I was writing an operating system in PHP, which ran in hypervisor written in PHP, which ran in a virtual machine written in PHP, which in a browser written in PHP, which ran in an operating system written in PHP...
- fctorial 5y ago
  > that is the achilles heel of Python I think that is pip.
xiaodai 5y ago
How does it compare to Julia? Anyone with experience in both Nim and Julia?
- agons 5y ago
  I would have thought Cython would be the closest analogue for comparison.
- Sanguinaire 5y ago
  I have tinkered with both. Basically if you need to write a FORTRAN application but don't want to use FORTRAN, Julia is the correct choice. If ease of use is more important than having a diverse collection of pre-existing numerical libraries to play with, I'd go with Nim. If Nim had cloud SDKs I would use it as my default language for pretty much everything.
ur-whale 5y ago
what's up we the all the shouting in the code ?
- choneone 5y ago
  It's because the `font-feature-settings` of the main font leak to the code font. The feature 'case' turns the code all uppercase.
  - ghostly_s 5y ago
    Ha, I thought the author was intentionally writing the python in all-caps to highlight the similarity in syntax to the alternative language (not being familiar with Nim, I figured it must be conventional there). I was going to post a comment expressing my surprise that was even legal.
blondin 5y ago
i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.
- shoo 5y ago
  that can be a good approach where the computation maps cleanly onto some C or fortran code prepared earlier and wrapped as a numpy or scipy primitive. but sometimes expressing what you want to do as a bunch of numpy operations becomes a lot harder to read and perhaps also slower than if you just directly wrote the raw loops over some arrays in C. in cases where what you want to do doesn't exactly fit standard operations, cython can be pretty nice. e.g. 200x -- 1000x speedups for translating C-oriented number crunching code from python to cython. but if you do want performance, you have to think about it while writing the code (avoid needlessly allocating memory in tight loops, data-oriented programming with simple arrays, statically type all of your variables, ...).
ellimilial 5y ago
A context might be useful. From what I gather, the author is a researcher in bioinformatics related field. This may indicate that they tend to work either alone or in a relatively small group. The domain is small scope data processing/manipulation, research/exploratory code, ,likely short-lived or even one-off. The progress in this context will possibly be governed by sheer processing speed (e.g. it’s unlikely anyone will delve deep into the code, a lot of iterations to ‘just get it done’ instead of testing etc.). If this is more or less correct, the point that Nim might be more useful than Python for the author sounds very sensible to me. It’s a nice spot between command line tools and more functionality-loaded languages.
- benjamin-lee 5y ago
  Author here. This is spot on. The majority of the code I write is either piping data around to existing tools using shell scripting and Snakemake or writing the data processing code myself when there isn't a tool that does what I need. Usually, I'm working alone or with a few other computational biologists. Many of my scripts are one-off but they have the distinct tendency of growing in complexity and scope if they are useful. That's one of the big advantages with Nim in my mind: you can write a quick and dirty script and have it be pretty fast and then go back later and optimize it to a few percent of C without having to rewrite your code in another language. In this sense, it's quite like Julia (another really good language).
DeathArrow 5y ago
If anyone is interested to see how Nim fares against some other programming languages, here are some benchmarks: https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks
dilawar 5y ago
NIM also has a very good JavaScript backend. You can generate both C and JavaScript code from a nim program. Last time I used it, I liked it but didn't use it long enough to have a strong opinion.
brabel 5y ago
> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes ridiculous amounts of RAM even for hello world. Did the author post the data set they used for the examples? Would be nice to try it out on a few languages to see how fast that can compile and run on a mature language like Common Lisp (which is just as easy to write) or even node.js.
- benjamin-lee 5y ago
  I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Severe%20acute%20respiratory%20syndrome%20coronavirus%202,%20taxid:2697049 https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...
  - brabel 5y ago
    I clicked on the big Download button and selected "all records", it downloaded over 3.5GB before I gave up... which file exactly should I use??
    - benjamin-lee 5y ago
      I'm sorry, I completely forgot that the file I used was from six months ago when I wrote the blog post (and then promptly forgot to publish it). In the last half year, the number of coronavirus sequences has increased dramatically. One thing that you could do to drop the file size down is to filter for only complete and unambiguous sequences, which drops the number down from 1.6 million to ~100k [1]. Alternatively, the exact file I used for the post is available for one week here with MD5 sum 3c33c3c4c2610f650c779291668450c9 [2]. Anyone who wants the file is free to reach out to me directly (email is on site). [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Severe%20acute%20respiratory%20syndrome%20coronavirus%202,%20taxid:2697049&QualNum_i=0&Completeness_s=complete https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType... [2] https://file.io/nUNc7cG5i8gj https://file.io/nUNc7cG5i8gj
piqufoh 5y ago
Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish before the kettle has boiled.
- elcritch 5y ago
  That’s where Nim can shine. For simple scripts both Python and Nim are about as easy to write. But the Nim version usually runs a lot faster. Static types help for basic data munging when you haven’t used a script for months to get up to speed and make tweaks.
  - jimbob45 5y ago
    Sadly, I think you’re spot-on about Nim’s future as the realization of the alternative timeline where Python didn’t make several stupid design choices (e.g. the GIL, Python 3). It’s a shame because I think Nim has some neat features that allow it to present as a serious competitor to Rust but it will ultimately have to compete against Python instead to secure its niche.
    - elcritch 5y ago
      Oh yes, Nim definitely feels like an alternate reality where Python 2 became static and dumped some poor design choices. Well I believe there's room between Rust and Python where Nim can grow. It made the TIOBE top 50 lately even. Likely it can eat enough market share from the edges of both Rust & Python to become more well known (more libs, tools, etc). Rust is fantastic but tedious to program (to me at least) and it's community focuses on more formal type traits, etc making "scripting" trickier. Python is great for a mix of quick scripts, web dev, and data science but it's slow enough (and getting complex enough!) for many to want something faster and more stable yet still easy to write. Nim lives in between them and is more enjoyable to write than either for many. Also, Nim _could_ add Rust as a backend target and be relevant even if Rust displaces C/C++. ;) Nim is also great for embedded systems too! I've been using it a fair bit and it's really nice [1]. There's a lot of room to grow in that field. 1: https://github.com/elcritch/nesper https://github.com/elcritch/nesper
oxfordmale 5y ago
The author makes a fair point, however, that is a rather non optimal implementation in Python. You likely could use chunked Pandas to speed up or the code, or at least replace some of the for loops with a list comprehension syntax. However, in any case I would never replace Python with Nim as it is too niche of a language and you would struggle with recruiting. I could consider Julia if it's popularity keeps growing. That is the ultimate challenge of a language. It either needs a large backer (Go and Google) or be so good, it gets a natural market adaptation(Julia). As a manager I am reluctant to adapt yet another language unless there is a healthy job market for it.
- anyfactor 5y ago
  There are objectively better niche solutions for niche problems out there. But we pickup things that can be applied to solve a number of different problem, that are more versatile and has a community behind them.
- pdimitar 5y ago
  There's a class of technologies falling under what I'd call "most of your engineers would pick that in a weekend". Not all technologies require the full cycle and the normal risk management.
- Daishiman 5y ago
  Agreed. I am certainly inclined to believe that Nim is a better language than Python, but it's not so much better thlo justify moving off of the ecosystem.
cycomanic 5y ago
One pitfall that I ran into when trying out Nim for scientific computing is that Nim follows more a computer science convention than mathematics convention for exponentiation and negation operators. That is in Nim -2^3=(-2)^3 unlike more scientific computing oriented languages where -2^3=-(2^3). To someone like myself who mainly does scientific work this was quite unexpected and it causes a surprising amount of mental overhead to avoid mistakes. I did like Nim quite a bit otherwise, but essentially found that I was missing some important numerical libraries so did not continue using it.
- jb1991 5y ago
  To be honest, regardless of the behavior of the language, I would be wrapping things in parentheses to make it explicit anyway.
- leephillips 5y ago
  Although Nim using weird order of operations is unfortunate, your example is not well chosen. Replace the exponent 3 with an even integer and your point will be clear.
  - leephillips 5y ago
    Maybe not that weird, although not the most common syntax. In Nim 0 - 2^2 evaluates to -4, while -2^2 evaluates to +4. So in -2, the - is treated as a unary minus and binds tightly to the 2. It would be bad if Nim were doing + or - before ^, or before *. I have a feeling some other languages treat the unary minus the same way, but don’t know of any examples off hand.
    - cycomanic 5y ago
      Yes when I asked a question about this on their gitter channel, people brought up several languages I think bc is one of them for example. I understand the reasoning I still find it extremely unintuitive personally.
- tzs 5y ago
DeathArrow 5y ago
Yes, Python is painfully slow, but it shouldn't matter. If you are using Python where performance, speed, correctness, testability, maintainability matters, you are doing it wrong. Python is good where speed of development matters, where you write throw-away code testing some ideas and you want to do it fast, where you write glue code, for prototypes, for small code bases. Once you are getting outside of that area, you better should use a language more suited for the task. As for myself, even if I can use Python in some cases, I can churn C# code almost as fast so I prefer doing it that way in case I want to grow the code later or use it somewhere else. Being lazy, I dislike rewriting code.
santiagobasulto 5y ago
I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edit: my code is not perfect (I’m typing from my phone, I’m surprised I could even match parentheses). My point is: this is a highly I/O bound program. The implementation matters. With the correct implementation there shouldn’t be much difference between the languages.
- kiidev 5y ago
  Isn't that going to be even slower, since a 156mb file would probably make gc_lines a very big tuple before summing it?
  - henbruas 5y ago
    gc_lines isn't a tuple here, it's a generator expression. It will be lazily evaluated.
  - makapuf 5y ago
    You could do gc=sum(1 for _ on lines if ...) or use Counter
- deleted 5y ago
  [deleted]
- ellimilial 5y ago
  Isn’t this counting the lines with any G/C in them vs the total number of G/C literals?
- _dain_ 5y ago
  >total = len(list(gc_lines)) That won't work properly; you've already exhausted the gc_lines generator in the previous line.
pella 5y ago
related: Biofast benchmark ( Nim, Julia, Go, Pypy, C, Crystal, .. ) "Benchmarking programming languages/implementations for common tasks in Bioinformatics" https://github.com/lh3/biofast#fqcnt https://github.com/lh3/biofast#fqcnt https://lh3.github.io/2020/05/17/fast-high-level-programming-languages https://lh3.github.io/2020/05/17/fast-high-level-programming... HN: https://news.ycombinator.com/item?id=23229657 https://news.ycombinator.com/item?id=23229657
DeathArrow 5y ago
While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people, it would mean we would have to write many things ourselves, which in other languages we can take for granted as they have libraries for almost anything. So having a nice, performant and good language is just a small part of achieving your goals. You also need the people and the ecosystem. Go, Rust, Kotlin, Swift and even Julia have the luck of having some industry heavyweights behind them, pushing the ecosystem and contributing with money and developers. Nim has only a bunch of passionate people behind it.
- cnmlp 5y ago
  In the case of Python, the "industry heavyweights" do very little and have a negative influence now. Why the phrase "only a bunch of passionate people"? This is how software gets written, parasitical corporations and their unproductive developers who are installed in existing OSS projects come later and mainly associate themselves with the result (speaking of Python again).
- dejj 5y ago
  > How would I convince my employer Turntables: if you were an employer, what would convince you to use Nim? Hiring for Nim skills can be a signal that a company has people who learn languages beyond the run-of-the-mill ones. A bunch of passionate people you might say. That would make the company promising to work for.
  - WJW 5y ago
    Interestingly this was exactly the place Python was in back in the day: niche enough that anyone knowing it must've learned it because they thought it was cool rather than because it would get them a job. Those days are long gone of course.
- egwor 5y ago
paulluuk 5y ago
As a Data Engineer, I mainly use Python in conjunction with PySpark. So essentially I just use Python as an easy-to-read wrapper around Spark, and it works great when working together with Data Scientists, who are mostly used to Pandas, Keras, Tensorflow, etc. In my use case, I don't really see how Nim would make my life easier right now.
haxscramper 5y ago
Another nim & python thread that has not been mentioned yet here https://news.ycombinator.com/item?id=28506531 https://news.ycombinator.com/item?id=28506531 - project allows creating pythonic bindings for your nim libraries pretty easily, which can be useful if you still want to write most of your toplevel code in python, but leverage nim's speed when it matters. If you want to make your nim code even more "pythonic" there is a https://github.com/Yardanico/nimpylib https://github.com/Yardanico/nimpylib, and for calling some python code from nim there is a https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy
nxpnsv 5y ago
Missing : on the first line of code. When speed mattes, people use libraries that are considerably faster than plain python. It’s these libraries that turned python so popular in data science. Giving them up maybe makes sense, but that mans a whole lot of learning and development to replace already pretty well established tools.
- rualca 5y ago
  > When speed mattes, people use libraries that are considerably faster than plain python. This. The general guideline has always been that Python is ideal for glue code and non-performance-critical code, and when performance became an issue then Python code would simply be used as glue code to invoke specialized libraries. Perhaps the most popular example of this approach is bumpy, which uses BLAS and LAPACK internally to handle linear algebra stuff. This Nim advertisement sounds awfully desperate with the way it resorts to what feels like a poorly assembled strawman, while giving absolutely nothing in return.
samuel 5y ago
For me the corollary of this post should be, try PyPy. You may get a 10x speedup for free.
- scoopertrooper 5y ago
  For me the corollary of this comment should be, try reading the post. You may learn that the author did get a 10x speedup for free, but it was still 3.3x slower than Nim.
  - samuel 5y ago
    I did, I don't think the snarkiness was necessary. My point, which apparently it wasn't evident enough, is that you can get the most of the benefits by doing nothing, just trying a different Python implementation, without the hassle of learning a niche language, as easy as it might be. BTW, if you take into account compilation times the difference is even meager, and in all fairness the PyPy warmup period should have had to be discounted.
  - stillblue 5y ago
    Do you feel better about yourself by being snarky to people on the internet? How does it work? I'm curious
mkl95 5y ago
I'm an experienced Python developer who also dabbles with some applications written in C++. In my experience Python is much faster when it comes to development speed but it's way more demanding when it comes to optimization. When you write C++, you kind of cheat because even code with high computational complexity is pretty fast. Whereas the equivalent code in Python will be awfully slow. So, while it's true that Python requires less development time, this statement can't be used generally. I have spent hours optimizing Python code when in C++ I would have just moved on to my next task.
nine_k 5y ago
Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, while being comparably ergonomic.
- stutonk 5y ago
  Apparently there's also a data processing library for Nim called Arraymancer[0] that's inspired by Numpy and PyTorch. It claims to be faster than both. [0] https://mratsim.github.io/Arraymancer/ https://mratsim.github.io/Arraymancer/
- teleforce 5y ago
  Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
  - leephillips 5y ago
    Julia is in the process of replacing many C and Fortran numerical libraries with pure Julia implementations, because they have similar performance.
  - dTal 5y ago
    >Julia libraries are still wrapping Fortran numerical library You say that, but Julia is rapidly acquiring native numerical libraries that outperform OpenBLAS: https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...
  - 5y ago
losvedir 5y ago
I notice that python has `rstrip` while `nim` doesn't. Python's `rstrip()` is allocating a whole new line there. Does the nim iterator skip over whitespace or something? Do those bits of code output the same thing? The presence or absence of the whitespace will affect the `total` count.
- jorams 5y ago
  This is explained a bit further into the article: > A nice feature of the lines function is that it automatically strips newline characters such as LF and CRLF so we no longer need to doline.rstrip().
  - losvedir 5y ago
    Oh, nice! Missed that. Thanks.
ignorem3 5y ago
So many python speed apologists. Yes, you can pour over your python code and given enough time and effort you can eek out another 25% improvement, but it's still much slower than the alternatives.
- adenozine 5y ago
  Python gets WAY too much work done in WAY too many fields to just handwave away "waaah, it's faster to use blah-lang"
- PartiallyTyped 5y ago
  Counter argument, how many times does one need to do data processing and what's the most expensive process in the equation? The answer for the latter is programmer time, and some things can be scaled easily using `joblib`, or `dask`. Now, it isn't as trivial as importing parallel iterators with rust and changing `.into_iter` to `.into_par_iter`, but still needs less time, and once it is done, I don't need to think about it again.
soundmasterj 5y ago
This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("orthocoronavirinae.fasta") as f: for line in f.readlines(): if not line.startswith('>'): line = line.rstrip() gc += line.count('C') + line.count('G') total += len(line) I didn't test Nim but the author reports Nim is 30x faster than his Python implementation, so mine would be about 3x slower than his Nim.
- epidemian 5y ago
  I think this is missing the point of the article. Yes, you can implement a faster Python version, but notice also: * This faster version is reading all the file into memory (except comment lines). The article mentions the data being 150MB, which should fit in memory, but for larger datasets, this approach would be unfeasible * The faster version is actually delegating a lot of work to Python's C internals by using text.count('G'). All the internal looping and comparisons is done in C, while on the original version, goes through Python So yes, you can definitely write faster Python by delegating most of the work to C. The point of the article is not about how to optimize Python, but about how given almost identical implementations in Python and Nim, Nim can outperform Python by 1 or 2 orders of magnitude without resorting to use C internals for basic things like looping or comparing characters.
  - soundmasterj 5y ago
    I didn't try to write optimized code, but idiomatic Python. Which also happens to be 10x faster. To make it streaming, take the second version and remove the readlines (directly iterate over f). Delegating work to Python's C internals is fine IMO because "batteries included" is a key feature of Python. "Nim outperforms unidiomatic Python that deliberately ignores key language features" is perhaps true, but less flashy of a headline. And to be honest, I mainly wrote this because the other top level Python implementations for this one were terrible at the time of the post.
mark_l_watson 5y ago
Nice writeup, glad the author has a language and environment that they like. Python has never been one of my favorite languages, but easy support in Google Colab, AWS SageMaker, etc. as well as most of my professional deep learning work using TensorFLow + Keras, it makes Python a go-to language for me. If you want a Lisp syntax on top of Python, you can try Hy (and get a free copy of my Hy book at https://leanpub.com/hy-lisp-python https://leanpub.com/hy-lisp-python by setting the price to $0.00). That said, for unpaid experiments I like Julia + Flux, which also solves the author's preference to avoid slow programming languages. Julia is really a nice language but no one has ever paid me to use it.
adsharma 5y ago
Why not do both? cat test.py | py2many --nim=1 - http://dpaste.com//5ALVT7MK4
Mikeb85 5y ago
I'll probably always dislike Python. I do like Nim, but for strictly data processing I'd take R over either in a heartbeat. R has too many libraries, its semantics are perfect for data processing, RStudio is too nice and while pure R is slow as shit, in practice it's fast because it's basically a scripting language for a bunch of C and Fortran bits that are doing the real work. If I were writing something from scratch that dealt with data, I would probably use Nim though. It's super easy to write something fast in and is more pleasant than pretty much any other compiled language.

256 points by benjamin-lee 5y ago

While I trust the author on this, I don’t think DNA datasets and string analysis was a great example. One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do. Not only does it compress your dataset (2 bits vs 16 bits), it allows absurdly faster numerical libraries to be used in lieu of string methods. There’s no real point in showing off that a compiled language is faster at doing something the slow way…

benjamin-lee 5y ago
You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucleotide [2]. One could get creative with a 4 bit encoding and still end up saving space (assuming you don't care about the distinction between upper versus lowercase characters in your sequence [3]). Or, if you know in advance your sequence is unambiguous (unlikely in DNA sequencing-derived data) you could use the 2 bit encoding. When dealing with short nucleotide sequences, another approach is to encode the sequence as an integer. I would love to see a library—Python, Nim, or otherwise—that made using the most efficient encoding for a sequence transparent to the developer. [1] https://github.com/biocore/scikit-bio/blob/b470a55a8dfd054ae109d813ef2fd28a246dab6e/skbio/sequence/_genetic_code.py#L536 https://github.com/biocore/scikit-bio/blob/b470a55a8dfd054ae... [2] https://en.wikipedia.org/wiki/Nucleic_acid_notation https://en.wikipedia.org/wiki/Nucleic_acid_notation [3] https://bioinformatics.stackexchange.com/questions/225/uppercase-vs-lowercase-letters-in-reference-genome https://bioinformatics.stackexchange.com/questions/225/upper...
- liamwestray 5y ago
  Yeah, this is why my comment led with “I trust the author”… I’m surprised you need the full 4 bits to deal with ambiguous bases, but it probably makes sense at some lower level I don’t understand.
  - benjamin-lee 5y ago
    This is because there's four bases and each can either be included or excluded from a given combination. So there are 4*2 = 16 combinations each of which with their own letter. In all honesty, these are pretty rarely used in practice these days except for N (any base) although they do sometimes show up when representing consensus sequences.
    - pdimitar 5y ago
      What do you mean that each base can be included or excluded? Isn't only one extra value needed? Sort of like nil?
      - ac29 5y ago
        Because there are notations for any combination of bases. There's a way to indicate "C or G", "A or T", "C, G, or A", etc.
        pdimitar 5y ago
        Oh. So what's the grand total of all possible permutations of single and multiple (connected with an "or") values? I'll also read through your links, thanks for posting them.
jjtheblunt 5y ago
Aren't the reads emitting a set of size greater than 4 bases per position, with a wildcard or "?" perhaps one option? (As in GATTACA might be read as is, but might be read as GAT?ACA.) Still that's a minimal of 3 bits versus much longer. [Edit : i see another commenter with the same observation, more thoroughly explained! ]

joshu 5y ago

does nim have anything like scikit yet?

styluss 5y ago
Closest I can find is https://nimble.directory/pkg/science https://nimble.directory/pkg/science

zmmmmm 5y ago

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a whole suite of languages that easily cruise to a 10x performance win over Python. And that isn't counting multicore - if you count that you quickly get to a 100x improvement. Personally I use Groovy for much of what I do for similar reasons (which is somewhat unusual) but its just a placeholder for "use anything except python".

amyjess 5y ago
> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via direct SQL or a REST API to whatever service owns the database). All these tasks were heavily network-bound. The amount of time the CPU was doing any work was minuscule compared to the amount of time it was waiting to get data back from the network. I doubt Nim or any other language would have been a significant performance improvement in this case. For what it's worth, that made these scripts excellent candidates for multithreading. I'd run them with 20+ threads, and it was glorious. At first I did multiprocessing, because of all the GIL horror stories, but multiprocessing made it very difficult to cache data, so eventually I said "well, all this is network-bound so the GIL doesn't even apply" and switched over to multiprocessing.dummy (which implements pools using the same API as multiprocessing but with threads instead of processes), and I never looked back. Edit: For what it's worth, Nim sounds like a really cool language, and it's right up my alley in several ways, I just don't think Python is particularly slow at network-bound tasks that use very little CPU.
- otherme123 5y ago
  You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again). But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.

TekMol 5y ago

TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.

ur-whale 5y ago
Python is indeed slow, but why would anyone ever use php as a comparison point? And for data processing of all things ... Php is also very slow, on top of being many other kinds of unpleasant and broken.
- pjmlp 5y ago
  PHP has a proper JIT compiler on its reference implementation.
  - ur-whale 5y ago
    > PHP has a proper JIT compiler on its reference implementation. That doesn't make it fast at all. Just faster than if it wasn't jitted. And that certainly does not eradicate the vast ocean of other problems php has, the first of which being that is was never, ever "though out" and instead grew like a cancerous mushroom.
- DeathArrow 5y ago
  I had a nightmare where I was writing an operating system in PHP, which ran in hypervisor written in PHP, which ran in a virtual machine written in PHP, which in a browser written in PHP, which ran in an operating system written in PHP...
fctorial 5y ago
> that is the achilles heel of Python I think that is pip.

xiaodai 5y ago

How does it compare to Julia? Anyone with experience in both Nim and Julia?

agons 5y ago
I would have thought Cython would be the closest analogue for comparison.
Sanguinaire 5y ago
I have tinkered with both. Basically if you need to write a FORTRAN application but don't want to use FORTRAN, Julia is the correct choice. If ease of use is more important than having a diverse collection of pre-existing numerical libraries to play with, I'd go with Nim. If Nim had cloud SDKs I would use it as my default language for pretty much everything.

ur-whale 5y ago

what's up we the all the shouting in the code ?

choneone 5y ago
It's because the `font-feature-settings` of the main font leak to the code font. The feature 'case' turns the code all uppercase.
- ghostly_s 5y ago
  Ha, I thought the author was intentionally writing the python in all-caps to highlight the similarity in syntax to the alternative language (not being familiar with Nim, I figured it must be conventional there). I was going to post a comment expressing my surprise that was even legal.

blondin 5y ago

i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.

shoo 5y ago
that can be a good approach where the computation maps cleanly onto some C or fortran code prepared earlier and wrapped as a numpy or scipy primitive. but sometimes expressing what you want to do as a bunch of numpy operations becomes a lot harder to read and perhaps also slower than if you just directly wrote the raw loops over some arrays in C. in cases where what you want to do doesn't exactly fit standard operations, cython can be pretty nice. e.g. 200x -- 1000x speedups for translating C-oriented number crunching code from python to cython. but if you do want performance, you have to think about it while writing the code (avoid needlessly allocating memory in tight loops, data-oriented programming with simple arrays, statically type all of your variables, ...).

ellimilial 5y ago

A context might be useful. From what I gather, the author is a researcher in bioinformatics related field. This may indicate that they tend to work either alone or in a relatively small group. The domain is small scope data processing/manipulation, research/exploratory code, ,likely short-lived or even one-off. The progress in this context will possibly be governed by sheer processing speed (e.g. it’s unlikely anyone will delve deep into the code, a lot of iterations to ‘just get it done’ instead of testing etc.). If this is more or less correct, the point that Nim might be more useful than Python for the author sounds very sensible to me. It’s a nice spot between command line tools and more functionality-loaded languages.

benjamin-lee 5y ago
Author here. This is spot on. The majority of the code I write is either piping data around to existing tools using shell scripting and Snakemake or writing the data processing code myself when there isn't a tool that does what I need. Usually, I'm working alone or with a few other computational biologists. Many of my scripts are one-off but they have the distinct tendency of growing in complexity and scope if they are useful. That's one of the big advantages with Nim in my mind: you can write a quick and dirty script and have it be pretty fast and then go back later and optimize it to a few percent of C without having to rewrite your code in another language. In this sense, it's quite like Julia (another really good language).

DeathArrow 5y ago

If anyone is interested to see how Nim fares against some other programming languages, here are some benchmarks: https://github.com/kostya/benchmarks https://github.com/kostya/benchmarks

dilawar 5y ago

NIM also has a very good JavaScript backend. You can generate both C and JavaScript code from a nim program. Last time I used it, I liked it but didn't use it long enough to have a strong opinion.

brabel 5y ago

> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes ridiculous amounts of RAM even for hello world. Did the author post the data set they used for the examples? Would be nice to try it out on a few languages to see how fast that can compile and run on a mature language like Common Lisp (which is just as easy to write) or even node.js.

benjamin-lee 5y ago
I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Severe%20acute%20respiratory%20syndrome%20coronavirus%202,%20taxid:2697049 https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...
- brabel 5y ago
  I clicked on the big Download button and selected "all records", it downloaded over 3.5GB before I gave up... which file exactly should I use??
  - benjamin-lee 5y ago
    I'm sorry, I completely forgot that the file I used was from six months ago when I wrote the blog post (and then promptly forgot to publish it). In the last half year, the number of coronavirus sequences has increased dramatically. One thing that you could do to drop the file size down is to filter for only complete and unambiguous sequences, which drops the number down from 1.6 million to ~100k [1]. Alternatively, the exact file I used for the post is available for one week here with MD5 sum 3c33c3c4c2610f650c779291668450c9 [2]. Anyone who wants the file is free to reach out to me directly (email is on site). [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Severe%20acute%20respiratory%20syndrome%20coronavirus%202,%20taxid:2697049&QualNum_i=0&Completeness_s=complete https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType... [2] https://file.io/nUNc7cG5i8gj https://file.io/nUNc7cG5i8gj

piqufoh 5y ago

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish before the kettle has boiled.

elcritch 5y ago
That’s where Nim can shine. For simple scripts both Python and Nim are about as easy to write. But the Nim version usually runs a lot faster. Static types help for basic data munging when you haven’t used a script for months to get up to speed and make tweaks.
- jimbob45 5y ago
  Sadly, I think you’re spot-on about Nim’s future as the realization of the alternative timeline where Python didn’t make several stupid design choices (e.g. the GIL, Python 3). It’s a shame because I think Nim has some neat features that allow it to present as a serious competitor to Rust but it will ultimately have to compete against Python instead to secure its niche.
  - elcritch 5y ago
    Oh yes, Nim definitely feels like an alternate reality where Python 2 became static and dumped some poor design choices. Well I believe there's room between Rust and Python where Nim can grow. It made the TIOBE top 50 lately even. Likely it can eat enough market share from the edges of both Rust & Python to become more well known (more libs, tools, etc). Rust is fantastic but tedious to program (to me at least) and it's community focuses on more formal type traits, etc making "scripting" trickier. Python is great for a mix of quick scripts, web dev, and data science but it's slow enough (and getting complex enough!) for many to want something faster and more stable yet still easy to write. Nim lives in between them and is more enjoyable to write than either for many. Also, Nim _could_ add Rust as a backend target and be relevant even if Rust displaces C/C++. ;) Nim is also great for embedded systems too! I've been using it a fair bit and it's really nice [1]. There's a lot of room to grow in that field. 1: https://github.com/elcritch/nesper https://github.com/elcritch/nesper

oxfordmale 5y ago

The author makes a fair point, however, that is a rather non optimal implementation in Python. You likely could use chunked Pandas to speed up or the code, or at least replace some of the for loops with a list comprehension syntax. However, in any case I would never replace Python with Nim as it is too niche of a language and you would struggle with recruiting. I could consider Julia if it's popularity keeps growing. That is the ultimate challenge of a language. It either needs a large backer (Go and Google) or be so good, it gets a natural market adaptation(Julia). As a manager I am reluctant to adapt yet another language unless there is a healthy job market for it.

anyfactor 5y ago
There are objectively better niche solutions for niche problems out there. But we pickup things that can be applied to solve a number of different problem, that are more versatile and has a community behind them.
pdimitar 5y ago
There's a class of technologies falling under what I'd call "most of your engineers would pick that in a weekend". Not all technologies require the full cycle and the normal risk management.
Daishiman 5y ago
Agreed. I am certainly inclined to believe that Nim is a better language than Python, but it's not so much better thlo justify moving off of the ecosystem.

cycomanic 5y ago

One pitfall that I ran into when trying out Nim for scientific computing is that Nim follows more a computer science convention than mathematics convention for exponentiation and negation operators. That is in Nim -2^3=(-2)^3 unlike more scientific computing oriented languages where -2^3=-(2^3). To someone like myself who mainly does scientific work this was quite unexpected and it causes a surprising amount of mental overhead to avoid mistakes. I did like Nim quite a bit otherwise, but essentially found that I was missing some important numerical libraries so did not continue using it.

jb1991 5y ago
To be honest, regardless of the behavior of the language, I would be wrapping things in parentheses to make it explicit anyway.
leephillips 5y ago
Although Nim using weird order of operations is unfortunate, your example is not well chosen. Replace the exponent 3 with an even integer and your point will be clear.
- leephillips 5y ago
  Maybe not that weird, although not the most common syntax. In Nim 0 - 2^2 evaluates to -4, while -2^2 evaluates to +4. So in -2, the - is treated as a unary minus and binds tightly to the 2. It would be bad if Nim were doing + or - before ^, or before *. I have a feeling some other languages treat the unary minus the same way, but don’t know of any examples off hand.
  - cycomanic 5y ago
    Yes when I asked a question about this on their gitter channel, people brought up several languages I think bc is one of them for example. I understand the reasoning I still find it extremely unintuitive personally.
tzs 5y ago

DeathArrow 5y ago

Yes, Python is painfully slow, but it shouldn't matter. If you are using Python where performance, speed, correctness, testability, maintainability matters, you are doing it wrong. Python is good where speed of development matters, where you write throw-away code testing some ideas and you want to do it fast, where you write glue code, for prototypes, for small code bases. Once you are getting outside of that area, you better should use a language more suited for the task. As for myself, even if I can use Python in some cases, I can churn C# code almost as fast so I prefer doing it that way in case I want to grow the code later or use it somewhere else. Being lazy, I dislike rewriting code.

santiagobasulto 5y ago

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edit: my code is not perfect (I’m typing from my phone, I’m surprised I could even match parentheses). My point is: this is a highly I/O bound program. The implementation matters. With the correct implementation there shouldn’t be much difference between the languages.

kiidev 5y ago
Isn't that going to be even slower, since a 156mb file would probably make gc_lines a very big tuple before summing it?
- henbruas 5y ago
  gc_lines isn't a tuple here, it's a generator expression. It will be lazily evaluated.
- makapuf 5y ago
  You could do gc=sum(1 for _ on lines if ...) or use Counter
deleted 5y ago
[deleted]
ellimilial 5y ago
Isn’t this counting the lines with any G/C in them vs the total number of G/C literals?
_dain_ 5y ago
>total = len(list(gc_lines)) That won't work properly; you've already exhausted the gc_lines generator in the previous line.

pella 5y ago

related: Biofast benchmark ( Nim, Julia, Go, Pypy, C, Crystal, .. ) "Benchmarking programming languages/implementations for common tasks in Bioinformatics" https://github.com/lh3/biofast#fqcnt https://github.com/lh3/biofast#fqcnt https://lh3.github.io/2020/05/17/fast-high-level-programming-languages https://lh3.github.io/2020/05/17/fast-high-level-programming... HN: https://news.ycombinator.com/item?id=23229657 https://news.ycombinator.com/item?id=23229657

DeathArrow 5y ago

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people, it would mean we would have to write many things ourselves, which in other languages we can take for granted as they have libraries for almost anything. So having a nice, performant and good language is just a small part of achieving your goals. You also need the people and the ecosystem. Go, Rust, Kotlin, Swift and even Julia have the luck of having some industry heavyweights behind them, pushing the ecosystem and contributing with money and developers. Nim has only a bunch of passionate people behind it.

cnmlp 5y ago
In the case of Python, the "industry heavyweights" do very little and have a negative influence now. Why the phrase "only a bunch of passionate people"? This is how software gets written, parasitical corporations and their unproductive developers who are installed in existing OSS projects come later and mainly associate themselves with the result (speaking of Python again).
dejj 5y ago
> How would I convince my employer Turntables: if you were an employer, what would convince you to use Nim? Hiring for Nim skills can be a signal that a company has people who learn languages beyond the run-of-the-mill ones. A bunch of passionate people you might say. That would make the company promising to work for.
- WJW 5y ago
  Interestingly this was exactly the place Python was in back in the day: niche enough that anyone knowing it must've learned it because they thought it was cool rather than because it would get them a job. Those days are long gone of course.
egwor 5y ago

paulluuk 5y ago

As a Data Engineer, I mainly use Python in conjunction with PySpark. So essentially I just use Python as an easy-to-read wrapper around Spark, and it works great when working together with Data Scientists, who are mostly used to Pandas, Keras, Tensorflow, etc. In my use case, I don't really see how Nim would make my life easier right now.

haxscramper 5y ago

Another nim & python thread that has not been mentioned yet here https://news.ycombinator.com/item?id=28506531 https://news.ycombinator.com/item?id=28506531 - project allows creating pythonic bindings for your nim libraries pretty easily, which can be useful if you still want to write most of your toplevel code in python, but leverage nim's speed when it matters. If you want to make your nim code even more "pythonic" there is a https://github.com/Yardanico/nimpylib https://github.com/Yardanico/nimpylib, and for calling some python code from nim there is a https://github.com/yglukhov/nimpy https://github.com/yglukhov/nimpy

nxpnsv 5y ago

Missing : on the first line of code. When speed mattes, people use libraries that are considerably faster than plain python. It’s these libraries that turned python so popular in data science. Giving them up maybe makes sense, but that mans a whole lot of learning and development to replace already pretty well established tools.

rualca 5y ago
> When speed mattes, people use libraries that are considerably faster than plain python. This. The general guideline has always been that Python is ideal for glue code and non-performance-critical code, and when performance became an issue then Python code would simply be used as glue code to invoke specialized libraries. Perhaps the most popular example of this approach is bumpy, which uses BLAS and LAPACK internally to handle linear algebra stuff. This Nim advertisement sounds awfully desperate with the way it resorts to what feels like a poorly assembled strawman, while giving absolutely nothing in return.

samuel 5y ago

For me the corollary of this post should be, try PyPy. You may get a 10x speedup for free.

scoopertrooper 5y ago
For me the corollary of this comment should be, try reading the post. You may learn that the author did get a 10x speedup for free, but it was still 3.3x slower than Nim.
- samuel 5y ago
  I did, I don't think the snarkiness was necessary. My point, which apparently it wasn't evident enough, is that you can get the most of the benefits by doing nothing, just trying a different Python implementation, without the hassle of learning a niche language, as easy as it might be. BTW, if you take into account compilation times the difference is even meager, and in all fairness the PyPy warmup period should have had to be discounted.
- stillblue 5y ago
  Do you feel better about yourself by being snarky to people on the internet? How does it work? I'm curious

mkl95 5y ago

I'm an experienced Python developer who also dabbles with some applications written in C++. In my experience Python is much faster when it comes to development speed but it's way more demanding when it comes to optimization. When you write C++, you kind of cheat because even code with high computational complexity is pretty fast. Whereas the equivalent code in Python will be awfully slow. So, while it's true that Python requires less development time, this statement can't be used generally. I have spent hours optimizing Python code when in C++ I would have just moved on to my next task.

nine_k 5y ago

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, while being comparably ergonomic.

stutonk 5y ago
Apparently there's also a data processing library for Nim called Arraymancer[0] that's inspired by Numpy and PyTorch. It claims to be faster than both. [0] https://mratsim.github.io/Arraymancer/ https://mratsim.github.io/Arraymancer/
teleforce 5y ago
Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
- leephillips 5y ago
  Julia is in the process of replacing many C and Fortran numerical libraries with pure Julia implementations, because they have similar performance.
- dTal 5y ago
  >Julia libraries are still wrapping Fortran numerical library You say that, but Julia is rapidly acquiring native numerical libraries that outperform OpenBLAS: https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is...
- 5y ago

losvedir 5y ago

I notice that python has `rstrip` while `nim` doesn't. Python's `rstrip()` is allocating a whole new line there. Does the nim iterator skip over whitespace or something? Do those bits of code output the same thing? The presence or absence of the whitespace will affect the `total` count.

jorams 5y ago
This is explained a bit further into the article: > A nice feature of the lines function is that it automatically strips newline characters such as LF and CRLF so we no longer need to doline.rstrip().
- losvedir 5y ago
  Oh, nice! Missed that. Thanks.

ignorem3 5y ago

So many python speed apologists. Yes, you can pour over your python code and given enough time and effort you can eek out another 25% improvement, but it's still much slower than the alternatives.

adenozine 5y ago
Python gets WAY too much work done in WAY too many fields to just handwave away "waaah, it's faster to use blah-lang"
PartiallyTyped 5y ago
Counter argument, how many times does one need to do data processing and what's the most expensive process in the equation? The answer for the latter is programmer time, and some things can be scaled easily using `joblib`, or `dask`. Now, it isn't as trivial as importing parallel iterators with rust and changing `.into_iter` to `.into_par_iter`, but still needs less time, and once it is done, I don't need to think about it again.

soundmasterj 5y ago

This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("orthocoronavirinae.fasta") as f: for line in f.readlines(): if not line.startswith('>'): line = line.rstrip() gc += line.count('C') + line.count('G') total += len(line) I didn't test Nim but the author reports Nim is 30x faster than his Python implementation, so mine would be about 3x slower than his Nim.

epidemian 5y ago
I think this is missing the point of the article. Yes, you can implement a faster Python version, but notice also: * This faster version is reading all the file into memory (except comment lines). The article mentions the data being 150MB, which should fit in memory, but for larger datasets, this approach would be unfeasible * The faster version is actually delegating a lot of work to Python's C internals by using text.count('G'). All the internal looping and comparisons is done in C, while on the original version, goes through Python So yes, you can definitely write faster Python by delegating most of the work to C. The point of the article is not about how to optimize Python, but about how given almost identical implementations in Python and Nim, Nim can outperform Python by 1 or 2 orders of magnitude without resorting to use C internals for basic things like looping or comparing characters.
- soundmasterj 5y ago
  I didn't try to write optimized code, but idiomatic Python. Which also happens to be 10x faster. To make it streaming, take the second version and remove the readlines (directly iterate over f). Delegating work to Python's C internals is fine IMO because "batteries included" is a key feature of Python. "Nim outperforms unidiomatic Python that deliberately ignores key language features" is perhaps true, but less flashy of a headline. And to be honest, I mainly wrote this because the other top level Python implementations for this one were terrible at the time of the post.

mark_l_watson 5y ago

Nice writeup, glad the author has a language and environment that they like. Python has never been one of my favorite languages, but easy support in Google Colab, AWS SageMaker, etc. as well as most of my professional deep learning work using TensorFLow + Keras, it makes Python a go-to language for me. If you want a Lisp syntax on top of Python, you can try Hy (and get a free copy of my Hy book at https://leanpub.com/hy-lisp-python https://leanpub.com/hy-lisp-python by setting the price to $0.00). That said, for unpaid experiments I like Julia + Flux, which also solves the author's preference to avoid slow programming languages. Julia is really a nice language but no one has ever paid me to use it.

adsharma 5y ago

Why not do both? cat test.py | py2many --nim=1 - http://dpaste.com//5ALVT7MK4

Mikeb85 5y ago

I'll probably always dislike Python. I do like Nim, but for strictly data processing I'd take R over either in a heartbeat. R has too many libraries, its semantics are perfect for data processing, RStudio is too nice and while pure R is slow as shit, in practice it's fast because it's basically a scripting language for a bunch of C and Fortran bits that are doing the real work. If I were writing something from scratch that dealt with data, I would probably use Nim though. It's super easy to write something fast in and is more pleasant than pretty much any other compiled language.