3 ms·

> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is t

> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? But I have a problem with talking real life application and using that to claim > it will be impossible for pure Python to beat pure Nim at raw performance Because it maybe be true. I didn't try it but there are many things that can be optimized in the python code example but away from that. In real life application in scientific computing I don't think anyone wouldn't use numpy to deal with that which will make things much better. Also the power of python in data analysis and scientific computing is the ecosystem and community. This will be very hard to beat. And there are more mature alternatives like Julia. Edit: The author code for reading the data reading is creating a new file object for each iteration. I would guess that in nim this would be a similar problem but I am not sure how it actually work or if has the same effect. But anyway you don't do this in real life application with python. Also it would be nice to use a list comprehension to count the occurrences of 'C' and 'G' in each line.

aargh_aargh 2y ago
> > so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. > Is this the scientific version of "rich people problems"? It's more like programming to solve many people's recurring problem and selling that as a product vs. programming to solve one man's many different ad hoc problems.
benjamin-lee 2y ago
>> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? > Is this the scientific version of "rich people problems"? Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not. However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.

aargh_aargh 2y ago

> > so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. > Is this the scientific version of "rich people problems"? It's more like programming to solve many people's recurring problem and selling that as a product vs. programming to solve one man's many different ad hoc problems.

benjamin-lee 2y ago

>> so it’s usually not worth spending a ton of time optimizing single-threaded performance for a single experiment when I can just perform a big MapReduce. Is this the scientific version of "rich people problems"? > Is this the scientific version of "rich people problems"? Author here. Yes, most certainly. In fact, it was one of the things that drew me towards the NIH for my PhD. My overall point in the post was to show that a somewhat naive Python implementation and a much faster Nim version have a small Levenshtein distance. For many people in bioinformatics who don't have a background in software engineering (that would be a significant fraction, if not a majority of them), this could be a huge boon. Combined with the fact that most bioinformatics researchers don't have the privilege of the world's largest biomedical HPC cluster at their disposal, I still think Nim would be great drop-in replacement for quick single-threaded line-oriented string processing. For numerical stuff, probably not. However, I am mostly writing in Rust these days for longer term projects that require threading and good ecosystem support. Perhaps I'll write a follow-up retrospective on Rust versus Nim in this area.