4 ms·
I think a central element of the article is its focus on the needs of genomics researchers. In this field there are many tasks that are highly parallel but not
by bscphil 4y ago
I think a central element of the article is its focus on the needs of genomics researchers. In this field there are many tasks that are highly parallel but not embarrassingly parallel. Ideally you want to scale out to as many processes as possible, but some basic IPC is mandatory.
Though not in the HPC field myself, I've been in a position of helping researchers working with an HPC cluster. On this hardware something like 16 cores per node was typical, so you run out of scaling room quickly using traditional threading libraries. And you really want to scale beyond this - jobs could take weeks or months otherwise. This means using MPI because that's all the HPC supported. This was a massive pain, because MPI is complicated in ways that are unfamiliar to many researchers, most of whom have no low-level language experience at all.
What the article calls for are HPC techniques that move communication to a lower level API, rather than exposing them to the programmer / end-user. I think that's exactly the right idea for genomics.
Disclaimer: I'm neither an expert in HPC nor genomics.