My lab in computational structural biology has had to develop software to face similar challenges using HPC resources. One PhD student in our lab probably spent

My lab in computational structural biology has had to develop software to face similar challenges using HPC resources. One PhD student in our lab probably spent a year writing a client-server application to execute hundreds of compute jobs using the systems job scheduler (instead of working on actual problems in biology). Due to restrictions in the HPC environment, the code had to be fault-tolerant to clients and server being killed due to wall clock exceeding 24 hours. As you might expect, the student had no experience with distributed computing, so the "software" is actually thousand of lines of incomprehensible BASH and C code. I do like the sound of an open-source framework for HPC though, I run computations across dozens of national supercomputers and they're all very different.