3 ms·

> Independent jobs go beyond what is usually understood as embarrassingly parallel. Independent = embarrassingly parallel, i.e. CPU bound. You want to apply so

> Independent jobs go beyond what is usually understood as embarrassingly parallel. Independent = embarrassingly parallel, i.e. CPU bound. You want to apply some f on x to get y ==> y = f(x). You have many cases of x, all with f applied independently to get all the independent y outputs. There is no difference between "independent" and "embarrassingly parallel" in this case. > In a typical bioinformatics workflow, you download the data, process it locally for hours, and send the results back. And I'm saying that you do not need to be waiting for hours, if you do the data storage and data processing using sane and modern methods (i.e. not with CERN's ROOT tool). This is not theory BTW -- I've been doing it for the last month. This is a very real, evidence based observation. --- I was writing out a bunch of other point by point replies here, but I get the sense that it's more likely going to cause you to dig in to your currently held beliefs rather than open you up to the magic methods used by data engineering teams, so I decided to give up and go eat some pizza. I enjoy pizza.

jltsiren 4y ago
I think you are missing the context here. We are talking about the kind of bioinformatics tools that establish new popular file formats. They are typically developed by researchers rather than software engineers. These tools are intended for end users to install and run on a wide variety of systems. And they are often released before the first high-profile papers on the topic are published. At that point, it's rare to find a software engineer who is familiar enough with the topic to be able to contribute. Waiting a few hours for the results is not a big deal with these tools, because you are probably going to submit a large number of jobs anyway. Getting the full results will likely take days, regardless of whether you are using naive or modern methods. Because you are probably not allowed to use unlimited compute, your primary constraint is throughput rather than latency. Naive file formats survive, because tools that solve independent problems locally provide similar throughput to state-of-the-art distributed systems. Bioinformatics, and particularly the subfield that focuses on genomics, is noteworthy because it relies more on freely available open source software than most other academic fields. That may be because the state of the art is changing so rapidly. Companies sometimes take over old well-established problems. By providing services instead of tools, they can use whatever technology they prefer internally. But you rarely see these companies attacking state-of-the-art problems.

jltsiren 4y ago

I think you are missing the context here. We are talking about the kind of bioinformatics tools that establish new popular file formats. They are typically developed by researchers rather than software engineers. These tools are intended for end users to install and run on a wide variety of systems. And they are often released before the first high-profile papers on the topic are published. At that point, it's rare to find a software engineer who is familiar enough with the topic to be able to contribute. Waiting a few hours for the results is not a big deal with these tools, because you are probably going to submit a large number of jobs anyway. Getting the full results will likely take days, regardless of whether you are using naive or modern methods. Because you are probably not allowed to use unlimited compute, your primary constraint is throughput rather than latency. Naive file formats survive, because tools that solve independent problems locally provide similar throughput to state-of-the-art distributed systems. Bioinformatics, and particularly the subfield that focuses on genomics, is noteworthy because it relies more on freely available open source software than most other academic fields. That may be because the state of the art is changing so rapidly. Companies sometimes take over old well-established problems. By providing services instead of tools, they can use whatever technology they prefer internally. But you rarely see these companies attacking state-of-the-art problems.