The linked papers demonstrate the performance gains in a specific arena: static compilation around a loop instead of mere data parallelism of the iterations. Fo

The linked papers demonstrate the performance gains in a specific arena: static compilation around a loop instead of mere data parallelism of the iterations. For example: for i in range(num_iterations): some_function(i) A library like Spark can execute the function in parallel, but must return control to the host language for each iteration. That leads to a restart of the distributed environment. HPAT is a compiler. The function is executed in parallel, just like in Spark, but the distributed environment doesn't return control the host language between iterations. Instead, the loop is run in its entirety in the distributed environment. Spark's driver must farm work to executors during each iteration. HPAT removes that runtime overhead.