4 ms·

Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the result

Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the results. How can you do that in an AOT language? Do you just express your functions at a higher level. Above the layer of containers?

blaenk 7y ago
Consider Just-in-Time compilation which is very much used in AOT languages [0]. That could be one avenue, as long as the compiled code works on each machine (common instruction set), or maybe something slightly higher-level like LLVM IR can be produced and transmitted and then compiled on the driver/executor. Another approach could simply be to transmit the function source to the driver, compile it, then distribute it to the executors assuming they're the same architecture, or compiling it on executors if not. There are certainly options. [0]: For example see https://llvm.org/docs/tutorial/BuildingAJIT3.html https://llvm.org/docs/tutorial/BuildingAJIT3.html