4 ms·
That's pretty misleading. All systems I am aware of essentially try to compile query-specific code and avoid re-compiling runtime code that doesn't vary between
by mpala 9y ago
That's pretty misleading. All systems I am aware of essentially try to compile query-specific code and avoid re-compiling runtime code that doesn't vary between queries (e.g. if you look at the HyPeR paper, that's exactly what they describe).
Compiling everything is questionable. There's not much point re-compiling runtime code or code outside of the hot path, it's expensive and doesn't bring any benefit.
E.g. things that aren't beneficial to compile per query include:
* Loops over a column of the same datatype, with no query specific branches (e.g. decoding a column of integers)
* Other static code, e.g. some hash table operations
* Outer loops that don't execute frequently
* Rarely executed code, e.g. error handling.
There are two general designs that let you compile only the necessary things. 1) the runtime calls into compiled code for hot loops vs 2) the compiled code drives the query and calls into the runtime. A lot of the systems you mentioned do the second, but Impala does the first, which seems to be the source of some misunderstanding. Also there were some cases where hot loops in earlier versions of Impala weren't compiled, but that's changed - generally we try to ensure that all hot loops are compiled.
I think generally the optimal design wouldn't be complete query compilation, but rather something more like a traditional JIT that selectively compiles parts of the query.
Source: work on Impala's query compilation