5 ms·

> Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a trad

> Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?

apavlo 3y ago
LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems. The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.
- zinclozenge 3y ago
  Do you know if there is anybody taking this approach? Alternatively, what would you consider to be the current state of the art when it comes to query optimizers?
- erichocean 3y ago
  One way to do that might be to merge the query optimizer and executor together and then execute them simultaneously in a self-adjusting code framework.[0] The relevant optimizer variables would then be set initially (using whatever mechanism already exists e.g. heuristics, stats, etc.) and then while the query executes, you continually update those optimizer variables. The self-adjusting code property would cause the query to self-adjust as it ran, while still producing the same end result. I'm sure there are details I'm missing here, but I do believe the general approach could do implemented in LingoDB (or similar) as a compiler transformation, so the actually cost-to-develop this approach would remain tractable. [0] I suspect you'd need to model the whole thing as a streaming network so that you can update the network parts as you go, effectively re-wiring the streams while not invalidating earlier results. So SAC+logic to map from one stream architecture to another. JITs that support de-optimization have to do something similar (with a lot of careful upfront design), so that's at least plausible.
zinclozenge 3y ago
There's also mutable that compiles to WASM and lets it get JITed by v8 https://github.com/mutable-org/mutable https://github.com/mutable-org/mutable.
- erichocean 3y ago
  Wow, good find! Thanks for the link.
- erichocean 3y ago
  I've now read through the mutable papers, and I think it's a great idea. A similar thing happened in physically-based rendering, with the publication of the PBTR series of books. As a result of that effort, a lot of really solid research improving various aspects of rendering occurred. Extremely influential long-term. mutable could have a similar kind of experience if presented that way to the public.

apavlo 3y ago

LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems. The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.

zinclozenge 3y ago
Do you know if there is anybody taking this approach? Alternatively, what would you consider to be the current state of the art when it comes to query optimizers?
erichocean 3y ago
One way to do that might be to merge the query optimizer and executor together and then execute them simultaneously in a self-adjusting code framework.[0] The relevant optimizer variables would then be set initially (using whatever mechanism already exists e.g. heuristics, stats, etc.) and then while the query executes, you continually update those optimizer variables. The self-adjusting code property would cause the query to self-adjust as it ran, while still producing the same end result. I'm sure there are details I'm missing here, but I do believe the general approach could do implemented in LingoDB (or similar) as a compiler transformation, so the actually cost-to-develop this approach would remain tractable. [0] I suspect you'd need to model the whole thing as a streaming network so that you can update the network parts as you go, effectively re-wiring the streams while not invalidating earlier results. So SAC+logic to map from one stream architecture to another. JITs that support de-optimization have to do something similar (with a lot of careful upfront design), so that's at least plausible.

zinclozenge 3y ago

There's also mutable that compiles to WASM and lets it get JITed by v8 https://github.com/mutable-org/mutable https://github.com/mutable-org/mutable.

erichocean 3y ago
Wow, good find! Thanks for the link.
erichocean 3y ago
I've now read through the mutable papers, and I think it's a great idea. A similar thing happened in physically-based rendering, with the publication of the PBTR series of books. As a result of that effort, a lot of really solid research improving various aspects of rendering occurred. Extremely influential long-term. mutable could have a similar kind of experience if presented that way to the public.