4 ms·
Congratulations, very cool project. I'm interested in learning about how federated data source query engines like this handle query planning and optimization.
by martypitt 4y ago
Congratulations, very cool project.
I'm interested in learning about how federated data source query engines like this handle query planning and optimization.
If I wanted to study your implementation for hints where would I start?
- cube2222 4y agoThanks! I'll start by saying that OctoSQL is single-machine-only, as I'm not sure what exactly you meant with "federated". I'd recommend starting by going with a debugger through the execution of the root function in root.go, as that calls out to all the macro transformations. Then, you can take a look at the optimizer (optimizer directory) and the Postgres plugin source[0], as an example of a plugin that is able to push down predicates to the underlying database. As well as the Typecheck (logical -> physical) and Materialize (physical -> execution) transformations. I'm planning to write a few technical documents about the implementation soon, while writing some actual usage documentation as well. [0]: https://github.com/cube2222/octosql-plugin-postgres https://github.com/cube2222/octosql-plugin-postgres
- martypitt 4y agoGreat, thanks!
- whoevercares 4y agoJoin AWS and study Athena Federated query feature