4 ms·
I'm wondering if it would be worth skipping the SQL layer and compiling straight to the execution plan level. This would make PRQL tightly coupled to a database
by haspok 4y ago
I'm wondering if it would be worth skipping the SQL layer and compiling straight to the execution plan level. This would make PRQL tightly coupled to a database, but would also unlock performance and fine-tuning potential. And, maybe it would be possible to run it inside the database process, enabling data processing right at the source.
(eg. in Postgres, this would be more or less a replacement of the PREPARE statement.)
- assface 4y ago> I'm wondering if it would be worth skipping the SQL layer and compiling straight to the execution plan level. You have to join five tables in a query. What join order will you pick in your execution plan? Will you use the same ordering for different WHERE clause parameters? What if you add an index? What you're proposing was tried in the 1970s (see CODASYL) and it was a terrible idea.
- snthpy 4y agoPRQL actually goes through two intermediary ASTs (PL and RQ) before being compiled into SQL. This is not that well documented yet but have a look at the comments at the top of the lib.rs [0]. Both representations are accessible as json in the Rust and Python APIs. On the roadmap are plans to support DataFrame backends such as Polars, Pandas, Ibis, ... and Substrait.io [1]. The aim of Substrait is similar to what you describe, i.e. to create a universal relational algebra representation that can be executed directly by backend systems without having to go through SQL. [0]: https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib.rs https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib... [1]: https://substrait.io/ https://substrait.io/