4 ms·

Control Pandas, Polars, or SQL from One DSL

34 points by jmount 4y ago

hatmatrix 4y ago
Reminds me of the sqldf project for R [0] which allowed you to work with R data frames and SQL databases with SQL queries. [0] https://cran.r-project.org/web/packages/sqldf/index.html https://cran.r-project.org/web/packages/sqldf/index.html
- jmount 4y ago
  SQLDF is a great project, thanks for bringing it up. We use it in our book ( https://win-vector.com/practical-data-science-with-r/ https://win-vector.com/practical-data-science-with-r/ ). Inspired by it I have a feature to the data algebra that treats a captive database connection as a pool of data frames. A test showing the idea is here: https://github.com/WinVector/data_algebra/blob/main/tests/test_data_space.py https://github.com/WinVector/data_algebra/blob/main/tests/te...
password4321 4y ago
Seems to be "transpile to SQL" week: 12/30 https://news.ycombinator.com/item?id=34181319 https://news.ycombinator.com/item?id=34181319 PRQL: a simple, powerful, pipelined SQL replacement 12/28 https://news.ycombinator.com/item?id=34159699 https://news.ycombinator.com/item?id=34159699 SQL + M4 = Composable SQL
- snthpy 4y ago
  PRQL actually goes through two intermediary ASTs (PL and RQ) before being compiled into SQL. This is not that well documented yet but you can have a look at the comments at the top of the lib.rs [0]. Both representations are accessible as json in the Rust and Python APIs. On the roadmap are plans to support DataFrame backends such as Polars, Pandas, Ibis, ... and Substrait.io [1]. The aim of Substrait is to create a universal relational algebra representation that can be executed directly by backend systems without having to go through SQL. [0]: https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib.rs https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib... [1]: https://substrait.io/ https://substrait.io/
- giovannibonetti 4y ago
  And no mention of Datalog so far
snthpy 4y ago
This looks very interesting. Have you looked at Substrait.io [1] at all?. The aim of Substrait is similar to what you describe, i.e. to create a universal relational algebra representation that can be executed directly by backend systems without having to go through SQL. [1]: https://substrait.io/ https://substrait.io/

34 points by jmount 4y ago

Reminds me of the sqldf project for R [0] which allowed you to work with R data frames and SQL databases with SQL queries. [0] https://cran.r-project.org/web/packages/sqldf/index.html https://cran.r-project.org/web/packages/sqldf/index.html

jmount 4y ago
SQLDF is a great project, thanks for bringing it up. We use it in our book ( https://win-vector.com/practical-data-science-with-r/ https://win-vector.com/practical-data-science-with-r/ ). Inspired by it I have a feature to the data algebra that treats a captive database connection as a pool of data frames. A test showing the idea is here: https://github.com/WinVector/data_algebra/blob/main/tests/test_data_space.py https://github.com/WinVector/data_algebra/blob/main/tests/te...

password4321 4y ago

Seems to be "transpile to SQL" week: 12/30 https://news.ycombinator.com/item?id=34181319 https://news.ycombinator.com/item?id=34181319 PRQL: a simple, powerful, pipelined SQL replacement 12/28 https://news.ycombinator.com/item?id=34159699 https://news.ycombinator.com/item?id=34159699 SQL + M4 = Composable SQL

snthpy 4y ago
PRQL actually goes through two intermediary ASTs (PL and RQ) before being compiled into SQL. This is not that well documented yet but you can have a look at the comments at the top of the lib.rs [0]. Both representations are accessible as json in the Rust and Python APIs. On the roadmap are plans to support DataFrame backends such as Polars, Pandas, Ibis, ... and Substrait.io [1]. The aim of Substrait is to create a universal relational algebra representation that can be executed directly by backend systems without having to go through SQL. [0]: https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib.rs https://github.com/PRQL/prql/blob/main/prql-compiler/src/lib... [1]: https://substrait.io/ https://substrait.io/
giovannibonetti 4y ago
And no mention of Datalog so far

snthpy 4y ago

This looks very interesting. Have you looked at Substrait.io [1] at all?. The aim of Substrait is similar to what you describe, i.e. to create a universal relational algebra representation that can be executed directly by backend systems without having to go through SQL. [1]: https://substrait.io/ https://substrait.io/