3 ms·

I am very curious to know how you feel about PRQL (prql-lang.org) ? IMHO it gives you the ergonomics and DX of Polars or Pandas with the power and universality

by snthpy 3y ago

I am very curious to know how you feel about PRQL (prql-lang.org) ? IMHO it gives you the ergonomics and DX of Polars or Pandas with the power and universality of SQL because you can still execute your queries on any SQL compatible query execution engine of your choice, including Polars and Pandas but also DuckDB, ClickHouse, BigQuery, Redshift, Postgres, Trino/Presto, SQLite, ... to name just a few popular ones. The join syntax and semantics is one of the trickiest parts and is under discussion again recently. It's actually one of the key parts of any data transformation platform and is foundational to Relational Algebra, being right there in the "Relational" part and also the R in PRQL. Most of the PRQL built-in primitive transforms are just simple list manipulations like map, filter or reduce but joins require care to preserve monadic composition (see for example the design of SelectMany in LINQ or flatmap in the List Monad). See this comment for some of my thoughts on this: https://github.com/PRQL/prql/issues/3782#issuecomment-1811312787 https://github.com/PRQL/prql/issues/3782#issuecomment-181131... That issue is closed but I would love to hear any comments and you are welcome to open a new issue referencing that comment or simply tagging me (@snth). Disclaimer: I'm a PRQL contributor.

theLiminator 3y ago
Do you compile to substrait or to SQL strings?
- snthpy 3y ago
  SQL strings as the final output but there are two intermediate representations that can be serialized to JSON. There's an open issue for Substrait but I don't think that anyone's started any work on that yet.
serjester 3y ago
First time I’ve heard of it but seems very cool. My background is data science though so being able to use DS libraries or even apply a python function is why I find myself in Pandas / Polars. This seems very powerful for a data engineer. I also think it’s awesome you guys have a duckdb integration - maybe I’ll try it out.

by snthpy 3y ago

theLiminator 3y ago

Do you compile to substrait or to SQL strings?

snthpy 3y ago
SQL strings as the final output but there are two intermediate representations that can be serialized to JSON. There's an open issue for Substrait but I don't think that anyone's started any work on that yet.

serjester 3y ago

First time I’ve heard of it but seems very cool. My background is data science though so being able to use DS libraries or even apply a python function is why I find myself in Pandas / Polars. This seems very powerful for a data engineer. I also think it’s awesome you guys have a duckdb integration - maybe I’ll try it out.