4 ms·
The SQL with pipe syntax is also been implemented at Databricks since Jan 30, 2025: https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-pipeli
by jakozaur 2y ago
The SQL with pipe syntax is also been implemented at Databricks since Jan 30, 2025:
https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-pipeline.html https://docs.databricks.com/en/sql/language-manual/sql-ref-s...
Still, the best is yet to come. Previously, SQL extensions were a pain. There was no good place, and table-value functions were a mess.
Now, it would be possible to have higher-order functions such as enrichment, predictions, grouping or other data contracts. Example:
FROM orders
|> WHERE order_date >= '2024-01-01'
|> AGGREGATE SUM(order_amount) AS total_spent GROUP BY customer_id
|> WHERE total_spent > 1000
|> INNER JOIN customers USING(customer_id)
|> CALL ENRICH.APOLLO(EMAIL > customers.email)
|> AGGREGATE COUNT(*) high_value_customer GROUP BY company.country
This may be called one SQL to determine distinct e-mail domains, then prepare an enriching dataset and later execute the final SQL with JOIN.
Iterative SQL with pipes may also work better with GenAI.
- memhole 2y agoThanks for sharing. I wasn't aware they pushed that out. The ordering of this makes so much more sense. My only real concern, is that I think CTEs are so common place and part of ANSI SQL that you'll see people trade a standard for this. But, I also gripe that Snowflake uses // for comments and people get confused when their code doesn't work on another DB. Or Oracles' join syntax is another example.
- mc_maurer 2y agoIf only it worked on Databricks serverless SQL warehouses...
- rxin 2y agoThis is coming to Serverless SQL warehouses soon!
- valyala 2y agoPipe-based query syntax has been appeared a better alternative to SQL in many databases, because it is easier to read, write and maintain. The only thing is that this syntax isn't standardized among databases yet. BTW, we at VictoriaLogs also use pipe-based syntax for LogsQL query language [1] since the first release in 2023. Recently I wrote an SQL -> LogsQL conversion guide [2], which has been appeared quite clear and easy to follow. This guide was written during the conversion of SQL queries to LogsQL at ClickBench benchmark from ClickHouse [3]. [1] https://docs.victoriametrics.com/victorialogs/logsql/ https://docs.victoriametrics.com/victorialogs/logsql/ [2] https://docs.victoriametrics.com/victorialogs/sql-to-logsql/ https://docs.victoriametrics.com/victorialogs/sql-to-logsql/ [3] https://github.com/ClickHouse/ClickBench/tree/main/victorialogs https://github.com/ClickHouse/ClickBench/tree/main/victorial...