It took me some time to get a good grasp of the power of SQL; and it really kicked in when I learned about optimization rules. A query is a program that you rew

It took me some time to get a good grasp of the power of SQL; and it really kicked in when I learned about optimization rules. A query is a program that you rewrite, just like an optimizing compiler would. You state what you want; then you have different ways to fetch and match and massage data; and you can search through this space to produce a physical plan. Hopefully you used knowledge to weight parts to be optimized (table statistics, like Java's JIT would detect hot spots). I find it fascinating to peer through database code to see what is going on. Lately, there's been new advances towards streaming databases, which bring a whole new design space. For example, now you have latency of individual new rows to optimize for, as opposed to batch it whole to optimize the latency of a dataset. Batch scanning will be benefit from better use of your CPU caches. And maybe you could have a hybrid system which reads history from a log and aggregates in a batched manner, and then switches to a stream-optimized execution plan when it reaches the end of the log. If you want to have a peek at that here are Flink's set of rules [1], generic and stream-specific ones. The names can be cryptic, but usually give a good sense of what is going on. For example: PushFilterIntoTableSourceScanRule makes the WHERE clause apply the earliest possible, to save some CPU/network bandwidth further down. PushPartitionIntoTableSourceScanRule tries to make a fan-out/shuffle happen the earliest possible, so that parallelism can be made use of. [1] https://github.com/apache/flink/blob/5f8fb304fb5d68cdb0b3e3c6203691b14be92935/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/FlinkStreamRuleSets.scala https://github.com/apache/flink/blob/5f8fb304fb5d68cdb0b3e3c...