2 ms·
On the other hand, weird things can happen at scale inside an RDBMS. For example: window queries, where you're trying to extract `row_number() = 1` of each win
by derefr 4y ago
On the other hand, weird things can happen at scale inside an RDBMS.
For example: window queries, where you're trying to extract `row_number() = 1` of each window-partition. (Or "DISTINCT ON" queries, which are basically syntax sugar for this.) These can be extremely slow, because in many RDBMSes† they generate a query plan that reads+materializes all the rows of each window-partition, in order to then just discard all-but-the-first row-tuple of each window-partition. (In other words, the filtering-out by rank doesn't get pushed back through the query plan to turn the whole-index scan into an index-prefix scan.)
The (surprising) solution to this over-materialization, is to use a recursive CTE, using the previous query's matched key as the inductive case's filter condition, to cause index-prefix matching to happen "one query step at a time." Which, if you think about it, is just "causing hundreds/thousands of individual queries"... just with all of those queries happening within a single tx, statement, query plan, table lock, and (hopefully!) page cursor.
† Some RDBMSes do know how to push this window-node attribute back into the fetch node; the feature is called "index skip scans" or "loose index scans." (Before anyone asks: Postgres isn't one of them... yet. https://wiki.postgresql.org/wiki/Loose_indexscan https://wiki.postgresql.org/wiki/Loose_indexscan)
- Widdershin 4y agoThis is good to know about, used a similar window function in Postgres recently. Will double check the performance of that query.