3 ms·
This is a great idea if you disable materialization (which the author of this post does not mention). Yes, materialization is fine for small hardcoded values i
by srcreigh 4y ago
This is a great idea if you disable materialization (which the author of this post does not mention).
Yes, materialization is fine for small hardcoded values in the post, but for most other lookup tables, it's definitely not fine.
For example, you read this article and think great, I'll make a lookup table to map various ids to be linked across tables.
with ids as (
select u.user_id, u.token, s.customer_id
from users u
join stripe_customer s on s.user_id = u.user_id
),
user_impressions as (
select i.*, ids.id
from impressions i join ids using token
),
payments as (
select p.*, ids.id from payments p
join ids using customer_id
),
select sum(i.views),
sum(i.clicks),
sum(p.amount)
from payments p
join user_impressions i using user_id
where user_id = $1
group by date;
You think, great, this gets some stats for one user since its filtered at the end, and with indexes on stripe_customer(user_id), user_impressions(token), and payments(customer_id), this will be really fast and efficient!
Nope.. since ids is referenced more than once in this query, in Postgres this causes the ids CTE to be materialized onto disk with no indexes. So not only does it take up lots of extra space on disk to store all the users in your system, to join user_impressions and payments you have to O(N) search across the ids dataset. No indexes on a materialized table.
It would be a lot faster to join to users or stripe_customer or both in a loop with indexes than O(N) search through all the users in your system.
This is particularly dangerous because if you remove the payments part, ids only has one reference, so Postgres doesn't materialize the table, and everything is fast.
This can be fixed by adding WITH foobar AS NOT MATERIALIZED to the CTE syntax. IMO it should be a syntax error to not specify AS MATERIALIZED or AS NOT MATERIALIZED.. The default has too many potential performance problems and folks should have to think about whether they want materialization or not.
- cyclotron3k 4y ago`NOT MATERIALIZED` is now the default in Postgres, and has been since PG12 I believe
- srcreigh 4y agoOnly when the CTE is referenced once. If it's referenced more than once, it is automatically materialized.
- switchbak 4y agoThis is a pretty huge caveat, one that I'm very thankful to know of. Thanks for the heads up!
- baq 4y agoside note: I asked chatgpt to rewrite this query to use subqueries and it did a reasonable job. next step: include cte-to-subquery translation step as a part of your build pipeline. never needed tools like that but i guess they must exist since it's been such a common issue. using gpt for this is like nuking a mosquito from orbit.
- srcreigh 4y agoAS NOT MATERIALIZED is equivalent to using sub queries