3 ms·
> Incremental View Maintenance It’s nice to have that as an automatic feature rather than something that you have to create for yourself with stored procedures
by derefr 5y ago
> Incremental View Maintenance
It’s nice to have that as an automatic feature rather than something that you have to create for yourself with stored procedures (doing e.g. dynamic programming in the DB), but a matview is still a matview — it’s a real table taking up space on disk and getting replicated through the WAL.
The thing Materialize gives you is the same thing you get from Apache Beam nee Google Cloud Dataflow at cluster scale, or from Elixir’s Flow / Java’s Akka Streams at the in-process scale on the business layer. Namely, the ability to have one stream node in a flow digraph that feeds N outputs that consume it at different rates, and to keep just enough of the intermediate stream node materialized in a buffer to keep all those readers served. Or: like the kind of ACK-truncated buffer that serves a Kafka/Redis Streams consumer group, but with all the stages in the same address-space without serialization+remoting overhead in-between.
This “garbage-collected buffered materialization” is a very important property, if your A query returns a million A IDs, and you’re doing a LEFT JOIN from Bs to As that’s dropping most of the As because they don’t have a related B. You don’t want to be holding onto — and replicating! — a thousand concurrent-request copies of that million A ID list. (The million A IDs are also why you’re doing CTEs in your B/C queries in the first place. With a small list of A IDs, it’d be sensible to just round-trip the A ID list to the client, getting sent back as IN (VALUES ...) bind-params for the B/C queries. But 1. naively, those million IDs become a million bind-params, and that’s stupid, slow, and actually impossible [PG only accepts 65K bind-params per statement!]; and 2. trying to pass the fetched list as an A.id[] and using <@ will deoptimize the search.)
The closest thing PG has is the ability to BEGIN a transaction; create a TEMPORARY table for that transaction; SELECT INTO the temporary table the result of a query; and then use that temporary table to run two queries. Which you have to do in PL/pgSQL (losing the benefits of “whole program” query-plan optimization you get from CTE queries); and then you still have to do something at the end to get your two queries' differently-shaped outputs back to the user. (For example, encoding every row into JSONB with an explicit embedded row-type-tag, and then double-decoding at the receiver; materializing the results of the queries into two more UNLOGGED [but not TEMPORARY] tables, fetching the results from them in two more queries, and then dropping them; etc.)
- hardwaresofton 5y ago> It’s nice to have that as an automatic feature rather than something that you have to create for yourself with stored procedures (doing e.g. dynamic programming in the DB), but a matview is still a matview — it’s a real table taking up space on disk and getting replicated through the WAL. Ah, yeah at the end of the day both unlogged materialized views aren't supported by postgres[0]. Also a patch that introduces temporary materialized views is stuck behind some refactoring that started in 2019[1] which is unfortunate. > (explanation of materialize) Thanks for this -- I think I get more clearly what you're getting out of materialize. I'm not sure I quite understand the stream analogy because the data being buffered here feels chunked (though I guess it could be incrementally maintained with some more smarts), but I think I understand the issue that basically ongoing queries need access to this temporarily-useful partial result. Basically a cross-session intermediate query result cache. This feels like it could be reduced to a custom table access method plugin, but if that temporary table patch gets merged it could be even easier to hack something together. [0]: https://www.postgresql-archive.org/CREATE-UNLOGGED-MATERIALIZED-VIEW-td6064079.html https://www.postgresql-archive.org/CREATE-UNLOGGED-MATERIALI... [1]: https://www.postgresql.org/message-id/CAKLmikMgkuwFKHem%2BSfBb_G7J01fzTvPJyG9dVOOgVCuakCDCA%40mail.gmail.com https://www.postgresql.org/message-id/CAKLmikMgkuwFKHem%2BSf...