3 ms·
> PostgreSQL databases typically don't generate huge amounts of data The live data set may not be huge, but the entire trail of all changes of all current and
by gunnarmorling 1y ago
> PostgreSQL databases typically don't generate huge amounts of data
The live data set may not be huge, but the entire trail of all changes of all current and all previously existing data may easily exceed the volume of data you can reasonably process with Postgres.
In addition, its row based storage format doesn't make it an ideal fit for typical analytical queries on large amounts of data.
Replicating the data from Postgres to Iceberg addresses these issues. But, of course, it's not without its own challenges, as demonstrated by the article.
- hodgesrm 1y agoRight, that's the archive case. One way to handle that is to replicate the log records and let users figure it out in the analytic storage. That also allows time travel. Either way the data should be frozen or it makes the problem way harder.