3 ms·
This is fundamentally a data virtualisation approach - even though it is achieved by streaming replicas and fdws instead of tradictional data replication tools.
by glogla 6y ago
This is fundamentally a data virtualisation approach - even though it is achieved by streaming replicas and fdws instead of tradictional data replication tools.
It is probably more complex than replicating a single database to a Snowflake, but one huge benefit of the described solution is that it's real-time - and getting realtime replication to data warehouse is a very different challenge.
It allows you to create both batch and realtime data products just by creating views or materialized views, which is pretty simple and neat. You also get real-time ad-hoc querying, which is very valuable.
The issue with this approach is it keeps the data in row-based format so from certain size it isn't going to be practical. For the "daily report" use case, it's probably fine for a while, but if the source system grow to terabyte size, you will be spending a lot of time waiting on Postgres to do it's thing.
So as the data grows, you might lose first the real-time ad-hoc querying (if you ever had it), then the real-time reporting and then the daily reporting.
But maybe the data will never grow that much and everything is going to be fine for a long time? Maybe this gets you where you need to be for a few years and who knows how the data platforms will look like in a few years.