4 ms·
You're right. Since we don't want to put too much pressure on the source database, we do save the (transformed) WAL items in an intermediary database (we call t
by WilcoKruijer 1y ago
You're right. Since we don't want to put too much pressure on the source database, we do save the (transformed) WAL items in an intermediary database (we call this the event store), so the source can clear its WAL.
This does mean the intermediary database can grow in an unbounded fashion. The use case really determines if this is fine or not. Since our focus right now is on (micro)service communication, we think this is fine in most cases, as the throughput usually is not gigantic.
Since the event store is just a Postgres database, it's easy to set up partitions to only retain data for a certain amount of time. On the near-term roadmap we also have back-fill support which will make it easier to work with shorter retention windows.