4 ms·
If the consumers stall, doesn't the WAL have to grow in unbounded fashion? Does it place any backpressure on the writers?
by jasonthorsness 1y ago
If the consumers stall, doesn't the WAL have to grow in unbounded fashion? Does it place any backpressure on the writers?
- WilcoKruijer 1y agoYou're right. Since we don't want to put too much pressure on the source database, we do save the (transformed) WAL items in an intermediary database (we call this the event store), so the source can clear its WAL. This does mean the intermediary database can grow in an unbounded fashion. The use case really determines if this is fine or not. Since our focus right now is on (micro)service communication, we think this is fine in most cases, as the throughput usually is not gigantic. Since the event store is just a Postgres database, it's easy to set up partitions to only retain data for a certain amount of time. On the near-term roadmap we also have back-fill support which will make it easier to work with shorter retention windows.