3 ms·
I'm curious how you guys are usoing Postgres as a log. I took a brief look at the code, and while the append-only dataset strategy is sound, it looks like your
by atombender 7y ago
I'm curious how you guys are usoing Postgres as a log.
I took a brief look at the code, and while the append-only dataset strategy is sound, it looks like your scenario only has a single reader and a single writer?
In my experience, it's not entirely trivial when you have:
1. Multiple readers who each needs to be able to follow the log from different positions in real time.
2. Multiple writers receiving events that need to be written to the end of the log.
3. Real-time requirements.
From what I can tell — I could be wrong here — your system doesn't need to poll the table constantly, because you also save the log to RAM, so whenever you receive an event, you can optimistically handle it in memory and merely issue status updates. If anything goes wrong, a reader can replay from the database.
But that doesn't work with multiple writer nodes where each node receives just a part of the whole stream. The only way for this to work would be dedicate a writer node to each stream so that it goes through the same RAM queue. So then you need a whole system that uses either Postgres or some consensus system like Etcd to route messages to a single writer, and you need to be able to recover when a writer has been unavailable.
Edit: I see you wrote that "we assign same user to same writer", so you're doing something like that.
- soumyadeb 7y agoAgreed. Our current implementation does not work when there are multiple readers for the same event stream and we need to track per-reader watermarks. We have a very simple model where one reader reads from DB and distributes the work to multiple workers (e.g. network writers) which in turn update the job status. Multiple writers should work though. StoreJob() should handle that. I missed the logging to RAM part. Yes, we always wanted to do that but haven't gotten to that yet. Right now, all events are moved through the DB - between gateway and processor and then router. Hence, we poll the table constantly. Would love if you join our discord channel https://discordapp.com/channels/625629179697692673/625629179697692677 https://discordapp.com/channels/625629179697692673/625629179.... Slightly easier to have technical discussion there :)