4 ms·
Always found the first method of keeping bookmarks to the positions in a table to be the best method. I don’t think the 3 problems given to be an issue personal
by mj2718 4y ago
Always found the first method of keeping bookmarks to the positions in a table to be the best method. I don’t think the 3 problems given to be an issue personally.
- just loop at the interval that your users can accept the delay
- querying a database 10 times a second is nothing, even 100 times, this wouldn’t cause resource limitations
- the scaling concern made no sense to me, seems a bit arbitrary.
Cool to know this is an option though, but I much prefer not relying on database internals like wal logs, this hurts scaling more IMO.
The only thing you need to worry about with the table method is out of order increment IDs, which is always possible in a transactional database. But there are many solutions for this.
- uneven88 4y agoDo you have a few pointers to the solutions regarding out of order IDs? I'm thinking of keeping track of gaps (yet unseen IDs) in another column and retrieving them in the next poll.
- dagss 4y agoNot OP, but there is an approach here of using a dedicated loop worker to assign post-commit ID sequence. I.e. using the outbox pattern once, simply to assign a post-commit ID. https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVATION.md https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA... I wish DBs had this more built in, it seems a critical feature of a DB these days and the commit log already have very similar sequence numbers internally...
- Redsquare 4y agoor just use a proper eventstore
- dagss 4y ago1) Well assuming data is stored in the SQL as the transactional store, how do you move data safely from SQL to the eventstore? You at least need the postbox pattern. It is not clear to me that the postbox pattern is less hacky than listening to a SQL change feed. E.g., with Azure Cosmos DB -- you would not use the postbox pattern there. You listen to the DB change feed -- since that is provided and is easily accessible. 2) In our case the schemas in SQL are mostly event based already (we prefer insert over update ...even if we do business logic/flexible querying on the data). So using an event store is mainly a duplication of the data. An event store is a database too. What exactly is it about a database that makes it a "proper event store"? I honestly think the focus on duplicating data you have in your DB in a separate event store DB too may be something of a fad that will pass in a while -- similar to NoSQL. It's needed for volumes larger than what a SQL DB can handle, but if you don't need such large volumes why introduce the extra component. Event sourcing architecture is great; but such thinking on an architecture level is really orthogonal to the storage and transport chosen.
- mj2718 4y agoI wrote a blog post on this. We were using event sourcing and went with locking+batching method described. For other tables we wanted to read and move to a data warehouse we used the txid check. A bit hacky, but on AWS RDS we didn’t want to deal with wal intricacies. https://mattjames.dev/auto-increment-ids-are-not-strictly-monotonic/ https://mattjames.dev/auto-increment-ids-are-not-strictly-mo...
- dagss 4y agoI really like this solution and agree "bookmarks" are a lot better than outbox, but it is hard to guard against races where the reader increments the bookmark before an active transaction commits an ID at an earlier position; if transaction commits in a different order than IDs are allocated. Perhaps this is what you mean with "out of order increment"... but what are the "many" solutions to this? I struggle for a long time to find a good way of doing this in Microsoft SQL and still not perfectly happy about the solution we found.