4 ms·

We had no data loss incidents. My concern was around tooling and maintenance of the database. Once you had to start clearing out bad events in dev environment

by eric_b 10y ago

We had no data loss incidents. My concern was around tooling and maintenance of the database. Once you had to start clearing out bad events in dev environments or doing common administrative chores there were many holes. I never did find out how to remove specific problem events. We ended up just purging the store every time something went wrong in lower environments. We had no solution for this problem in prod. How would you fix an event that should not have gotten in to the store? (Wrong contract, bad data etc) I understand it shouldn't happen, but in the real world all kinds of things go wrong. I can't imagine running a massive system on something like event store. Fortunately the projects always failed well before we got in to any kind of production with real users.

ehsanu1 10y ago
Not saying this is great, and I don't have experience with an ES system in production, but... Could you do one of: 1. Manually remove the specific problem event from the store, then re-run all events starting from the previous accumulated state snapshot stored before that event? Then you get the new state snapshot just by re-processing all the events. This of course assumes you have state snapshots, and may not be feasible if this is a common occurrence and there are too many events to process. Or: 2. Create a "reverse" event? This will cancel out the bad change and give you a new, valid state to work from and continue. This is nice since the historical state of the system is still represented, but that may be a disadvantage in some situations.
- ouro 10y ago
  my answer above ^^ assumes we really want to remove it. compensating actions are generally the preferred option
ouro 10y ago
I can see why your projects failed. That EventStore does not support you editing an event is not something that makes it "not production ready" it is in fact a feature that keeps you from doing stupid things. If you go and edit an event, how do subscribers receive that edit? Let's imagine I have a projection updating a sql db and you now edit an event,how will this projection receive the edit? "We had no solution for this problem in prod. How would you fix an event that should not have gotten in to the store? (Wrong contract, bad data etc) I understand it shouldn't happen, but in the real world all kinds of things go wrong." You should do some more research into eventsourced systems as there are patterns for handling these exact scenarios. http://files.movereem.nl/2017saner-eventsourcing.pdf http://files.movereem.nl/2017saner-eventsourcing.pdf discusses some. In your scenario the most common is read the problem stream out, write it to a new stream (with any changes that you want) then either delete the old stream or leave a last event in the old stream saying it has been migrated to the new stream.
hcarvalhoalves 10y ago
How would you fix an event that should not have gotten in to the store? - Reverse transaction (like in accounting) - Replay events and filter/modify the problematic events into a new event store Fortunately the projects always failed well before we got in to any kind of production with real users. That the project failed and the above was still an open question for you explains a lot.

by eric_b 10y ago

ehsanu1 10y ago

Not saying this is great, and I don't have experience with an ES system in production, but... Could you do one of: 1. Manually remove the specific problem event from the store, then re-run all events starting from the previous accumulated state snapshot stored before that event? Then you get the new state snapshot just by re-processing all the events. This of course assumes you have state snapshots, and may not be feasible if this is a common occurrence and there are too many events to process. Or: 2. Create a "reverse" event? This will cancel out the bad change and give you a new, valid state to work from and continue. This is nice since the historical state of the system is still represented, but that may be a disadvantage in some situations.

ouro 10y ago
my answer above ^^ assumes we really want to remove it. compensating actions are generally the preferred option

ouro 10y ago

I can see why your projects failed. That EventStore does not support you editing an event is not something that makes it "not production ready" it is in fact a feature that keeps you from doing stupid things. If you go and edit an event, how do subscribers receive that edit? Let's imagine I have a projection updating a sql db and you now edit an event,how will this projection receive the edit? "We had no solution for this problem in prod. How would you fix an event that should not have gotten in to the store? (Wrong contract, bad data etc) I understand it shouldn't happen, but in the real world all kinds of things go wrong." You should do some more research into eventsourced systems as there are patterns for handling these exact scenarios. http://files.movereem.nl/2017saner-eventsourcing.pdf http://files.movereem.nl/2017saner-eventsourcing.pdf discusses some. In your scenario the most common is read the problem stream out, write it to a new stream (with any changes that you want) then either delete the old stream or leave a last event in the old stream saying it has been migrated to the new stream.

hcarvalhoalves 10y ago

How would you fix an event that should not have gotten in to the store? - Reverse transaction (like in accounting) - Replay events and filter/modify the problematic events into a new event store Fortunately the projects always failed well before we got in to any kind of production with real users. That the project failed and the above was still an open question for you explains a lot.