5 ms·
My day-job is Data Engineering. We've carefully built our ETL patterns/tools/data structures to be idempotent. It's amazing how freeing this is. Dealing with ca
by jkubicek 3y ago
My day-job is Data Engineering. We've carefully built our ETL patterns/tools/data structures to be idempotent. It's amazing how freeing this is. Dealing with cascading failures, intermittent failures, delays, re-runs and time-zones are all os much more manageable when we know that just re-running any given task will do the right thing.
It's also amazing how hard of a concept idempotent data pipelines are for some folks to understand, and how much effort they'll put into _avoiding_ idempotency.
- Kim_Bruning 3y agoI can attest. I tried explaining Idempotency to people building an industrial laser controller once. Idempotent calls might be used redundantly to ensure that a particular state or setting exists at a particular time. I don't really care that a particular setting was already set yesterday or an hour ago. Conversely, it's probably a really a good idea to be sure it's on that particular setting right now, before I fire the laser!
- OscarCunningham 3y agoI try to make my pipelines idempotent, but a couple of them require us to send the output file to someone's email. Does anyone know how to send an email idempotently?
- mbb70 3y agoEmail is quintessentially "fire and forget" message passing, so there's no way to guarantee mail has been delivered and no way to "pull it back". You can embed tracking in your email to see if it's been opened, and not resend in that case, but this will be flakey as it's easy to disable email trackers.