13 ms·
Idempotence now prevents pain later
- dang 6y agoPossibly related past threads: What Is Idempotence? - https://news.ycombinator.com/item?id=19570815 https://news.ycombinator.com/item?id=19570815 - April 2019 (51 comments) Idempotence: What is it and why should I care? - https://news.ycombinator.com/item?id=17804617 https://news.ycombinator.com/item?id=17804617 - Aug 2018 (73 comments) You know how HTTP GET requests are meant to be idempotent? - https://news.ycombinator.com/item?id=16964907 https://news.ycombinator.com/item?id=16964907 - May 2018 (304 comments) Implementing Stripe-Like Idempotency Keys in Postgres - https://news.ycombinator.com/item?id=15569478 https://news.ycombinator.com/item?id=15569478 - Oct 2017 (41 comments) APIs, robustness, and idempotency - https://news.ycombinator.com/item?id=13707681 https://news.ycombinator.com/item?id=13707681 - Feb 2017 (50 comments) A simple distributed algorithm for small idempotent information - https://news.ycombinator.com/item?id=7276491 https://news.ycombinator.com/item?id=7276491 - Feb 2014 (14 comments) Idempotent Web APIs: What benefit do I get? - https://news.ycombinator.com/item?id=5662138 https://news.ycombinator.com/item?id=5662138 - May 2013 (53 comments) A word like that is particularly easy to search for: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E10%20Idempoten&sort=byDate&type=story&storyText=none https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- totetsu 6y agoWhen the same thread is appears on HN twice does it get the same results?
- thrwyoilarticle 6y agoThis kind of thing is allowed according to the FAQs. I think the links are just additional reading.
- josephcsible 6y agoI think you missed a joke there.
- thrwyoilarticle 6y agoOuch
- musingsole 6y agoI would've missed it too if not for this interaction!
- dang 6y agoIt's idempotent except it calls a RNG.
- totetsu 6y agoor is it only pseudorandom taking the time of day and day of week as a seed.
- efitz 6y agoYes, it always results in at least one “this has been posted before” post.
- mrbadideas 6y agoIs that really idempotence?
- omarhaneef 6y agoI'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which means "same" and potence is of course power/potent, so it has the same power/effect however many times you run it)
- sdenton4 6y agoMathematically, it's x^2 = x, which implies x^n = x for all positive integers n. Nilpotence (x^2 = 0) is also very helpful some times: it's a process which is self-reversing. Like the discrete Fourier transform (if you set up the constants properly).
- carreau 6y agoNo, you are confusing with involution. 1/x is an involution. Symetries are often involutions. Squaring a upper triangular matrix with 0 on the diagonal is nilpotent. Derivatiting a polynomial of degree N is nilpotent after N iteration.
- contravariant 6y agoCareful, for nilpotence the power doesn't have to be 2. Also you may be confusing it with x^n = 1 (which I'm not sure how to name, 'root of unity' perhaps). This would be the case for the Fourier transform (with n=4). If x^2 = 0 then applying the Fourier transform twice would null your function, which isn't the case.
- pdpi 6y agoX^2 is a weird way to describe it. A function f is idempotent if f(f(x)) = f(x).
- creata 6y agoIt's not that weird. People often write iterated composition as f^k, and this is especially true with matrices, where composition and multiplication mean the same thing.
- deleted 6y ago[deleted]
- smiley1437 6y agoI had only ever encountered idempotence in the context of system management (Ansible, Puppet, Chef, etc) Article made me think it's actually applicable to other management as well
- void_mint 6y agoPUT requests are intended to be idempotent, which is one of the things that distinguishes them from POST requests. This (in my experience) is most non-CS-backgrounded software developers' exposure to idempotence, but it actually has tons of value pretty much wherever you can apply it. The ability to (sometimes accidentally) do a thing twice and have it leave no unintended consequence is huge. UPSERTs can be idempotent as well. "If this doesn't exist, create it, and if it does, update it to match this state", implies that running it twice will leave no unintended side effects.
- physicles 6y agoIn the last couple years I've come to realize that nearly all POSTs should actually be PUTs, sometimes with a client-provided uniqueness token in the URL (like a UUID) that becomes part of the newly-created resource's ID. Now that I think about it, all the situations where I'm fine using POST are also idempotent, such as changing an object's metadata (technically PUT or PATCH), or sending a batch of HTTP requests to an endpoint bundled as a single request.
- Animats 6y agoIt's a basic property of HTTP "GET". Or it's supposed to be. "GET" is not supposed to change server site state. That's what "POST" is for. This matters if there's a cache in the middle, since caches tend to assume that GET requests are idempotent and can be served from cache. Cloudflare assumes that. POST requests have to go through to the real server.
- elcomet 6y agoGet is idempotent because it's the identity function. It does not change the state of the data. So it's a trivial case of idempotency. A more interesting function is PUT (idempotent) vs POST (not idempotent)
- globular-toast 6y agoI try to write idempotent software whenever I can. It's usually not much more difficult to make it work and affords so much more flexibility and less worry when it's done.
- staticassertion 6y agohttps://lostechies.com/jimmybogard/2013/06/06/acid-2-0-in-action/ https://lostechies.com/jimmybogard/2013/06/06/acid-2-0-in-ac... If you can build a system with ACID 2.0 life gets really easy. You can reason about your system without worrying about ordering, time, 'exactly once' semantics, etc. Idempotency is usually one of the simplest pieces to implement, and you definitely get a ton of benefit right off the bat - it's worth designing systems from scratch with it in mind.
- Justsignedup 6y agoI've done this exact thing many times before... I can honestly say that Eric is 100% right with his approach. It always leads to less headaches, more flexibility (oh trust me, someone is always gonna have a "but... there's like a special thing that I sometimes have to do" and it breaks some assumptions. In any case... yeah... let's just say any time you have to be worried "did we already schedule this", really think "can this never care if it was or not? Should be always safe to schedule it again"
- brsg 6y agoIdempotency is a pretty critical concept in system design, and I think most developers have run into issues related to it even if they aren't directly familiar with the term. To give another simple example as the OP - Suppose you have a product that relies on time series data. For demo purposes you might create a curated data set to present to clients, but the presenter doesn't want to show data from 2019 as the "most recent" Naturally, you decide to write a script. Do you A) Write as script that moves the data forward by 1 week explicitly, and simply run this once per week or B) Write a script that compares the current date to the data and moves it forward as much as it needs At first glance, these two approaches work the same, but what if (A) triggers twice? What if it runs once every 6 days by mistake? (B) is idempotent however - subsequent executions won't change the state. It's usually impossible to predict all of the ways that software breaks, but designing with idempotency in mind eliminates a lot of them.
- jayd16 6y agoI don't think B is technically idempotent either. Change still occurs but with minimal difference. You cannot cache the results and use them again next week. An idempotent change would be to pass in the current time instead of checking system time. In this case, as long as the input is the same, the result is the same. You could use cached results, but most likely you want to use new inputs.
- deleted 6y ago[deleted]
- elcomet 6y agoYes but this doesn't matter really. What's important is that running the script twice will still give you right values. From a practical point of vue it is the same as idempotency, which is what matters.
- pbreit 6y agoThe idempotency I’ve seen is usually an unnecessary extra complexity.
- firebaze 6y agoWe had one "special" team member who insisted on everything being idempotent. This was his only leading principle. Result: absolute chaos - the code aspired to be idempotent, but due to idempotency he avoided thinking problems through and just created a mess of individual functions - each being idempotent, aside from the unavoidable bugs - which didn't form a coherent flow at all. We did a major refactoring, threw out about all that code, rewrote everything in a logical manner. Now everything is still idempotent, but comprehensible. TLDR: idempotency is the same snakeoil as the majority of guiding principles: alone, it doesn't help at all. There are lots of other factors to consider, which make the developer/architect role demanding (and fun). Craftmanship at least, a sense for architecture (better) or understanding the whole picture of the requirements as a team of developers (best) is still required.
- longhairedhippy 6y agoI don't see this as any reflection on idempotency as principle (or other principles in general). Building systems poorly, without a plan, and no testing, will result in a bug-riddled mess, regardless of what pattern is being used.
- spaetzleesser 6y agoThat is a general problem with any principle used as rigid ideology. Almost very principle becomes a problem if applied too dogmatically . This applies to software dev but also others like politics or economics.
- smitty1e 6y agoSure, one little hobby horse, e.g. "inversion of control" can run amok to negative effect (looking at you, Java projects with object traces 75 layers deep) but that doesn't make idempotency or inversion of control into bad ideas. A bit of pragmatism goes a long way, like Python's odd x = (some_tuple,) . . .syntax amidst its generally clean approach. Inflexibility itself is the bugaboo.
- bob1029 6y agoI feel like immutability and a 100% leak-proof layer of domain methods which in turn manage domain mutations would ultimately bring more value than explicitly adding idempotence throughout. If I have an idempotent method like "CreateCustomerRecord", this can cause a lot of pain for audit features and other aspects of the domain model if it is internally making determinations about whether to actually create or silently skip creation. For me, I would much rather that the method throw an exception if there is a duplicate business key than have it silently complete without taking any actual action. Exceptions indicating attempts at invalid state transitions can be extremely valuable if you have the discipline to create & use them properly. Generally, seeking idempotence in otherwise mutable methods is a band-aid for when you have broken immutability rules and allowed things to leak out of the sacred garden of unit-tested state machines and other provably-correct code items. If you should only conditionally execute some method, perhaps the solution is to investigate the caller(s) of the method, rather than attempt to infer the intent of all possible callers within the method itself.
- eweise 6y agoYou have the caller pass in a unique ID to the CreateCustomerRecord. Don't create a new customer record if you receive a duplicate ID.
- jayd16 6y agoThis can create false negatives when a request must be retried due to network failure but actually succeeded because the failure was during the response. Idempotency is great for "debouncing" requests. If you want to tell difference between identical requests that are different transactions, add a unique transaction id of some kind.
- adrusi 6y agoThere's a place for idempotency tokens. They're relatively easy to retrofit onto old systems, and occasionally they are the best way to go about making changes idempotent, but they should be a red flag – an indication that you should step back and see if maybe you can redesign an API to make idempotency a natural guarantee rather than something you artificially strap on with a token. As a rule of thumb, I would always mention the idea of adding an idempotency token, and prompt for alternatives, with all stakeholders present. (In this particular example, I agree that an idempotency token is probably the way to go, as otherwise callers would need to somehow distinguish between their callers trying to create a duplicate account, versus something going wrong with them resulting in duplicate requests. I just have too often seen developers conflate "idempotency" with "use an idempotency token)
- telekid 6y agoIn general, you should be thinking about the delivery semantics of the systems calling your code. Many very useful callers offer "at least once" delivery guarantees, implying that your system should behave idempotently to their calls.
- tigger0jk 6y ago> 1. Query the database to find all dormant accounts with a balance, which haven't been charged the fee this month. > 2. Charge each of these accounts a fee > 3. Setup a cron job to run this every hour Note that if this job ever runs successfully, but takes more than an hour, you will double-count. Can easily happen if the box running these crons is overloaded. One fix is to automatically halt the job after 55 minutes, another would be to have the middle step be impotent, for each user you're doing the process on, ensure (ideally in a threadsafe manner) that they need the operation to be done still.
- alex_young 6y agoSounds like a good reason to use a pidfile or mutex so you can eliminate the possibility of any concurrent jobs.
- jayd16 6y agoWhy is this an issue? Are you not running the query and recording the charge in the same transaction??
- tesseract 6y agoThe article didn't really go into the details of the implementation but my mental image was basically your second proposal: the deduction of the fee should happen in the same database transaction as the updating of a "fee-last-charged" timestamp, relying on the database for thread safety and allowing the cron job itself to be stateless (and inherently protected against simultaneous execution of multiple instances of itself).
- samatman 6y agoThis could still lead to a double-charge, though. If some rogue deploy script creates fifty of the cron job, and weirder things happen every day, one of those could be checking the stale state during the transaction of another one. A classic data race. Eliminating this problem is left as an exercise for the interested reader...
- 6y ago
- jchw 6y agoThis is good but not enough. You also need to be sure that you can’t charge twice if the job runs twice. When you do that same query twice, you will get the same list of users. This could be done by exploiting database consistency rules, like using strongly isolated transactions. One simple more general approach is to use an idempotence token. You could, say, have a table with a uniqueness constraint, and generate IDs that will match for the same user in the same month. Then add that in the same transaction that subtracts the money. The table could be cleaned up periodically. If you’re making or using an API where repeating would be bad, consider using idempotency tokens for those too. I believe Stripe supports them. The basic idea is the same: if you pass a token into them, they will guarantee that in a certain time frame, no other requests with that ID can be duplicated. This is useful when the network flakes during the response. Is it safe to retry? Things get trickier when you combine network and database consistency measures; that’s when you get into locks and multi stage commit and etc. and it helps to know your database’s consistency model, since it’s often not as solid as you think! (In the past, even PostgreSQL had issues with providing serializable isolation.)
- jbluepolarbear 6y agoHave you used a iterator as the token? I had data that could be accessed and mutated from multiple different sources at the same time. With just a token and blocking the data caused race condition (cross server race conditions/deadlocks are just the worst). I solved this by giving an iterator with every read and a write required the same iterator back with the changes. If 2 servers try to write at the same time the first processed will go through and the other will get rejected. The rejected server will reread the data with a new iterator, apply its mutation, then attempt to write. It worked really well for me and I never had any race/deadlock issues.
- glic3rinu 6y agovector clocks?
- jbluepolarbear 6y agoI’m not familiar with Vector clocks. If it’s the same concept it’s purely by chance.
- ibirman 6y agoWatch out for edge cases. What happens to accounts that change at 11:35PM on the last day of the month?
- Pxtl 6y agoI handle the deployment pipeline at my office and I've got a hard rule that every step in the CI chain has to be idempotent. Considering how many points of failure there can be in CI, being able to retry without undoing the mess of the previous deployment is a lifesaver.
- eric4smith 6y agoYears ago when I was a green programmer filled with p*ss and vinegar - I had to write a system to send a few million emails every day. That’s when I came across Qmail and studied it’s design. Qmail was designed so well that it literally absolutely would not send an email twice. But my scheduler did. To send a few hundred thousand emails more than once is not good is not good for your reputation lol. It taught me a valuable lesson about Idempotency from an impressionable stage of my career and now, with creating workers running under Sidekiq, Oban and other job schedulers that run very frequently, this has become quite useful. At one point I used to lean on Redis to “help” me be more idempotent. But with the high throughput of some systems I was working on, even redis with its atomic nature of put and get and handy dandy sets and lists, would have race conditions. Now I hardly touch redis because my skills in making sure things get processed just once have improved greatly. And I can just use PostgreSQL and my code. Being Idempotent from the beginning is now the center of my life.
- Rule35 6y agoI feel that all of the protections let us chase the problems into smaller areas. Rust's unsafe doesn't eliminate unsafe code, it just means you put it in a small auditable area. Similarly, some part of the system remains imperative. The network card at least, will always resend a packet. The goal is to pass around idempotent messages except for the very leaves. For instance, instead of an endpoint 'email(customer, data)' you might have 'email_or_report_on_send_status(customer, data)' and the later endpoint would check the cache for (customer, data) and merely report the previous results if it found them. I agree though, this stuff used to keep me up at night and eventually I've grown more natural about not mutating things unless I mean it. (This phrase sounds like a comic-book villain.)
- kyrra 6y agoGoogler opinions are my own. I work on payments stuff. Payments at Google defined some specs for payment companies to build to, which allows easy onboarding as a form of payment on google's platform. We have a page talking about idempotency and expected behavior. https://developers.google.com/standard-payments/reference/idempotency https://developers.google.com/standard-payments/reference/id... I don't think it's that different from other payment companies, like adyen or worldpay, but we attempted to explain our view of idempotency on a single page.
- speedgoose 6y agoSorry if it's a bit off topic but are you required by your employer to tell us you work from them and whether your opinion represents them or not? I read such disclaimers a lot about people working in your company, but extremely rarely for any other company.
- Rule35 6y agoHN is more defensive about some companies than others. The F and G of FAANG mostly. Edit: Even mentioning the rule to someone who asked can bring out the haters.
- MertsA 6y agoAnother tech giant engineer here. All too often random opinionated comments get taken way out of context and misconstrued as something pertaining to the company they work for. Separate from any NDA concerns, no one wants to have some reporter quoting them as "Google insiders reveal targeting of Republicans". I can't speak for Google employees, but I certainly don't have to preface anything with "this is my own opinion", but I certainly won't say anything that implies that I'm voicing some stance or opinion of my employer.
- kyrra 6y agoGoogle has some wording in our contract thst says we are required to put a disclaimer when we're posting publicly, I don't remember the exact wording, but I always include it on any topic that is even tangentially related to the company. This is actually a lot better than previous companies. My employee contract when I was at Cisco, basically said I couldn't talk about anything Cisco related on the internet without sign off from upper management or PR or something.
- munificent 6y agoGreat article. This is really about a mental shift. When given a problem like "dormant customers should be charged", it's natural to model the problem as verbs: "find the dormant customers and charge them". However, that description relies on implicit state: it assumes a world where all dormant customers have not been charged. As we all know, relying on hidden implicit state is the source of all evil. When you describe your problem using verbs, you are essentially modeling it in terms of a diff between how the world is and how you want it to be. But that diff is only correct if you have a perfectly correct description of how the world currently is. It's better to take a step back and think of your problem purely in terms of the world you want to end up in as a result: "all dormant customers have been charged". Framing it that way makes questions like "which dormant customers have already been charged?" more obvious to consider.
- musingsole 6y ago> think of your problem purely in terms of the world you want to end up in as a result Great framing! Isn't this the difference between imperative and declarative statements/code/systems?
- zambal 6y agoA declarative programming language should indeed allow you to think mostly about the result you want and not much about the how, but I'd say GP's comment is more about the design phase. The actual implementation could still be highly imperative.
- Wistar 6y agoYour summary is the second application of understanding paint: the clarity coat, the second view into the problem space.
- Rule35 6y agoPerhaps the rule is "charge all customers - active at rate X, dormant at rate Y". And then the question adds state, "as of yyyy/mm/dd, which customers were more than $30 into credit?"
- interstice 6y agoPossibly way OT but this reminds me of a question I had. Lets say I have a task that creates a daily invoice that failed to run for X days, but when it's run again it needs to go back and create X invoices from when it was last run. Is there a name / concept for that?
- catlifeonmars 6y agoThis is closely related to commutativity, which is another useful property of a system, especially a distributed one, where events commonly arrive out of order.
- mateo411 6y agoAlso associativity, which is more than two events occurring in any order. Interesting I know floating point arithmetic is not associative, but I think it is communitative.
- sempire 6y agoWhat if it takes more than an hour to get through the initial list of customers?
- jbluepolarbear 6y agoI had to solve this for charges and interest and I solved it in a more elegant way than described (I think). Using mutation iterator, last update time and knowing the rate of change over time you can back apply any any calculations that have been missed regardless of how long it’s been since your cron job ran. I setup a server to run the job on startup and 1 time a day at midnight. This also allows multiple different charges and interests to be applied to the same account. I also set it up to apply this process every time an account was accessed to make sure the accounts were up to date.
- nikolay 6y agoIt's 2021 and we still don't have scalable computer systems with continuations support - that's why his cronjob does not work reliably in the first manner if that script crashed or the machine reboots, it will run again. The idempotence is nice and I use it as much as possible, but, his case is very simplistic. Imagine you have a complex data migration process, which crashes in the middle - how you safely continue from where you crashed or safely roll back?
- stevage 6y agoSOmetimes you can write the migrations as a set of idempotent operations: "If table doesn't have column X, add it. If row Y doesn't exist in table Z, add it." etc.
- nikolay 6y agoWell, I did not mean a sequential one and only involving the database, but, let's say, you have forks (i.e. parallel executions) and merges (waiting to sme dependent tasks to complete), etc. What if execution halts in the middle of a step and you don't really have transaction support outside of the DB? I typically would use a workflow engine, but rollback is not something workflow engines support either.
- emodendroket 6y agoGreat idea if you control all the inputs (I'd also suggest passing in a time instead of relying on time.Now or whatever your language gives you). Sometimes you don't and it's very hard to do.
- jbluepolarbear 6y agoEvery system I write uses some sort of time object/function that can be set to whatever time input is desired. In a lot of cases it’s just the system time with an offset from some synchronization service. It makes testing easier, makes replaying inputs possible, and can make time related features based off local or server time with only changing inputs.
- stevage 6y agoTotally agree. I wish UPSERT had been invented earlier, or was a more canonical part of SQL than INSERT and UPDATE for this reason. Also, in web front end programming, there are lots of cases where X needs to cause Y to happen, but Z also needs to cause Y to happen. It's much much simpler if Y is idempotent, rather than X checking if Z has already happened etc.
- MereInterest 6y agoI'm going to make an absolutist statement for which I will probably be corrected shortly. For any given table, interactions should either be done entirely with INSERT or be done entirely with UPSERT. The two should never mix on a single table, and UPDATE should never be used. My reasoning is that if a table represents the current state of the world, then any state changes should be made with UPSERT in order to bring the table up to date with the world. If a table represents the history of changes, then that history should be appended to with INSERT, but not modified. I'm sure that there are other cases that I'm not currently considering, but I'm also a newbie at SQL and would love to be told of them.
- DharmaPolice 6y agoI'm probably being stupid here but the majority of updates only involve a limited number of columns. How would you translate a statement like 'UPDATE customers SET surname='Smith' WHERE customer_id = 1234' into an UPSERT?
- MereInterest 6y agoI would argue that your SQL statement should match the information that a user put in. It would be pretty rare for a user to directly input a single field change and their id. More likely, there would be some "personal information" submission, where all fields are available to edit. I would have a separate "customer_personal_info" table that gets UPSERTed with the entire contents on the submission. One impact of this is that is forces the changes to be atomic. If an object-oriented interface updates a database whenever properties are changed, then it results in many single-column changes. In the example below, if there is a network failure between the two commands, then changing the name from "Jane Doe" to "John Smith" could result in an unintended intermediate state of "John Doe". customer_obj.set_forename("John") customer_obj.set_surname("Smith") (Again, I am by no means an expert, and in part posted this so that I can be corrected if/where I am wrong.)
- rexarex 6y agoEven better would be to have the hourly cron job add the accounts to deduct the balance from to a queue and have worker services consume those queue messages and do the deduction and database write.
- gregmac 6y agoI have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notification and save it to the list. "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes. The nice thing about this is it doesn't stop you from also doing the event-based stuff (for "real-time" notifications, for example), so long as it's the secondary approach from design point of view. Ironically when everything is working fine, the event-driven stuff will actually be doing all the work -- but as soon as there's a failure (like, the system happens to reboot at 23:59) you'll be grateful to build this way. I could write paragraphs about the nuance, different approaches, failure modes and the trade-offs with all this -- suffice to say treating events as optional in event-based systems has served me well.
- harlanji 6y agoApparently if you want spare cash, little books of this stuff could sell for $5-10. Just became aware myself, setting up to write again. That’s to say, this comment probably just saved or made me money. I’ve some hands on but clearly not as much as you exhibit. Thanks. Target audience: Sr SE, reporting to next level up and perhaps offering tradesoffs, or advising peers or reports working on those systems and need a refresher to give a complete answer to a question.
- pjc50 6y agoHow would you market those? It's been a very long time since I bought a book on software.
- layoutIfNeeded 6y agoIf this worked as a business model then everyone would be doing it. Everyone also includes people who don’t actually know this stuff, but want to make a quick buck, giving out bad advice. And if you don’t already have the experience, then you don’t have the knowledge to weed out bad advice.
- lamontcg 6y agoDeclarative idempotency is even better. #1 : collect the state the "user" (might be you) desires the system to be in. #2 : collect the state the system is actually in. #3 : compare #1 to #2 and see if you're already there and if you are, you're done. #4 : if they differ, then do the work once to fix them. #5 : bonus: go collect the system state again and send it out to report on what changed
- vishnugupta 6y agoIt's a pity the author doesn't get into implementation details as it's quite fun for that specific example because it involves payments. A couple of key aspects are: 1. The payments API must be idempotent for the Cron to be idempotent. Otherwise they can't safely retry timeout failures. 2. The operations <charge-the-customer> and <mark-the-customer-as-charged> have to be (to loosely use the term) atomic. For #1, a payment processor should explicitly state idempotency semantics. Look at Stripe [1] for example. They require you to pass Idempotency Key header in charge request. Clients must carefully choose this key to fit the needs for their business flow and use case. For the author since they charge monthly the idempotency key must be such that it should be constant for a given customer and given month. So something like "customerid-YYYY-MM" makes sense. Now no matter which day of the month cron runs the customer is guaranteed to charge at most once. As for #2 most of data stores now a days support atomic operations so it should be reasonably straight forward. Get-and-set in Redis, start/end transaction with "for update" in MySQL etc., Note that #2 is optional if you don't mind spurious retries to Stripe. Stepping back from the details it can be noticed that for a system/operation to be idempotent all its downstream dependencies should support idempotent operations. In this example, if Stripe weren't idempotent then you have to resort to manual fixes if charge operation times out. Another observation is, if a downstream system can assure idempotency then don't bother making your operation idempotent. Failed payment? Retry. No need to store if a payment was attempted or not. Make sure the idempotency-key generation logic is robust. PS: Source - I've been in payments industry for a while so spend a lot of time thinking about and implementing idempotency. The latest one being when my team worked on payments API at Uber. We included a idempotency in the glossary [2] section ;-) [1] https://stripe.com/docs/api/idempotent_requests https://stripe.com/docs/api/idempotent_requests [2] https://developer.uber.com/docs/payments/glossary https://developer.uber.com/docs/payments/glossary
- k_sze 6y agoI think an even higher level of thinking is to think thoroughly about the ways the system can fail. Idempotence just happens to be _one_ way of dealing with certain failure modes. As an engineer, the more exposure and experience you get, the more insight you have about the ways things can fail. Identifying the ways something can fail is the really important step here. You can’t know what failsafe to implement if you don’t actually know how something can fail. But once you do know how something can fail, implementing a proper solution is easy a lot of the time, even if you are not explicitly aware of the concept of idempotence, for instance. Only in some tricky Byzantine edge cases do you need very specific, well-established, track-proven algorithmic solutions.
- ronenlh 6y agoThis is my ideal type of article on HN. A namable concept, applicable, and without trying to be subversive. It was immediately define, had a real world example, and was short and well written.
- chmod775 6y ago> 1. Query the database to find all dormant accounts with a balance, which haven't been charged the fee this month. > 2. Charge each of these accounts a fee. There's still a race condition between 1. and 2. if you don't atomically check the account hasn't been charged yet as you're charging it. With long running processes like these it's also pretty much guaranteed to happen if you accidentally run two instances of this process at the same time.
- juanuys 6y agoJust my 2 cents: Impotence now also prevents pain later: having kids is such a pain in the arse.
- pc86 6y agoI really don't want to go on a totally off-topic tangent, but I feel like this needs to be brought up. > We need to charge dormant customers a monthly fee so that we don't have to keep their money on our accounting books forever. That's absolutely and objectively not the reason this happens. Keeping their money "on [your] books" has zero cost associated with it outside of the escheatment[0] process, which can be completely automated. I worked at a company that had to do a decent amount of escheatment for almost every US state, Canada, and Mexico, and for a total company size of under ten people the entire organization probably spent 45 minutes working on it in any given month. It's a trivial amount of time. This happens to that the company can turn it's customers' money into its money. I don't know enough about to escheatment process to know for sure but I'd imagine regular debits like this would also get around the need to return the money to the estate after a given period of time. So someone puts $500 in their account, gets hit by a bus the next day, and the company gets to take a little bit every month/quarter/year until it's all theirs. It's a pretty scummy thing to do and I'm surprised the author just takes it at face value that they "have to" do it. [0] https://en.wikipedia.org/wiki/Escheat#Transfer_agents_and_escheatment https://en.wikipedia.org/wiki/Escheat#Transfer_agents_and_es...
- seaerkin 6y agoReally awesome, simple example.
- MrWiffles 6y agoI somehow read this as “impotence now prevents pain later”. There’s a hole in there about parenting somewhere