5 ms·
Ask HN: How to make server side code atomic?
Imagine the following:
1. You submit some value to the DB
2. After that value is submitted, you need to do a POST to an email API
Your server could crash between steps 1 & 2, resulting in the email never being sent.
What is the common accepted solution here? The only thing I can think of is temporal.io, but many, many pieces of software have been built before temporal.io.
- Fire-Dragon-DoL 5y agoWelcome to distributed systems. The answer is "you can't, you can only increase the chance of the correct thing happening". It boils down to choosing between: - possibility of sending 0-1 emails - possibility of sending 1+ emails (in reality is 0+,your service could still go down permanently)
- deleted 5y ago[deleted]
- cyco130 5y agoIf sending the email is mission critical you could start a transaction, update the DB, send the email, commit the transaction. Maybe slightly better is: start the transaction, update the DB, put the "send email" job in a job queue, commit the transaction. And the job queue takes care of sending the email, retrying on fail, notifying the admin if sending the email keeps failing etc.
- gabereiser 5y agoI was going to write this as this is how I would do it but you beat me to it. +1
- mustardo 5y agoFor your "maybe slightly better is" what happens when the db commit fails? FK constraint or some such, now you have sent an email (called the API) and have no record in your DB Without two phase commits I would 1) Write to the DB in the original POST request a record indicating that an email should be sent 2) Process that record outside of the original request and mark it as sent
- cyco130 5y agoI assumed the job queue was on the same database so either both would succeed or both would fail. Your should_send_email flag effectively corresponds to my job queue.
- vsroy 5y agoThere's one hole here where it feels like things could slip through. - You call the send email API but your server crashes before the API returns 200. This would be fine in the case of email servers (you just end up sending the email twice). It seems a lot less fine in the case of Stripe transactions. I guess, the concept of idem-potency keys can solve this issue. (https://brandur.org/idempotency-keys https://brandur.org/idempotency-keys)
- moritonal 5y agoGotta love the state of today where "I need to solve a basic state-execution flow" translates to "Let's pay a third-party service to do it". Almost makes me forgive the interview process tech-companies demand just to check dev's can actually cover the basics.
- c0wb0yc0d3r 5y agoTo be fair, I haven't come across an interview where knowledge like this is checked. Which is sad, this is more likely a bit of information a dev would need to know, instead of inverting a binary tree.
- pacifika 5y agoBackground queue pulling from the db and pausing / retrying failed jobs
- theshrike79 5y agoThe correct solution is to make this event based and not sequential. You have service A that checks if it needs to send an email. Service B submits a value to DB and adds a task to the email system inside a transaction. If the DB is down, the service will retry later. If the email service isn't responding, it can roll back the DB change.
- skybrian 5y agoI don't know how it's done now or what APIs support, but here's an old-school design: During the first database commit, set a field indicating that the associated email is pending. After sending it and getting a success, do another database commit setting the flag to sent. You will need a server-side task, maybe a cron job, to periodically query the database for pending emails and attempt to resend them, then set the flag to sent. Also, hopefully the API you use to send email has a way to avoid duplicate emails, perhaps by attaching a unique ID to the request that's used to detect duplicate sends.
- vsroy 5y ago> Also, hopefully the API you use to send email has a way to avoid duplicate emails, perhaps by attaching a unique ID to the request that's used to detect duplicate sends. I think this is the interesting part. Storing a value in the table to be processed by a queue seems normal / not rocket science. But what happens when we 1. Call the API 2. The API succeeds 3. We crash before we can write success Now we'll send duplicate emails. I don't think all APIs have ways to avoid duplicate emails/transactions, so that could be a hairy problem.
- skybrian 5y agoUnless you use an API that supports duplicate request removal, either you drop some emails (at-most-once delivery) or sometimes you send duplicates (at-least-once delivery). For email, I think the occasional duplicate is usually better. (Effectively, it's up to the email recipient to ignore the duplicate.)
- mping 5y agobegin transaction; write to db; post /api commit; Assuming rollback is implicit when an exception occurs, failure at any point results in the whole operation not executing.
- cjg 5y agoWhat happens when the REST server accepts your POST to /api and starts processing it but then, before the response can be sent, the connection drops? This code would see an error from the POST to /api due to a connection failure / timeout, and the transaction rolls back. Now you've got the /api invoked, but no DB write.
- tompazourek 5y agoTry searching online for "outbox pattern", I think you'll find your answer there.
- wizwit999 5y agoSQS
- derekperkins 5y agoThis is a great article going into more specifics. https://brandur.org/idempotency-keys https://brandur.org/idempotency-keys
- vsroy 5y agoThanks! This was a good article. I guess there's a question of what to do if the API does not have idem-potency keys, (maybe it's possible to wrap it?)
- derekperkins 5y agoYou clearly didn't read the article, that's what the whole thing discusses
- vsroy 5y agoI'll take another look. I skimmed the article to get an understanding of idempotency keys without reading it in full detail.
- vsroy 5y agoI see a few solutions in the comments: but one thing that is not acknowledged is: "What do you do if the email service executes successfully but your server crashes before the POST is returned". I guess this isn't the worst thing in the world since that just means you send the email twice. Edit: Actually the worst case is pretty bad, you could accidentally double-charge a user using Stripe.