3 ms·
If you have the opportunity, please do not build it like this. Referring to the architectural diagram, it is going to be much more efficient for the "Frontend"
by jeffbee 6y ago
If you have the opportunity, please do not build it like this. Referring to the architectural diagram, it is going to be much more efficient for the "Frontend" to persist the task data into a durable data store, like they show, but then the Frontend should simply directly call the "Store Consumer" with the task data in an RPC payload. There is no reason in the main execution path why the store consumers should ever need to read from the database, because almost all tasks can cut-through immediately and be retired. Reading from the database should only need to happen due to restarts and retries of tasks that fail to cut through.
Disclaimer/claim: I worked on this system and on gmail delivery.
- mrfox321 6y agoIn other words, the frontend gets some ACK from the DB before calling the "Store Consumer"? (just trying to make sure I understand your critique of the design)
- jeffbee 6y agoWell it doesn't necessarily need to happen in that order, I think. Frontend needs to ensure that the task is durably stored before it acknowledges the end of the operation to its caller. Using email as an analogy, you have to commit the message to durable storage before you respond 250 to DATA.
- neolog 6y agoIf you do it that way, how do you make sure the task gets completed successfully and exactly once?
- jeffbee 6y agoI wouldn't. Exactly-once is a fool's quest, and the scheme in this article does not offer it. To achieve at-least-once, you need only track which tasks have been successfully retired, and persist that knowledge in the database by either deleting or mutating the task. During a cold start you scan the persistent store to find tasks that were still pending/live at the time your process began.
- dropofwill 6y agoMaybe I'm missing something, but how would this meet the requirement of scheduling tasks in the future?