3 ms·
The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as ma
by simonw 9y ago
The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position.
As a producer, you just need to know that your message has definitely been delivered to the log. What consumes it is none of your business.
Have you read https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... ? It's one of my favourite essays on software architecture of all time, and it goes deep into this concept.
- markbnj 9y ago> The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position. Right I think we're saying the same thing. In the article checkpointing was introduced to enable safely managing the size of the log, i.e. it appears that in this use case _somebody_ needs to know whether the consumers are caught up in order to adhere to at-least-once semantics. It just seemed to me that if you reach the point where you're rolling your own to solve that problem there are other possible solutions. I haven't read the linked-in piece yet but I intend to. It was also linked in the article. Thanks for linking it here.