4 ms·
I'm pretty sure that this is the algorithm flink uses: http://arxiv.org/pdf/1506.08603.pdf http://arxiv.org/pdf/1506.08603.pdf EDIT: fyi sewen is one of the au
by faizshah 11y ago
I'm pretty sure that this is the algorithm flink uses: http://arxiv.org/pdf/1506.08603.pdf http://arxiv.org/pdf/1506.08603.pdf
EDIT: fyi sewen is one of the authors of the paper and a founder of flink
- sewen 11y agoYou do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
- falcolas 11y agoRolling back changes means that at some point in time, you have "more than once". It also means you can only ever implement logic which can be rolled back. In a pipeline scenario, that is challenging to do without some sort of external locking system.
- sewen 11y agoTrue. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ http://data-artisans.com/extending-the-yahoo-streaming-bench... For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.