2 ms·
You do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
by sewen 11y ago
You do not need physical exactly once. If you can "roll back" changes, than physical "at least once" is enough to realize logical "exactly once".
- falcolas 11y agoRolling back changes means that at some point in time, you have "more than once". It also means you can only ever implement logic which can be rolled back. In a pipeline scenario, that is challenging to do without some sort of external locking system.
- sewen 11y agoTrue. In most streaming systems, exactly-once refers to the state that is maintained and controlled by the streaming system. For that, Flink can guarantee such semantics. That state can also be exposed externally, see for example here: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ http://data-artisans.com/extending-the-yahoo-streaming-bench... For interactions (writes) with external systems, you need transactional integration. There is WIP to do this for transactional databases, for non transactional systems, there are cases where it cannot be strictly solved.