4 ms·
> there is no foolproof way of implementing that pseudocode's "has_seen(message.id)" method Wait why? Just because you'd have to store the list of seen message
by kmicklas 9y ago
> there is no foolproof way of implementing that pseudocode's "has_seen(message.id)" method
Wait why? Just because you'd have to store the list of seen messages theoretically indefinitely?
- convolvatron 9y agosure, if you assume that everything can fail, then it doesnt help to store the list of messages you've seen but if you can persist a monotonic sequence number, thats gets you pretty far. we use tcp all the time even though its has no magic answer to distributed consensus (and uses a super-weak checksum). 2pc doesnt guarantee progress and/or consistency either and its pretty effective.
- sethev 9y agoThere's also a race condition in there when you receive the duplicate before publish_and_commit is done doing its thing - assuming they're not actually serializing all messages through a single thread like the pseudocode implies. What they've done is shift the point of failure from something less reliable (client's network) to something more reliable (their rocksdb approach) - reducing duplicates but not guaranteeing exactly once processing.
- dastbe 9y agoits not so much that they are serializing all messages through a single thread, but that they are consistently routing messages (duplicates and all) into separate shards that are processed by a single thread.