3 ms·
What I'd like to understand at a glance about Storm is durability, idempotency, and specifically those two things in the face of all your standard failure modes
by newobj 14y ago
What I'd like to understand at a glance about Storm is durability, idempotency, and specifically those two things in the face of all your standard failure modes.
Is Storm something that is going to get 99.9% of my data there which is good enough for some side-channel processing like ad targeting, or is it something that's D durable in the strongest sense of the word? If I'm ultimately delivering data to something that is 5 9's durable, or better yet S3 10-11 9's durable, is Storm going to be my "low point" of durability?
- djb_hackernews 14y agoRead the wiki: "Guarantees no data loss: A realtime system must have strong guarantees about data being successfully processed. A system that drops data has a very limited set of use cases. Storm guarantees that every message will be processed, and this is in direct contrast with other systems like S4." https://github.com/nathanmarz/storm/wiki/Rationale https://github.com/nathanmarz/storm/wiki/Rationale
- mattmcknight 14y agoStorm supports transactional topologies that can provide durability. They are dependent on a spout that is repeatable- where you can request the failed input again. https://github.com/nathanmarz/storm/wiki/Transactional-topologies https://github.com/nathanmarz/storm/wiki/Transactional-topol...
- nathanmarz 14y agoActually transactional topologies are superseded by Trident, which is much easier to use: https://github.com/nathanmarz/storm/wiki/Trident-tutorial https://github.com/nathanmarz/storm/wiki/Trident-tutorial Here's an explanation on how Trident achieves fully fault-tolerant, idempotent semantics: https://github.com/nathanmarz/storm/wiki/Trident-state https://github.com/nathanmarz/storm/wiki/Trident-state