5 ms·
Some amazingly cool stuff that any engineer would want to be part of! Could anyone answer a few questions? 1. Is the Spark cluster running on top of an in-hous
by alstream 12y ago
Some amazingly cool stuff that any engineer would want to be part of! Could anyone answer a few questions?
1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3?
2. Any more details about the custom message system, specifically how to guarantee exactly once delivery?
3. How do you guarantee data is never dropped?
4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in?
Again, thank you for the insights!
- apple-fan 12y agoThank you! Let me try to address those questions: > 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3? It runs on an in-house cluster. > 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery? This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery. > 3. How do you guarantee data is never dropped? We write each data point into 2-3 locations in our system and also back it up in S3. > 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in? Basically, batch things and keep a lot of signatures and checksums around.
- alstream 12y agoThanks for your time! About point 4, I'm assuming you built in idempotency into your message replay system. Is your unit of idempotency, say, 1 hour or some other unit of time, or is every single message have idempotent results? Because the latter would be pretty amazing :)