Thank you! Let me try to address those questions: > 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3? It runs on an in-house clu

Thank you! Let me try to address those questions: > 1. Is the Spark cluster running on top of an in-house cluster or on AWS EMR/S3? It runs on an in-house cluster. > 2. Any more details about the custom message system, specifically how to guarantee exactly once delivery? This topic might deserve a separate article. We have built a custom locking system that ensures proper delivery. > 3. How do you guarantee data is never dropped? We write each data point into 2-3 locations in our system and also back it up in S3. > 4. "Make sure every message can be re-played without data corruption" how to achieve this when there is so many data points coming in? Basically, batch things and keep a lot of signatures and checksums around.