Good tips, on that note you really don't have to use HBase or some always-on managed filesystem. You can actually use S3! I know this sounds weird as you'd rack

Good tips, on that note you really don't have to use HBase or some always-on managed filesystem. You can actually use S3! I know this sounds weird as you'd rack up API costs way faster than storage cost, but we implemented a simple batching approach for timeseries data that could handle 100M+ messages for $10/day (over 100GB+ of data, includes all costs, processing, storage, backup). Did a 2 min screen cast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8 . This hopefully can be a lifesaver on costs for some people, and keeps your system overall simpler.