We use kinesis firehose and Athena as a SQL API for customer interactions analytics on our digital properties (among other things). Data sizes are upwards of 10

We use kinesis firehose and Athena as a SQL API for customer interactions analytics on our digital properties (among other things). Data sizes are upwards of 10M rows per day. Athena will blow through months of data pretty quickly at very reasonable cost. Some gochas: AWS lock in. Especially since S3 is your storage back end. Kinesis firehose writes files by insert date and you may need ETLs to partition by some other time dimension inherent to the data. This gets tricky if stuff comes in really late and you may have to decide to drop records older than N days to make such an ETL work at reasonable cost. Some pros: easy to POC. Pure variable cost architecture. Serverless (also a downside because if any of the services go down so do you) with little administrative work. It can get expensive if you don’t think through partitioning for analytics, but Kinesis firehose can be configured to write parquet to S3 natively allowing you to use existing spark/hive/whatever if your org has that as your SQL layer. Edit: come to think of it, there is another division in my company using bigquery with whatever google’s streaming ingestion product is. They’re achieving similar results at similar cost. I haven’t personally done development in GCP though so I couldn’t comment