The downside to Athena/PrestoDB is you need to get the data into an optimal format/file size to get the best performance (e.g. Parquet or other columnar format)

The downside to Athena/PrestoDB is you need to get the data into an optimal format/file size to get the best performance (e.g. Parquet or other columnar format) This is fine for batch workloads where 'real time', i.e. latency of < 5 minutes, ideally < 1 minute, isn't much of a concern. but adds complication if you need to make data available quickly, because you either adopt some hybrid approach where you have 'recent' data in database and everything else in S3 - meaning your query layer has added complexity. I believe this is how BigQuery works with streaming inserts, where recent streamed data is actually stored in BigTable, and asynchronously copied into Capacitor over time (this might be outdated information) So while BigQuery seems expensive, the other solutions have a lot of other costs that you need to factor in!