I've had great luck using either Athena or DuckDB with parquet files in s3 using a few partitions. You can query across the partitions pretty efficiently and if

I've had great luck using either Athena or DuckDB with parquet files in s3 using a few partitions. You can query across the partitions pretty efficiently and if date/time is one of your partitions, then it's very efficient to add new data.