16 ms·
Ok, I guess I am repeating myself but so does this post since I assume it spawned from this discussion https://news.ycombinator.com/item?id=40284225 https://new
by fifilura 2y ago
Ok, I guess I am repeating myself but so does this post since I assume it spawned from this discussion https://news.ycombinator.com/item?id=40284225 https://news.ycombinator.com/item?id=40284225
Why does no-one mention Trino or the AWS branded Trino - Athena?
This will solve basically all of the problems listed.
In the case of Athena and provided you already have your data in s3, without having to set up a new service and very little work.
Using Athena is not like setting up a new database. You just point Athena to the files and it will use resources provided by AWS to query. No import required, no service required.
I think Athena is an amazing tool. I can borrow like 120 CPUs (my tasks often parallelize to that) for free during 5 minutes or even an hour to join / group / modify all my data.
- wenc 2y agoAthena is great. It’s essentially querying your data using lambdas on your S3 data. DuckDB outperforms it slightly on a single node but Athena is a good choice if you need elastic scaling.
- fifilura 2y agoI would not compare it to lambda. It is backed by Trino https://trino.io/ https://trino.io/ and can do map/reduce kind of jobs that lambdas can not.