3 ms·
What is then, in your opinion, an actually good ETL stack? Many use Spark/Databricks/Synapse, which has all 3, but these seem bulky and needlessly complicated f
by pletnes 4y ago
What is then, in your opinion, an actually good ETL stack? Many use Spark/Databricks/Synapse, which has all 3, but these seem bulky and needlessly complicated for many smaller projects. Even TB sized datasets don’t need Spark, in my opinion.
- RobinL 4y agoI'd advocate starting as simple as possible. It's very context specific, but one very simple architecture to consider if seeing if duckdb can handle your data (maybe on quite a large machine). For larger data, i'm a fan of writing everything in SQL, and executing it using AWS Athena (a Presto engine), which is extremely cheap. If you have even larger data, you can consider AWS Glue (steering clear of any aws specific stuff and literally using it as spark SQL as a service). For many simple cases you don't really need to know any Spark, and you're just submitting SQL to AWS for execution. But all of this assumes you're storing your data as files on disk (e.g. parquet). I'm less experienced in running everything in more traditional SQL engines like MS SQL server or postgres.