4 ms·
Core is SQL + Python (or Java/Scala for the actual Engineering part). Most of the data warehouse work is data transformation and scheduling, so Apache Airflow
by beckingz 4y ago
Core is SQL + Python (or Java/Scala for the actual Engineering part).
Most of the data warehouse work is data transformation and scheduling, so Apache Airflow and writing DAGs in python is a large part. For some of the larger streaming things you've got Kafka and others.
Also every cloud has it's own stack that is sometimes worth using.
- bogomipz 4y agoThanks, what's the name of the AWS stack? Is it worth using?