4 ms·
I hate YAML based ETL solutions that uses jinja to render YAMLs to Airflow DAGs, with a passion.
by hnthrowaway0315 3y ago
I hate YAML based ETL solutions that uses jinja to render YAMLs to Airflow DAGs, with a passion.
- verdverm 3y agoCan you add a link to an example of this? Working on another Yaml post with a dedicated section for this
- hnthrowaway0315 3y agoSorry they are all company resources so I cannot share a screenshot. But imagine writing dags using yaml, so you would have a field called "dag_id", one called "schedule", things like that.
- verdverm 3y agofair enough Yeah, declarative DAGs 101 lol(?) Does python code get generated from the Yaml, or does python code load the Yaml and do Airflow tasks? I never thought specifying DAGs in airflow / python was that difficult. What problem is this solving? I have a thing that does CUE + text/template = anything maybe what you are describing is like the prior Yaml based input we used to use? [1] https://github.com/hofstadter-io/hof https://github.com/hofstadter-io/hof [2] https://docs.hofstadter.io/getting-started/code-generation/#data--templates https://docs.hofstadter.io/getting-started/code-generation/#...