3 ms·
I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow bo
by gdulli 10y ago
I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow boilerplate.
I imagine I'll eventually need to add some sort of management system to move these dynamic jobs in and out of Airflow to keep them from bloating the database or cluttering the UI.
- tedmiston 10y agoI'd be interested in hearing more about your project and if you plan to open source. Are you using something like Cookiecutter or Yeoman, or a different level of abstraction?
- gdulli 10y agoMy effort is a collection of custom operators for operations that we use very commonly (run a query, python script, or mapreduce job) and a code generator that takes a simple text spec describing a set of operations and generates the Python DAG script. I'm not familiar with Cookiecutter or Yeoman but generating Python code myself hasn't been complicated. Open-sourcing it isn't an option right now with my current employer.
- tedmiston 10y agoThanks, this sounds really interesting. If this changes or you decide to write personal code with similar structure, I'd definitely like to discuss further. I'm also currently exploring DAG structure but with class and function abstractions with the goals of increasing code reuse and minimizing boilerplate around our custom operators. Cookiecutter takes a Python code file as input with Jinja interspersed (the template). When you want to make a new instance from the template, it gives command-line prompts, and then evaluates the Jinja logic — custom variables, loops, etc to output Python code. It took me 15–30 minutes to get started and has already paid off.