5 ms·

I've used Airflow for a few years and here's what I don't like about it: - Configuration as code. Configuration should be a way to change an application's beh

I've used Airflow for a few years and here's what I don't like about it: - Configuration as code. Configuration should be a way to change an application's behavior _without_ changing the code. Make me write a workflow as JSON or XML. If I need a for-loop, I'll write my own script to generate the JSON. - It's complicated. You almost need a dedicated Airflow expert to handle minor version upgrades or figure out why a task isn't running when you think it should. - Operators often just add an API layer over top of existing ones. For example, to start a transaction on Spanner, Google has a Python SDK with methods to call their API. But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call. Sometimes the operator author makes "helpful" (opinionated) changes that refactor or rewrite the native API. I would love a framework that just orchestrates tasks (defined as a command + an image) according to a schedule, or based on the outcome of other tasks, and gives me a UI to view those outcomes and restart tasks, etc. And as configuration, not code!

ricklamers 4y ago
The flexibility of code as configuration is indeed somewhat of a footgun at times. That’s why with Orchest we went with a declarative JSON config approach. We take inspiration from the Kubeflow project and run tasks as containers. With a GUI for editing pipelines and managing scheduled runs we come pretty close to what you’re asking for (bring an image and run a command). And it’s OSS, of course. https://github.com/orchest/orchest https://github.com/orchest/orchest
KptMarchewa 4y ago
>If I need a for-loop, I'll write my own script to generate the JSON. That's how you end with extreme mess in logs, UI and metrics. > But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call. Or you can use PythonOperator with hooks, that generally integrate external APIs with Airflow connection system. https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/gcs.py https://github.com/apache/airflow/blob/main/airflow/provider... I think the bigger problem with Airflow's Operator concept is N*M problem of integrating multiple systems. That's how you end with GoogleCloudStorageToS3Operator and stuff like that.
atombender 4y ago
What you're asking for is basically Argo Workflows, I think. Not that I recommend it. It's quite lovely in principle, but really flawed in practice. It's YAML hell on top of Kubernetes hell (and I say that as someone who loves Kubernetes and uses it for everything every day). Having worked with some of these tools, what I've started to wish for is a system where pipelines are written in just plain code. I'd like to run and debug my pipeline as a normal, compiled program that I can run on my own machine using the tools I already use to build software, including things like debuggers and unit testing tools. Then, when I'm really to put it into production, I want a super scalable scheduler to take my program and run it across dozens of autoscaling nodes in Kubernetes or whatever. The only thing I've come across that uses this model is Temporal, but it's got a rather different execution model than a straightforward pipeline scheduler.
- leafmeal 4y ago
  Check out Toil: https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil
  - atombender 4y ago
    Looks Python-specific? I don't use Python.
- brylie 4y ago
  Prefect might work for your use-case: https://docs.prefect.io/ https://docs.prefect.io/
  - FridgeSeal 4y ago
    IMO the issue with things like Prefect (and the sinking comment suggestion of Toil) is the inextricable coupling with Python. We use Argo at my work, because we made the “unit of work” just a container that’s run by K8s. There isn’t a Python runtime around to run this stuff, and that’s a good thing. Same as with Pulumi, I’ve seen what devs do when you let them write arbitrary code anywhere-you end up with code that lives everywhere, confusingly cuts across different infrastructural layers with wild-abandon. It becomes difficult to on-board, difficult to maintain, the blast radius becomes unknown, and just because it’s Python, it will randomly break in 6 months because nobody pinned a transient dependency. YAML is awful. Argo could do with some trimming in its configs, but I’ll take it, and other “constrained” alternatives over arbitrary-code-in-Python-that-controls-resources-that-run-more-arbitrary code.
rtpg 4y ago
> If I need a for-loop, I'll write my own script to generate the JSON. OK, so you are using some service that expects JSON to config. You have a script to generate the JSON. Now you gotta make sure to run the script on changes. You also need to check in the JSON into your VCS despite it being a generate artifact. This leads to the perpetual "somebody fixes an issue in the artifact, and it gets wiped after someone uses the script" issues. What you're saying is theoretically very good! But it doesn't play well with the combo of assumptions made everywhere in hosted services and some PL infrastructure: - every project is 1 git repository, and vice versa - commit security is an all or nothing proposition (so it's hard to have permissions like "this file can only be edited by this user" or "this commit can be made directly to `main`, _if_ it only touches this file") - configuration as configuration - version numbers have to be written into code And obviously this is a problem, because everyone keeps on inventing YAML-based programming idioms for each of their services! Because static configuration doesn't map to even the simplest of setups very well. People come and them mention stuff like Dhall. But really people want to be able to write programs that read files, make HTTP requests, and then make decisions about what to do based on that. And things like "if this test suite fails, do this other orchestration". The ideal CI flow is not staticly definable at the start of execution! There is a killer protocol out there, and it's ... something like Skylark, with a standard library that gives file reading, checking of statuses, probably HTTP and the like.... and an executable that allows for configuring permissions in certain ways. The bonus would be just re-doing Bazel but with less guardrails. If someone showed up with CI/CD that was more in this vein they would become the dominate CD host very quickly in my opinion.
urthor 4y ago
Strongly disagree. IMO I think GUI variables are a big weakness of Airflow. I, simply, never want to be responsible for a typo in a prod config variable. Ever. I'm all for version controlling every, single, variable.

ricklamers 4y ago

The flexibility of code as configuration is indeed somewhat of a footgun at times. That’s why with Orchest we went with a declarative JSON config approach. We take inspiration from the Kubeflow project and run tasks as containers. With a GUI for editing pipelines and managing scheduled runs we come pretty close to what you’re asking for (bring an image and run a command). And it’s OSS, of course. https://github.com/orchest/orchest https://github.com/orchest/orchest

KptMarchewa 4y ago

>If I need a for-loop, I'll write my own script to generate the JSON. That's how you end with extreme mess in logs, UI and metrics. > But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call. Or you can use PythonOperator with hooks, that generally integrate external APIs with Airflow connection system. https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/hooks/gcs.py https://github.com/apache/airflow/blob/main/airflow/provider... I think the bigger problem with Airflow's Operator concept is N*M problem of integrating multiple systems. That's how you end with GoogleCloudStorageToS3Operator and stuff like that.

atombender 4y ago

What you're asking for is basically Argo Workflows, I think. Not that I recommend it. It's quite lovely in principle, but really flawed in practice. It's YAML hell on top of Kubernetes hell (and I say that as someone who loves Kubernetes and uses it for everything every day). Having worked with some of these tools, what I've started to wish for is a system where pipelines are written in just plain code. I'd like to run and debug my pipeline as a normal, compiled program that I can run on my own machine using the tools I already use to build software, including things like debuggers and unit testing tools. Then, when I'm really to put it into production, I want a super scalable scheduler to take my program and run it across dozens of autoscaling nodes in Kubernetes or whatever. The only thing I've come across that uses this model is Temporal, but it's got a rather different execution model than a straightforward pipeline scheduler.

leafmeal 4y ago
Check out Toil: https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil
- atombender 4y ago
  Looks Python-specific? I don't use Python.
brylie 4y ago
Prefect might work for your use-case: https://docs.prefect.io/ https://docs.prefect.io/
- FridgeSeal 4y ago
  IMO the issue with things like Prefect (and the sinking comment suggestion of Toil) is the inextricable coupling with Python. We use Argo at my work, because we made the “unit of work” just a container that’s run by K8s. There isn’t a Python runtime around to run this stuff, and that’s a good thing. Same as with Pulumi, I’ve seen what devs do when you let them write arbitrary code anywhere-you end up with code that lives everywhere, confusingly cuts across different infrastructural layers with wild-abandon. It becomes difficult to on-board, difficult to maintain, the blast radius becomes unknown, and just because it’s Python, it will randomly break in 6 months because nobody pinned a transient dependency. YAML is awful. Argo could do with some trimming in its configs, but I’ll take it, and other “constrained” alternatives over arbitrary-code-in-Python-that-controls-resources-that-run-more-arbitrary code.

rtpg 4y ago

> If I need a for-loop, I'll write my own script to generate the JSON. OK, so you are using some service that expects JSON to config. You have a script to generate the JSON. Now you gotta make sure to run the script on changes. You also need to check in the JSON into your VCS despite it being a generate artifact. This leads to the perpetual "somebody fixes an issue in the artifact, and it gets wiped after someone uses the script" issues. What you're saying is theoretically very good! But it doesn't play well with the combo of assumptions made everywhere in hosted services and some PL infrastructure: - every project is 1 git repository, and vice versa - commit security is an all or nothing proposition (so it's hard to have permissions like "this file can only be edited by this user" or "this commit can be made directly to `main`, _if_ it only touches this file") - configuration as configuration - version numbers have to be written into code And obviously this is a problem, because everyone keeps on inventing YAML-based programming idioms for each of their services! Because static configuration doesn't map to even the simplest of setups very well. People come and them mention stuff like Dhall. But really people want to be able to write programs that read files, make HTTP requests, and then make decisions about what to do based on that. And things like "if this test suite fails, do this other orchestration". The ideal CI flow is not staticly definable at the start of execution! There is a killer protocol out there, and it's ... something like Skylark, with a standard library that gives file reading, checking of statuses, probably HTTP and the like.... and an executable that allows for configuring permissions in certain ways. The bonus would be just re-doing Bazel but with less guardrails. If someone showed up with CI/CD that was more in this vein they would become the dominate CD host very quickly in my opinion.

urthor 4y ago

Strongly disagree. IMO I think GUI variables are a big weakness of Airflow. I, simply, never want to be responsible for a typo in a prod config variable. Ever. I'm all for version controlling every, single, variable.