3 ms·

Lots of the comments here seem to be commenting on their own experiences of complexity and frustration with Airflow, but I'd venture to say that's most data orc

Lots of the comments here seem to be commenting on their own experiences of complexity and frustration with Airflow, but I'd venture to say that's most data orchestration tools. In fact, that sort of feedback is so consistent that I'm half tempted to start a podcast for "orchestration horror stories" (contact if interested). What I've found while building out Shipyard, a hosted lightweight orchestration platform, is that teams want something that "just works". Servers that "just scale". Observability that doesn't require digging. Notifications and retries that work automatically. Workflows that don't mix business logic with platform logic. Code and workflows that sync with git. Deployment that only takes a few minutes. For the straightforward use cases, where you need to run tasks A -> G daily, with a bit of branching logic, Airflow is overkill. Yes, Airflow has a lot of great complex functionality that can help you down the road. But Airflow keeps getting suggested to everyone even if it's not best suited to their use case, resulting in lots of lost time and engineering overhead. While I have definitely have bias, there are a lot of other high quality alternatives out there to explore nowadays!

FridgeSeal 4y ago
I’d 100% listen to an “orchestration horror stories” podcast. I’d also like to throw Matillion into the mix of “horror story tools”. I have far too many matillion-induced bad stories. * Teammates making inscrutable ETL flows because it lets you make nice/visual based flows, and instead of laying them out sanely, they draw pictures with them. One was particularly fond of making flowers and fish, which-while amusing-was entirely unhelpful when there were pressing prod issues and you’re trying to follow the flow around some flower petals. * nigh incomprehensible generated SQL * a design that let users mix orchestration and transformation, so prior teammates created jobs that would stomp on each other’s outputs, because they didn’t orchestrate them sanely * sometimes the runtime would just…stop for reasons we were still unable to resolve. * lets you run arbitrary Python/bash/etc script, so people would put all sorts of wild scripts in the flow. Oh also, it’s not Python it’s actually Jython, and it could mutate variables in the shared environment-an old teammate would (ab)use this functionality to set the date-time of some variable that another ETL-flow would use to make a decision (see issues above about mixing orchestration + transformation).
rgreasons 4y ago
+1 for the orchestration horror stories podcast.

FridgeSeal 4y ago

I’d 100% listen to an “orchestration horror stories” podcast. I’d also like to throw Matillion into the mix of “horror story tools”. I have far too many matillion-induced bad stories. * Teammates making inscrutable ETL flows because it lets you make nice/visual based flows, and instead of laying them out sanely, they draw pictures with them. One was particularly fond of making flowers and fish, which-while amusing-was entirely unhelpful when there were pressing prod issues and you’re trying to follow the flow around some flower petals. * nigh incomprehensible generated SQL * a design that let users mix orchestration and transformation, so prior teammates created jobs that would stomp on each other’s outputs, because they didn’t orchestrate them sanely * sometimes the runtime would just…stop for reasons we were still unable to resolve. * lets you run arbitrary Python/bash/etc script, so people would put all sorts of wild scripts in the flow. Oh also, it’s not Python it’s actually Jython, and it could mutate variables in the shared environment-an old teammate would (ab)use this functionality to set the date-time of some variable that another ETL-flow would use to make a decision (see issues above about mixing orchestration + transformation).

rgreasons 4y ago

+1 for the orchestration horror stories podcast.