3 ms·
Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice
by djtriptych 7y ago
Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it.
It's amazing it works at all in my opinion.
This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with it's GIL, weak typing, lack of concurrency primitives, and generally OOP / imperative style is just the wrong tool for the job.
[0]: https://github.com/apache/airflow/blob/master/airflow/jobs/scheduler_job.py https://github.com/apache/airflow/blob/master/airflow/jobs/s...
- thesorrow 7y agoI'm using Airflow for a lot of critical tasks and it works really well. But I agree that Python may not be the best language to implement a workflow engine.
- djtriptych 7y agoIt's fine for moderate workflows. We ran into several hard limits when scaling up, and thought to try to patch some limitations. I think it's got a number of edge cases / scalability issues that will be very hard for them to fix without a full rewrite of the internals.
- j88439h84 7y agoI don't know if Python is the best tool for the job, but with modern tooling it is leagues better for complex applications than old python. https://trio.readthedocs.io https://trio.readthedocs.io is an extremely good python concurrency library based on the model of Structured Concurrency (https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful https://vorpus.org/blog/notes-on-structured-concurrency-or-g...). The typing issues are far improved in current Python with annotations and attrs/dataclasses.