3 ms·

GCP Cloud Composer (Airflow) has been a nightmare. I cannot recommend it. BQ is great.

• verdverm 7y ago
  GKE is amazing too, none of the other major clouds even come close.
• reilly3000 7y ago
  I found Airflow for a small to medium project to be a nightmare on its own; I'm sure its a powerful framework, but plain Python ended up meeting our needs more than adequately. I was self-hosting it while kicking the tires and found it to be a lot to admin. I always thought I would look at Cloud Composer if I wanted to revisit Airflow. Were your issues with Airflow or Cloud Composer? Agreed BQ IS great.
  • DevKoala 7y ago
    My issues have been with cloud-composer. I like Airflow.
• chimerasaurus 7y ago
  Would love to know the pain points so we can make it better - Composer, Airflow, or both.
  • DevKoala 7y ago
    Triggering a DAG externally fails consistently for 1% of requests, and 100% of the time when the web server API goes down, which is too often (502). I understand that in the realm of Airflow, triggering a DAG externally is perhaps a bit of an anti-pattern, but it should not have been documented then: https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf https://cloud.google.com/composer/docs/how-to/using/triggeri... I have cases open with the composer team, and I am upgrading the cluster again this weekend, but I have lost hope at this point; I am designing a solution to stop using external triggers.