stkbailey

3 ms·

1. 1.▲
   by stkbailey 4y ago

   We didn't look into a dataflow engine specifically, in part because we have a heterogeneous set of workfloads. Our core use case is loading mission critical data in chunks, but it is also coordinating SaaS tools and managed services li
2. 2.▲
   by stkbailey 4y ago

   Interesting, Astronomer was actually my last choice for orchestrator. We went with Dagster, but I didn't want to make the takeaway "Dagster solves these problems", because it doesn't directly. Astronomer was just the bes
3. 3.▲
   by stkbailey 4y ago

   Nope -- just MWAA, Astronomer, Dagster Cloud, and Prefect Cloud. In the past I used Argo Workflows pretty extensively and have talked about its pros and cons here: https://www.youtube.com/watch?v=-cyr_kL-9fc
4. 4.▲
   by stkbailey 4y ago

   Yep, that's what I tried to point out in the article.
5. 5.▲
   by stkbailey 4y ago

   Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Man