4 ms·

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow,

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases. In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp. Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think! For more information: Composable Site - https://composableanalytics.com/ https://composableanalytics.com/ Try it yourself - https://cloud.composableanalytics.com/ https://cloud.composableanalytics.com/ Composable's Blog - http://blog.composable.ai/ http://blog.composable.ai/

j_s 10y ago
Is any aspect of your product open source? Doesn't look like it; this makes it a bit of an apples-vs-apple-pie comparison.
batbomb 10y ago
It sounds like you are more of a Beam/dataflow competitor than an airflow/workflow competitor.
caravel 10y ago
[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 https://medium.freecodecamp.com/the-rise-of-the-data-enginee... I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).
- larsf 10y ago
  You can use Composable's Fluent API to author and run dataflows in code. No GUI required. Composable's platform is open and completely extensible in sense that anyone can add applications or first class modules to the system, including swapping out some of the more internal components.

j_s 10y ago

Is any aspect of your product open source? Doesn't look like it; this makes it a bit of an apples-vs-apple-pie comparison.

batbomb 10y ago

It sounds like you are more of a Beam/dataflow competitor than an airflow/workflow competitor.

caravel 10y ago

[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: https://medium.freecodecamp.com/the-rise-of-the-data-engineer-91be18f1e603 https://medium.freecodecamp.com/the-rise-of-the-data-enginee... I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).

larsf 10y ago
You can use Composable's Fluent API to author and run dataflows in code. No GUI required. Composable's platform is open and completely extensible in sense that anyone can add applications or first class modules to the system, including swapping out some of the more internal components.