4 ms·
I think you're 100% right that the tasks that can be accomplished in Airflow are currently being unbundled by tools in the modern data stack, but that doesn't e
by blakeburch 5y ago
I think you're 100% right that the tasks that can be accomplished in Airflow are currently being unbundled by tools in the modern data stack, but that doesn't erase the need for tools like Airflow. Sure, you can now write less code to load your data, transform it, and send it out to other tools. As the unbundling occurs, the end result is more fragmentation and fragility in how teams manage their data.
Data teams I talk to can't turn to any single location to see every touchpoint their data goes through. They're relying on each tool's independent scheduling system and hoping that everything runs at the right time without errors. If something breaks, bad data gets deployed and it becomes a mad scramble to verify which tool caused the error and which reports/dashboards/ML models/etc. were impacted downstream.
While these unbundled tools can get you 90% of the way to your desired end goal, you'll inevitably face a situation where your use case or SaaS tool is unsupported. In every situation like this I've ever faced, the team ultimately ends up writing and managing their own custom scripts to account for this situation. Now you have your unbundled tool + your custom script. Why not just manage all of the tools and your scripts from a singular source in the first place?
While unbundling is the reality, this new era of data technology will always still have a need for data orchestration tools that serve as a centralized view into your data workflows, whether that's Airflow or any of the new players in the space.
(Disclosure: I'm a co-founder of https://www.shipyardapp.com/ https://www.shipyardapp.com/, building better data orchestration for modern data teams)
- jasonhansel 5y agoAgreed. Even when they do support one's intended use case, these unbundled tools seem like classic examples of the inner platform effect: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect No amount of tooling will make data transformation a painless process; all you end up doing is burying the business logic under so many layers of abstraction that it becomes impossible for anyone to understand.
- contravariant 5y agoIsn't the main selling point of airflow the bundling in the first place? Why would you want many different specialized tools to manage scheduled tasks?
- jsemrau 5y agoExactly my thoughts as well. I have one point where I can see if all the remote services that I am using are operating correctly. I don't need to connect to various other apps to figure this out.
- misterbwong 5y agoCue the famous Jim Barksdale quote: "There’s only two ways to make money in business: One is to bundle; the other is unbundle"
- blakeburch 5y agoI think there's two factors at play here: 1) Specialized tools reduce the amount of engineering overhead. As a business, I primarily care about time to value. If I can use specialized SaaS to get my data centralized, clean, and synced across my tools in a week, why would I want to spend months building all of these processes from scratch? Sure, I lose control, visibility, and more... but I was able to deliver value 3 months ahead of schedule. 2) Existing tools like Airflow are highly technical to get started with. You can't just focus on building out scripted solutions. You have to set up and manage the infrastructure. You have to sift through the tool's documentation to understand how to effectively build DAGs. You have to inject your business logic with platform logic to make sure your code will run on Airflow. Because the demand for data professionals is high and the supply is low, the technology ends up trying to offset the need for those highly technical skills in your organization.