3 ms·

Airflow is a great example of technology being used at a massive company for massive company problems...which is now being pushed as a solution to everything P

by nojito 6y ago

Airflow is a great example of technology being used at a massive company for massive company problems...which is now being pushed as a solution to everything Papermill is another example.

detaro 6y ago
Any hints at solutions that help if you're a small company having small company sized problems but don't want to DIY the entire flow execution logic?
- walleeee 6y ago
  I'm the lone developer on a project which will likely never scale beyond a few thousand users and I'm really liking Dagster. You can deploy it on a bunch of other platforms (Airflow, Dask, Celery, K8s) which is really nice for my use case (automating workflows in HPC environments from the browser) or run it standalone
mliker 6y ago
Airflow can be used to process data that only adds up to 10s or 100s of MBs. In aggregate, a big tech company processes petabytes or exabytes, but remember, technology like Airflow isn't processing all of that simultaneously within one task. So no, I would not characterize Airflow as a solution to just "massive company problems".
pgoggijr 6y ago
Been using airflow in Production at a small start-up for almost 4 years now and been very happy with it (small dockerized deployment, and simple syncing with github) - what big company problems does it solve that make it unsuitable for use at small companies?
higeorge13 6y ago
I have set it up, configured and performed all the upgrades just by myself in a small startup, performing very complex daily tasks. I miss features, sure. The whole execution timing might be confusing the first time, sure. But I can't figure out why i see so many comments regarding deployment difficulties.
rywalker 6y ago
We were a very small company when we discovered Airflow, just saying, and it provided a lot of value to us vs. alternatives. We fell in love with Airflow's capability to allow us to write one small python program that generated hundreds of tailored dags by iterating over the configuration of each of our customers. This dynamism is very powerful for similar use cases.
bhntr3 6y ago
When Max started building airflow Airbnb's engineering team was probably 80 or so people. The data engineering and data infrastructure teams couldn't have been more than ten combined. I'm not sure that's massive but it is the point where the previous solution (Chronos: https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d https://medium.com/airbnb-engineering/chronos-a-replacement-...) was starting to strain. Airflow may not be necessary for a ten person team but it could be if that team has complicated data orchestration needs and doesn't want to incrementally replace their infrastructure every couple years.

by nojito 6y ago

Airflow is a great example of technology being used at a massive company for massive company problems...which is now being pushed as a solution to everything Papermill is another example.

detaro 6y ago

Any hints at solutions that help if you're a small company having small company sized problems but don't want to DIY the entire flow execution logic?

walleeee 6y ago
I'm the lone developer on a project which will likely never scale beyond a few thousand users and I'm really liking Dagster. You can deploy it on a bunch of other platforms (Airflow, Dask, Celery, K8s) which is really nice for my use case (automating workflows in HPC environments from the browser) or run it standalone

mliker 6y ago

Airflow can be used to process data that only adds up to 10s or 100s of MBs. In aggregate, a big tech company processes petabytes or exabytes, but remember, technology like Airflow isn't processing all of that simultaneously within one task. So no, I would not characterize Airflow as a solution to just "massive company problems".

pgoggijr 6y ago

Been using airflow in Production at a small start-up for almost 4 years now and been very happy with it (small dockerized deployment, and simple syncing with github) - what big company problems does it solve that make it unsuitable for use at small companies?

higeorge13 6y ago

I have set it up, configured and performed all the upgrades just by myself in a small startup, performing very complex daily tasks. I miss features, sure. The whole execution timing might be confusing the first time, sure. But I can't figure out why i see so many comments regarding deployment difficulties.

rywalker 6y ago

We were a very small company when we discovered Airflow, just saying, and it provided a lot of value to us vs. alternatives. We fell in love with Airflow's capability to allow us to write one small python program that generated hundreds of tailored dags by iterating over the configuration of each of our customers. This dynamism is very powerful for similar use cases.

bhntr3 6y ago

When Max started building airflow Airbnb's engineering team was probably 80 or so people. The data engineering and data infrastructure teams couldn't have been more than ten combined. I'm not sure that's massive but it is the point where the previous solution (Chronos: https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d https://medium.com/airbnb-engineering/chronos-a-replacement-...) was starting to strain. Airflow may not be necessary for a ten person team but it could be if that team has complicated data orchestration needs and doesn't want to incrementally replace their infrastructure every couple years.