3 ms·
Can someone enlighten me whether Apache Airflow is suitable as a business process engine? We have something like orders. So people put orders into our system,
by vbezhenar 4y ago
Can someone enlighten me whether Apache Airflow is suitable as a business process engine?
We have something like orders. So people put orders into our system, some orders are imported from external system. We have something around 100-1000 orders per day, I think. Each order goes through several states. Like CREATED, SOME_INFO_ADDED, REVIEWED, CONCLUSION_CREATED, CONCLUSION_SENT_TO_EXTERNAL_SYSTEM and so on. Some states are simple to change, like few milliseconds to call some web services, some states are 5 minutes from operator, some states are few days. This logic is encoded into our program code. We have plenty of timers, every timer usually transfers orders from one state to another. This is further complicated by the fact that this processing is done via several services, so it's not a single monolith but some kind of service architecture.
Our management wants something to have clear monitoring, so you can find a given task by some property values, monitor its lifetime, check logs for every step, find out why it's failing, etc.
What I usually see is that Apache Airflow is used more like cron replacement. I've read some articles but it's still not clear whether it could be used as a business process engine. I had some experience with Java BPMN engines in the past, it was not very pleasant, but I guess time moved on.
- nooorofe 4y agoApache Airflow is tool for *analytical data* pipelines. If you want to create a daily/monthly data feed from your operational DB, move to it into Data warehouse, run jobs to process/aggregate that data and create daily/weekly/monthly reports, Airflow may help with all those.
- DenseComet 4y agoThis is exactly the usecase Temporal meant for. I've used the Java SDK and its a pretty pleasant experience. Definitely worth taking a look at it.