4 ms·

I'm going into get-off-my-lawn mode here if you don't mind. I don't see why this requires a new-fangled technology or buzzword. Just have a status code(s) or in

by tabtab 8y ago

I'm going into get-off-my-lawn mode here if you don't mind. I don't see why this requires a new-fangled technology or buzzword. Just have a status code(s) or indicator(s) on a given request. The client side or requesting service(s) can periodically check on the status using polling and/or user status update requests. For example, poll automatically every 2 minutes (to avoid flooding the network), but give user the option of clicking a button to check current status. Give the requester an option of a time-limit, if applicable. If the process takes too long, the status changes to "timed-out". The client/requester can then issue a "re-submit" request, if applicable. The technique is pretty much the same whether using ESB, microservices, Stored Procedures, or carrier pigeons.

tedmiston 8y ago
The benefits of the frameworks become more tangible as the structure of your workflows become more complex and with (acyclic) dependencies and with distributed execution.
- tabtab 8y ago
  True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework. But I've seen some really ugly frameworks where one put in Cadillacs for every part when Chevys would do just fine 98% of the time. Staff have to learn, understand, maintain, and tune complex frameworks.
  - tedmiston 8y ago
    I understand the concern. It's kind of like using a web framework — you probably wouldn't write your own form processing and ORM code to avoid injection attacks when it's already been built. Airflow has analogous features and protections for tasks/workflows. Most companies adopting Airflow already have workflow requirements like this, even if it's just a single transform or moving data from one system to another. Even if you have just a two-step workflow with Task B dependent upon success of Task A, Airflow offers protection, historical stats, email alerting, etc over trying to schedule successive cron jobs with built-in assumptions, hacking together a dependency system, etc. To me, Airflow is the Honda of this domain. Overall it's a relatively small and simple framework from the DAG author's perspective.

by tabtab 8y ago

tedmiston 8y ago

The benefits of the frameworks become more tangible as the structure of your workflows become more complex and with (acyclic) dependencies and with distributed execution.

tabtab 8y ago
True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework. But I've seen some really ugly frameworks where one put in Cadillacs for every part when Chevys would do just fine 98% of the time. Staff have to learn, understand, maintain, and tune complex frameworks.
- tedmiston 8y ago
  I understand the concern. It's kind of like using a web framework — you probably wouldn't write your own form processing and ORM code to avoid injection attacks when it's already been built. Airflow has analogous features and protections for tasks/workflows. Most companies adopting Airflow already have workflow requirements like this, even if it's just a single transform or moving data from one system to another. Even if you have just a two-step workflow with Task B dependent upon success of Task A, Airflow offers protection, historical stats, email alerting, etc over trying to schedule successive cron jobs with built-in assumptions, hacking together a dependency system, etc. To me, Airflow is the Honda of this domain. Overall it's a relatively small and simple framework from the DAG author's perspective.