4 ms·

Other than the examples, what kinds of interesting things could this be used for? What are the advantages over writing code for the coordination?

by mej10 10y ago

talawahdotnet 10y ago
Well similar to SWF it looks like they provide the infrastructure for tracking the state changes and the input/output data at each step so you can: (1) Quickly get a visual overview of all your executions (2) See how many workflows failed (3) Drill down and see which specific step failed and the data that might have triggered the failure (4) re-run specific workflows It provides built in error handling, retries/backoff, and parallelization. These are all things you could code yourself, but at some point you would have built a workflow system. Not to mention that the infrastructure of the workflow system itself is managed for you, so you don't have to worry about that going down. You can write workflows that execute over days, weeks or months and not worry about what happens if your server goes down. It would also come in handy for mutli-language workflows. If your were interacting with a variety of third party APIs that had SDKs in different languages, you could quickly string together a Python Lambda, Java Lambda and some .NET code that used the long polling API. It would be nice if it integrated seamlessly with Cloudwatch and X-Ray to provide a unified view of logging and tracing right from the same interface. I also expect a more seamless integration with AWS Batch at some point. Could be useful for chaining together multiple independently maintained batch processes (video conversion, resizing, publish to youtube). I personally use SWF for a 10 step e-commerce workflow that calls multiple API. It works great, but I could have developed it much more quickly using Step Functions if it were available at the time.
cle 10y ago
One major advantage over its predecessor is the first image: your workflow is defined as data instead of code, so you get a nice visualization of your workflow state. Try reading SWF execution histories for comparison. Are you asking why you wouldn't just write your own coordination code and not use a coordination service at all? The advantage is scaling and robustness. These workflows can run for days/weeks/months, and each step in the workflow can run on different hosts that are optimized for that step. And workflow state is handled by AWS, and so is basic workflow logic (retries, timeouts, event logging, etc.). Unlike Oozie, Luigi, etc., your workflow state is not coupled to any of your infrastructure, so it can survive all kinds of crazy situations (fleets dropping offline, replacing clusters, etc.).
talawahdotnet 10y ago
Another example just occurred to me. Say for example that I need to download a CSV from a third-party everyday, but their server is not reliable, so sometimes it times out, or sometimes the file is not uploaded when it should be. With Step Functions it is now trivial for me to write a simple Python lambda to do the HTTP request and then have the state machine deal with retrying with an exponential backoff (30min, 2hr, 8hr) for as long as I want. If you wrote that by hand you would either need a long running process or you would need to manage scheduling and state management yourself. I was actually planning on doing something like this with lambda, but I was basically just going hack something together to fire the Lambda multiple times per day using CloudWatch Events and then use DynamoDB to keep track of whether or not I had already succeeded. This is much more elegant and robust and you get a visual overview and execution history for free.

Other than the examples, what kinds of interesting things could this be used for? What are the advantages over writing code for the coordination?

by mej10 10y ago

Other than the examples, what kinds of interesting things could this be used for? What are the advantages over writing code for the coordination?

talawahdotnet 10y ago

Well similar to SWF it looks like they provide the infrastructure for tracking the state changes and the input/output data at each step so you can: (1) Quickly get a visual overview of all your executions (2) See how many workflows failed (3) Drill down and see which specific step failed and the data that might have triggered the failure (4) re-run specific workflows It provides built in error handling, retries/backoff, and parallelization. These are all things you could code yourself, but at some point you would have built a workflow system. Not to mention that the infrastructure of the workflow system itself is managed for you, so you don't have to worry about that going down. You can write workflows that execute over days, weeks or months and not worry about what happens if your server goes down. It would also come in handy for mutli-language workflows. If your were interacting with a variety of third party APIs that had SDKs in different languages, you could quickly string together a Python Lambda, Java Lambda and some .NET code that used the long polling API. It would be nice if it integrated seamlessly with Cloudwatch and X-Ray to provide a unified view of logging and tracing right from the same interface. I also expect a more seamless integration with AWS Batch at some point. Could be useful for chaining together multiple independently maintained batch processes (video conversion, resizing, publish to youtube). I personally use SWF for a 10 step e-commerce workflow that calls multiple API. It works great, but I could have developed it much more quickly using Step Functions if it were available at the time.

cle 10y ago

One major advantage over its predecessor is the first image: your workflow is defined as data instead of code, so you get a nice visualization of your workflow state. Try reading SWF execution histories for comparison. Are you asking why you wouldn't just write your own coordination code and not use a coordination service at all? The advantage is scaling and robustness. These workflows can run for days/weeks/months, and each step in the workflow can run on different hosts that are optimized for that step. And workflow state is handled by AWS, and so is basic workflow logic (retries, timeouts, event logging, etc.). Unlike Oozie, Luigi, etc., your workflow state is not coupled to any of your infrastructure, so it can survive all kinds of crazy situations (fleets dropping offline, replacing clusters, etc.).

talawahdotnet 10y ago

Another example just occurred to me. Say for example that I need to download a CSV from a third-party everyday, but their server is not reliable, so sometimes it times out, or sometimes the file is not uploaded when it should be. With Step Functions it is now trivial for me to write a simple Python lambda to do the HTTP request and then have the state machine deal with retrying with an exponential backoff (30min, 2hr, 8hr) for as long as I want. If you wrote that by hand you would either need a long running process or you would need to manage scheduling and state management yourself. I was actually planning on doing something like this with lambda, but I was basically just going hack something together to fire the Lambda multiple times per day using CloudWatch Events and then use DynamoDB to keep track of whether or not I had already succeeded. This is much more elegant and robust and you get a visual overview and execution history for free.