5 ms·

AWS Step Functions – Build Distributed Applications Using Visual Workflows

64 points by pradeepchhetri 10y ago

mej10 10y ago
Other than the examples, what kinds of interesting things could this be used for? What are the advantages over writing code for the coordination?
- talawahdotnet 10y ago
  Well similar to SWF it looks like they provide the infrastructure for tracking the state changes and the input/output data at each step so you can: (1) Quickly get a visual overview of all your executions (2) See how many workflows failed (3) Drill down and see which specific step failed and the data that might have triggered the failure (4) re-run specific workflows It provides built in error handling, retries/backoff, and parallelization. These are all things you could code yourself, but at some point you would have built a workflow system. Not to mention that the infrastructure of the workflow system itself is managed for you, so you don't have to worry about that going down. You can write workflows that execute over days, weeks or months and not worry about what happens if your server goes down. It would also come in handy for mutli-language workflows. If your were interacting with a variety of third party APIs that had SDKs in different languages, you could quickly string together a Python Lambda, Java Lambda and some .NET code that used the long polling API. It would be nice if it integrated seamlessly with Cloudwatch and X-Ray to provide a unified view of logging and tracing right from the same interface. I also expect a more seamless integration with AWS Batch at some point. Could be useful for chaining together multiple independently maintained batch processes (video conversion, resizing, publish to youtube). I personally use SWF for a 10 step e-commerce workflow that calls multiple API. It works great, but I could have developed it much more quickly using Step Functions if it were available at the time.
- cle 10y ago
  One major advantage over its predecessor is the first image: your workflow is defined as data instead of code, so you get a nice visualization of your workflow state. Try reading SWF execution histories for comparison. Are you asking why you wouldn't just write your own coordination code and not use a coordination service at all? The advantage is scaling and robustness. These workflows can run for days/weeks/months, and each step in the workflow can run on different hosts that are optimized for that step. And workflow state is handled by AWS, and so is basic workflow logic (retries, timeouts, event logging, etc.). Unlike Oozie, Luigi, etc., your workflow state is not coupled to any of your infrastructure, so it can survive all kinds of crazy situations (fleets dropping offline, replacing clusters, etc.).
- talawahdotnet 10y ago
  Another example just occurred to me. Say for example that I need to download a CSV from a third-party everyday, but their server is not reliable, so sometimes it times out, or sometimes the file is not uploaded when it should be. With Step Functions it is now trivial for me to write a simple Python lambda to do the HTTP request and then have the state machine deal with retrying with an exponential backoff (30min, 2hr, 8hr) for as long as I want. If you wrote that by hand you would either need a long running process or you would need to manage scheduling and state management yourself. I was actually planning on doing something like this with lambda, but I was basically just going hack something together to fire the Lambda multiple times per day using CloudWatch Events and then use DynamoDB to keep track of whether or not I had already succeeded. This is much more elegant and robust and you get a visual overview and execution history for free.
talawahdotnet 10y ago
Nice. So this is basically a replacement for SWF (Simple Workflow Service) that allows you to build the decision steps using a GUI/JSON and allows you to execute tasks using either Lambda or the traditional activity pollers. Building a multi-step workflow that handles branching, retries and failure just got a lot easier. I was hoping they would do something like this, working with SWF has always felt unnecessarily complicated. The Java Flow Framework just felt like an awkward abstraction to me, but this makes a lot more sense. Combine this with their new AWS Batch service and I think people are going to save countless man hours that would have otherwise been spent working on repetitive, undifferentiated code.
- cle 10y ago
  The Flow Framework is an abomination to engineering, and should die a pathetic death. Liberal annotations, codegen, aspect weaving and bytecode manip, requires a special JUnit test runner--it embodies all the aspects of the Java culture that I despise. I rather like the idea of SWF, but its API is too low-level, it has poor tooling, and the workflow/activity versioning is awkward and makes the systems too rigid. I am glad that AWS is moving to data-driven workflow definitions, that means they can build much better operational tooling (like the GUI) to manage and understand workflow executions.
monksy 10y ago
This sounds a lot like HP Operations Orchestration.
- warrenm 10y ago
  A primitive version ... but it does look a LOT like it
- cle 10y ago
  I didn't know what that was, and went to HP's website to look it up. > "IT process automation and run book software that improves service quality and customer satisfaction, and lowers costs by eliminating latency between silos, increasing first time right rate, enforcing standards, and delivering reports for ROI, audits and more." Ugh, seriously HP? What a load of BS. What the hell does it do? Compare that to the AWS description of Step Functions: > "AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly."
  - monksy 10y ago
    It's a wyswig way of doing integrations and orchestrating stuff. [was going to say operations like start this vm, or provision this vm based on the template, post cat meme to s3, etc] This is what the studio client looks like: http://www.www8-hp.com/us/en/images/auto_oo_studio_ss_tcm245_1626213_tcm245_1626234_tcm245-1626213.jpg http://www.www8-hp.com/us/en/images/auto_oo_studio_ss_tcm245...
- dstainer 10y ago
  Reminds me a lot of BPEL back in the day. Great now I'm having flashbacks
eva1984 10y ago
Not useful. Those graphical programming interface all end up miserably. Don't understand why people still trying to sell it.
- mcphage 10y ago
  > Those graphical programming interface all end up miserably. This is factually not true. There are a number of fields where graphical programming interfaces have a long history of success.
- novembermike 10y ago
  This isn't really a graphical programming interface. The stuff inside of the blocks is code that's deployed separately and does a thing. There's a graphical thingy that allows you to describe state transitions between those blocks of code but you aren't really programming.
- warrenm 10y ago
  False. By a long shot. Just look at LabVIEW & Operations Orchestration.

mej10 10y ago

Other than the examples, what kinds of interesting things could this be used for? What are the advantages over writing code for the coordination?

talawahdotnet 10y ago
Well similar to SWF it looks like they provide the infrastructure for tracking the state changes and the input/output data at each step so you can: (1) Quickly get a visual overview of all your executions (2) See how many workflows failed (3) Drill down and see which specific step failed and the data that might have triggered the failure (4) re-run specific workflows It provides built in error handling, retries/backoff, and parallelization. These are all things you could code yourself, but at some point you would have built a workflow system. Not to mention that the infrastructure of the workflow system itself is managed for you, so you don't have to worry about that going down. You can write workflows that execute over days, weeks or months and not worry about what happens if your server goes down. It would also come in handy for mutli-language workflows. If your were interacting with a variety of third party APIs that had SDKs in different languages, you could quickly string together a Python Lambda, Java Lambda and some .NET code that used the long polling API. It would be nice if it integrated seamlessly with Cloudwatch and X-Ray to provide a unified view of logging and tracing right from the same interface. I also expect a more seamless integration with AWS Batch at some point. Could be useful for chaining together multiple independently maintained batch processes (video conversion, resizing, publish to youtube). I personally use SWF for a 10 step e-commerce workflow that calls multiple API. It works great, but I could have developed it much more quickly using Step Functions if it were available at the time.
cle 10y ago
One major advantage over its predecessor is the first image: your workflow is defined as data instead of code, so you get a nice visualization of your workflow state. Try reading SWF execution histories for comparison. Are you asking why you wouldn't just write your own coordination code and not use a coordination service at all? The advantage is scaling and robustness. These workflows can run for days/weeks/months, and each step in the workflow can run on different hosts that are optimized for that step. And workflow state is handled by AWS, and so is basic workflow logic (retries, timeouts, event logging, etc.). Unlike Oozie, Luigi, etc., your workflow state is not coupled to any of your infrastructure, so it can survive all kinds of crazy situations (fleets dropping offline, replacing clusters, etc.).
talawahdotnet 10y ago
Another example just occurred to me. Say for example that I need to download a CSV from a third-party everyday, but their server is not reliable, so sometimes it times out, or sometimes the file is not uploaded when it should be. With Step Functions it is now trivial for me to write a simple Python lambda to do the HTTP request and then have the state machine deal with retrying with an exponential backoff (30min, 2hr, 8hr) for as long as I want. If you wrote that by hand you would either need a long running process or you would need to manage scheduling and state management yourself. I was actually planning on doing something like this with lambda, but I was basically just going hack something together to fire the Lambda multiple times per day using CloudWatch Events and then use DynamoDB to keep track of whether or not I had already succeeded. This is much more elegant and robust and you get a visual overview and execution history for free.

talawahdotnet 10y ago

Nice. So this is basically a replacement for SWF (Simple Workflow Service) that allows you to build the decision steps using a GUI/JSON and allows you to execute tasks using either Lambda or the traditional activity pollers. Building a multi-step workflow that handles branching, retries and failure just got a lot easier. I was hoping they would do something like this, working with SWF has always felt unnecessarily complicated. The Java Flow Framework just felt like an awkward abstraction to me, but this makes a lot more sense. Combine this with their new AWS Batch service and I think people are going to save countless man hours that would have otherwise been spent working on repetitive, undifferentiated code.

cle 10y ago
The Flow Framework is an abomination to engineering, and should die a pathetic death. Liberal annotations, codegen, aspect weaving and bytecode manip, requires a special JUnit test runner--it embodies all the aspects of the Java culture that I despise. I rather like the idea of SWF, but its API is too low-level, it has poor tooling, and the workflow/activity versioning is awkward and makes the systems too rigid. I am glad that AWS is moving to data-driven workflow definitions, that means they can build much better operational tooling (like the GUI) to manage and understand workflow executions.

monksy 10y ago

This sounds a lot like HP Operations Orchestration.

warrenm 10y ago
A primitive version ... but it does look a LOT like it
cle 10y ago
I didn't know what that was, and went to HP's website to look it up. > "IT process automation and run book software that improves service quality and customer satisfaction, and lowers costs by eliminating latency between silos, increasing first time right rate, enforcing standards, and delivering reports for ROI, audits and more." Ugh, seriously HP? What a load of BS. What the hell does it do? Compare that to the AWS description of Step Functions: > "AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly."
- monksy 10y ago
  It's a wyswig way of doing integrations and orchestrating stuff. [was going to say operations like start this vm, or provision this vm based on the template, post cat meme to s3, etc] This is what the studio client looks like: http://www.www8-hp.com/us/en/images/auto_oo_studio_ss_tcm245_1626213_tcm245_1626234_tcm245-1626213.jpg http://www.www8-hp.com/us/en/images/auto_oo_studio_ss_tcm245...
dstainer 10y ago
Reminds me a lot of BPEL back in the day. Great now I'm having flashbacks

eva1984 10y ago

Not useful. Those graphical programming interface all end up miserably. Don't understand why people still trying to sell it.

mcphage 10y ago
> Those graphical programming interface all end up miserably. This is factually not true. There are a number of fields where graphical programming interfaces have a long history of success.
novembermike 10y ago
This isn't really a graphical programming interface. The stuff inside of the blocks is code that's deployed separately and does a thing. There's a graphical thingy that allows you to describe state transitions between those blocks of code but you aren't really programming.
warrenm 10y ago
False. By a long shot. Just look at LabVIEW & Operations Orchestration.