5 ms·

Easily Create Complex Workflows with AWS Step Functions

96 points by vrudyi 10y ago

cle 10y ago
Step Functions is a massive improvement over SWF. As the author said, there are major differences and areas for improvement. Lifecycle management of state machines is pretty poor, especially considering that deleting state machines is hit-or-miss. You'll need to version your state machines yourself--which is still less painful than SWF versioning. Also, because there are no "domains" as in SWF, development can be more of a pain. The last SWF feature that would let me drop SWF completely would be signaling ("sleeping" a workflow until an event occurs).
- 2_listerine_pls 10y ago
  I have been awaiting for the wait-for-signal feature too.
  - nyxie 10y ago
    I've worked around this with Lambda tasks that write a result to a presigned S3 URL and a wait loop that polls it (like CloudFormation custom resources do) but it would be nice if this was built-in.
    - 2_listerine_pls 10y ago
      http://i.imgur.com/A5QnFkO.png http://i.imgur.com/A5QnFkO.png
      - nyxie 10y ago
        Sorry, that's what I meant.
        2_listerine_pls 10y ago
        Have you tried to handle it as an error and retry? Retry": [ { "ErrorEquals": ["S3resourceNotFound"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }, "ErrorEquals": ["OtherErrorName"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }] Simpler.
        nyxie 10y ago
        I couldn't figure out how to return a named error from a python2.7 Lambda function; as far as I could tell it would always return UnhandledError. How'd you do it?
        forrestbrazeal 10y ago
        I don't think you can set custom error names from a Lambda function: "Important Custom Error Names, like those shown above ("ErrorA", "ErrorB", etc.) can be generated by an Activity, but not Lambda Functions. (See "FunctionError" field in the Lambda Response Syntax for more information"[0] [0] https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html https://docs.aws.amazon.com/step-functions/latest/dg/concept...
        2_listerine_pls 10y ago
        Yeah, I was wrong. Just read the AWS Step Functions Developer Guide (1.0) and it's not currently possible to generate custom named errors from a Task. Only possible from Activities.
- striglia 10y ago
  Is it possible to accept an activity task with no timeout, store the state in a DB, then when your event occurs, load state from the DB and send the task_success signal? Not convenient, but seems feasible..
deleted 10y ago
[deleted]
forrestbrazeal 10y ago
My team has been using Step Functions for a couple of months now to orchestrate application deployments. It is way better than our old SWF-based solution - I especially love how easy it is to implement dynamic backoff between states. Biggest gripe: updating a state machine is still not supported (you can manage state machines through CloudFormation now, but I believe it updates with replacement, meaning the state machine gets blown away and replaced by a new one with the same name.) We've had to roll our own versioning to get around this. If you'd like to know more about how we are using Step Functions, as well as some specific benchmark comparisons against our old SWF architecture, feel free to check out this post: https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-aws-my-journey-from-swf-to-step-functions/ https://forrestbrazeal.com/2016/12/29/serverless-workflows-o...
- deleted 10y ago
  [deleted]
- PKop 10y ago
  Step functions has pre-defined parallel states, but have you come up with a solution for dynamic parallel states? Such as: a parent state's output will determine the size of the input to pass on to multiple copies of identical parallel state children (the number of which is determined by the parent's output). These get the same input or a specific slice of the same input from their parent. I guess these could be created dynamically within the parent, then deleted afterwards. Not as clean as declaring this in the states markup though.
  - vrudyi 10y ago
    I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.
  - forrestbrazeal 10y ago
    Right now, we have to coordinate our own batch processing, since dynamic parallel states aren't supported, as you note. I've actually asked the Step Functions team about this feature, and I know I'm not the only person to do so, so hopefully it's on their roadmap :)
djhworld 10y ago
The blog post talks about triggering the workflow off an upload to S3, but neglects to discuss the critical step around the fact that you need a Lambda function to trigger the step function state machine (if you have a look at the github it's there) I wish step functions just supported the triggers directly tbh, i.e. have the ability to invoke a state machine on an S3 put, passing the event information to the first step in the machine.
- forrestbrazeal 10y ago
  Step Functions does integrate directly with API Gateway now. [0] I'm not sure if that helps your specific use case, though. [0] https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-api-gateway-integration-with-aws-step-functions/ https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-ap...
- vrudyi 10y ago
  Thanks a lot, I've just extended the post with this information.
striglia 10y ago
Very much looking forward to an upcoming project with Step Functions at $DAYJOB. I suspect there are many use cases for turning complicated intersections of batch jobs and blobs of code into discrete, independent tasks like SFN requires. Particularly interested in composing higher and lower level state machines for better separation of concerns, but haven't yet come up with a compelling use case :)
casperc 10y ago
I am actually starting to look into creating a batch workflow, which would be very suited for this (or SWF). However, I would like to know what the alternatives are that are non-amazon specific. Are there any open source projects that cover this space of defining workflows?

96 points by vrudyi 10y ago

cle 10y ago

Step Functions is a massive improvement over SWF. As the author said, there are major differences and areas for improvement. Lifecycle management of state machines is pretty poor, especially considering that deleting state machines is hit-or-miss. You'll need to version your state machines yourself--which is still less painful than SWF versioning. Also, because there are no "domains" as in SWF, development can be more of a pain. The last SWF feature that would let me drop SWF completely would be signaling ("sleeping" a workflow until an event occurs).

2_listerine_pls 10y ago
I have been awaiting for the wait-for-signal feature too.
- nyxie 10y ago
  I've worked around this with Lambda tasks that write a result to a presigned S3 URL and a wait loop that polls it (like CloudFormation custom resources do) but it would be nice if this was built-in.
  - 2_listerine_pls 10y ago
    http://i.imgur.com/A5QnFkO.png http://i.imgur.com/A5QnFkO.png
    - nyxie 10y ago
      Sorry, that's what I meant.
      - 2_listerine_pls 10y ago
        Have you tried to handle it as an error and retry? Retry": [ { "ErrorEquals": ["S3resourceNotFound"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }, "ErrorEquals": ["OtherErrorName"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }] Simpler.
        nyxie 10y ago
        I couldn't figure out how to return a named error from a python2.7 Lambda function; as far as I could tell it would always return UnhandledError. How'd you do it?
        forrestbrazeal 10y ago
        I don't think you can set custom error names from a Lambda function: "Important Custom Error Names, like those shown above ("ErrorA", "ErrorB", etc.) can be generated by an Activity, but not Lambda Functions. (See "FunctionError" field in the Lambda Response Syntax for more information"[0] [0] https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html https://docs.aws.amazon.com/step-functions/latest/dg/concept...
        2_listerine_pls 10y ago
        Yeah, I was wrong. Just read the AWS Step Functions Developer Guide (1.0) and it's not currently possible to generate custom named errors from a Task. Only possible from Activities.
striglia 10y ago
Is it possible to accept an activity task with no timeout, store the state in a DB, then when your event occurs, load state from the DB and send the task_success signal? Not convenient, but seems feasible..

deleted 10y ago

[deleted]

forrestbrazeal 10y ago

My team has been using Step Functions for a couple of months now to orchestrate application deployments. It is way better than our old SWF-based solution - I especially love how easy it is to implement dynamic backoff between states. Biggest gripe: updating a state machine is still not supported (you can manage state machines through CloudFormation now, but I believe it updates with replacement, meaning the state machine gets blown away and replaced by a new one with the same name.) We've had to roll our own versioning to get around this. If you'd like to know more about how we are using Step Functions, as well as some specific benchmark comparisons against our old SWF architecture, feel free to check out this post: https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-aws-my-journey-from-swf-to-step-functions/ https://forrestbrazeal.com/2016/12/29/serverless-workflows-o...

deleted 10y ago
[deleted]
PKop 10y ago
Step functions has pre-defined parallel states, but have you come up with a solution for dynamic parallel states? Such as: a parent state's output will determine the size of the input to pass on to multiple copies of identical parallel state children (the number of which is determined by the parent's output). These get the same input or a specific slice of the same input from their parent. I guess these could be created dynamically within the parent, then deleted afterwards. Not as clean as declaring this in the states markup though.
- vrudyi 10y ago
  I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.
- forrestbrazeal 10y ago
  Right now, we have to coordinate our own batch processing, since dynamic parallel states aren't supported, as you note. I've actually asked the Step Functions team about this feature, and I know I'm not the only person to do so, so hopefully it's on their roadmap :)

djhworld 10y ago

The blog post talks about triggering the workflow off an upload to S3, but neglects to discuss the critical step around the fact that you need a Lambda function to trigger the step function state machine (if you have a look at the github it's there) I wish step functions just supported the triggers directly tbh, i.e. have the ability to invoke a state machine on an S3 put, passing the event information to the first step in the machine.

forrestbrazeal 10y ago
Step Functions does integrate directly with API Gateway now. [0] I'm not sure if that helps your specific use case, though. [0] https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-api-gateway-integration-with-aws-step-functions/ https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-ap...
vrudyi 10y ago
Thanks a lot, I've just extended the post with this information.

striglia 10y ago

Very much looking forward to an upcoming project with Step Functions at $DAYJOB. I suspect there are many use cases for turning complicated intersections of batch jobs and blobs of code into discrete, independent tasks like SFN requires. Particularly interested in composing higher and lower level state machines for better separation of concerns, but haven't yet come up with a compelling use case :)

casperc 10y ago

I am actually starting to look into creating a batch workflow, which would be very suited for this (or SWF). However, I would like to know what the alternatives are that are non-amazon specific. Are there any open source projects that cover this space of defining workflows?