3 ms·

My team has been using Step Functions for a couple of months now to orchestrate application deployments. It is way better than our old SWF-based solution - I es

My team has been using Step Functions for a couple of months now to orchestrate application deployments. It is way better than our old SWF-based solution - I especially love how easy it is to implement dynamic backoff between states. Biggest gripe: updating a state machine is still not supported (you can manage state machines through CloudFormation now, but I believe it updates with replacement, meaning the state machine gets blown away and replaced by a new one with the same name.) We've had to roll our own versioning to get around this. If you'd like to know more about how we are using Step Functions, as well as some specific benchmark comparisons against our old SWF architecture, feel free to check out this post: https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-aws-my-journey-from-swf-to-step-functions/ https://forrestbrazeal.com/2016/12/29/serverless-workflows-o...

deleted 10y ago
[deleted]
PKop 10y ago
Step functions has pre-defined parallel states, but have you come up with a solution for dynamic parallel states? Such as: a parent state's output will determine the size of the input to pass on to multiple copies of identical parallel state children (the number of which is determined by the parent's output). These get the same input or a specific slice of the same input from their parent. I guess these could be created dynamically within the parent, then deleted afterwards. Not as clean as declaring this in the states markup though.
- vrudyi 10y ago
  I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.
- forrestbrazeal 10y ago
  Right now, we have to coordinate our own batch processing, since dynamic parallel states aren't supported, as you note. I've actually asked the Step Functions team about this feature, and I know I'm not the only person to do so, so hopefully it's on their roadmap :)
striglia 10y ago
As someone currently at the "prove it's a good idea" stage of moving production from SWF to SFN, I'm happy to hear it's been such a good experience in practice! I think I'm mostly excited to stop locally representing/maintaining/enforcing the state machine -- it's been a huge source of toil for very little benefit. Hoping the cheapness of a new state machine will encourage splitting them when it makes sense instead of inflating/complicating an existing workflow... How are the latency gains in practice? Specifically the activity -> decision -> activity interstitial latency in SFN vs the old SWF style.
- forrestbrazeal 10y ago
  I haven't actually worked with activities in SFN, just the pure Lambda state machines. But I can tell you those are fast. Like single-digit millisecond latency between states fast.

deleted 10y ago

[deleted]

PKop 10y ago

Step functions has pre-defined parallel states, but have you come up with a solution for dynamic parallel states? Such as: a parent state's output will determine the size of the input to pass on to multiple copies of identical parallel state children (the number of which is determined by the parent's output). These get the same input or a specific slice of the same input from their parent. I guess these could be created dynamically within the parent, then deleted afterwards. Not as clean as declaring this in the states markup though.

vrudyi 10y ago
I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.
forrestbrazeal 10y ago
Right now, we have to coordinate our own batch processing, since dynamic parallel states aren't supported, as you note. I've actually asked the Step Functions team about this feature, and I know I'm not the only person to do so, so hopefully it's on their roadmap :)

striglia 10y ago

As someone currently at the "prove it's a good idea" stage of moving production from SWF to SFN, I'm happy to hear it's been such a good experience in practice! I think I'm mostly excited to stop locally representing/maintaining/enforcing the state machine -- it's been a huge source of toil for very little benefit. Hoping the cheapness of a new state machine will encourage splitting them when it makes sense instead of inflating/complicating an existing workflow... How are the latency gains in practice? Specifically the activity -> decision -> activity interstitial latency in SFN vs the old SWF style.

forrestbrazeal 10y ago
I haven't actually worked with activities in SFN, just the pure Lambda state machines. But I can tell you those are fast. Like single-digit millisecond latency between states fast.