4 ms·

Running Cross-Account Workflows with AWS Step Functions and Amazon API Gateway

35 points by kiyanwang 5y ago

kgbcia 5y ago
all that no code has a lot of code.
alextheparrot 5y ago
Is this just an elaborate way to trigger someone else’s AWS code using RBAC instead of an API token? Seems like the “normal” way I’d go about this would be someone sets up a server, I ping it with token auth that’s been provided, thing happens. Now instead of doing auth through headers we require both parties to be on AWS to use their auth magic? Guess it could be useful if both organizations are within the same company, so you can just enable app-to-app auth. No one is probably worried about cloud vendor compatibility in that case. No diss on it, but it seems kinda niche?
- sokoloff 5y ago
  It’s on the AWS Compute blog, so they’re fairly willing to spot an initial condition of “obviously, this is all running on AWS; here’s how to something cross-account that you know how to do in a single account.”
EE84M3i 5y ago
Why can't you do this directly without the API Gateway? https://docs.aws.amazon.com/step-functions/latest/dg/concepts-nested-workflows.html https://docs.aws.amazon.com/step-functions/latest/dg/concept...
- yonixw 5y ago
  It's in another account. Essentially this entire article boils down to be able to allow ApiGateway invocation by specific StateMachineARN which is being kept when calling any external ApiGatway: > "aws:SourceArn": ["<SourceAccountStateMachineARN>"]
  - EE84M3i 5y ago
    Yeah, I understand that, but I don't understand why you can't execute a state machine in another account/region - assuming the IAM policy allows it. If this is a limitation of the startExectuion resource, I don't see it documented.
XorNot 5y ago
I am yet to find a single use case where AWS step functions look appealing. Am I right to think that if you don't want to do some "no code" weirdness, they're just not for you? Because deploying a thing I can't store in source control looks like insanity to me.
- c1yd3i 5y ago
  You store the State Machine using Terraform, in source control.
- mcintyre1994 5y ago
  I've used it before to do lightweight scheduling: you can include a timestamp and then have a "sleep until" step that waits until that timestamp before continuing. So I'd pass in timestamp + event, it'd wait and then execute the event by passing it to a lambda. For something only occasionally I think it was probably easier and less intrusive into the rest of the infrastructure than any alternative. In terms of source control, the step function definition is just JSON (IIRC). I deployed it with CDK so the structure of the step function, its metadata, permissions etc. were all in source control. AWS docs like this tend to show things done in either the web UI or the CLI, but generally you can use an infrastructure-as-code solution instead. Also the local testing experience is surprisingly good: there's a stepfunctions-local by Amazon and I think that's used by localstack too (which is what I used).
- jon_adler 5y ago
  It is great for large, asynchronous, parallelised processing jobs. I’ve used it to integrate and orchestrate with numerous services within AWS ecosystem. We use source control for the step function JSON and deploy with terraform.
alephnan 5y ago
Can someone please provide an example of a large-scale distributed state machine? I can’t think of any. Maybe a train / rail network?
belter 5y ago
A little friendly warning, about setting orchestrated logic with AWS Step Functions or similar third party solutions, particularly when calling Lambda functions. Don't forget to setup your AWS billing alarms using AWS Budgets plus AWS Budget Actions. A standard best practice in any case. Also enforce timeouts at the State Machine level: https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-state-machine-structure.html https://docs.aws.amazon.com/step-functions/latest/dg/amazon-... The reason being that, for a Lambda function, and independently of your execution logic, the Lambda execution in any case terminates, ( depending on configuration), in the worst case after 15 min. So also terminates the corresponding billing for that execution also. With Step Functions or similar orchestration, a logic error could expose you to create an infinite loop of continuous invocations and the corresponding never ending growing invoice accompanying it :-)
- moltar 5y ago
  Good advice. But a small correction. Step Functions have a limit too. I think it’s 25,000 transitions per execution.

35 points by kiyanwang 5y ago

all that no code has a lot of code.

Is this just an elaborate way to trigger someone else’s AWS code using RBAC instead of an API token? Seems like the “normal” way I’d go about this would be someone sets up a server, I ping it with token auth that’s been provided, thing happens. Now instead of doing auth through headers we require both parties to be on AWS to use their auth magic? Guess it could be useful if both organizations are within the same company, so you can just enable app-to-app auth. No one is probably worried about cloud vendor compatibility in that case. No diss on it, but it seems kinda niche?

sokoloff 5y ago
It’s on the AWS Compute blog, so they’re fairly willing to spot an initial condition of “obviously, this is all running on AWS; here’s how to something cross-account that you know how to do in a single account.”

EE84M3i 5y ago

Why can't you do this directly without the API Gateway? https://docs.aws.amazon.com/step-functions/latest/dg/concepts-nested-workflows.html https://docs.aws.amazon.com/step-functions/latest/dg/concept...

yonixw 5y ago
It's in another account. Essentially this entire article boils down to be able to allow ApiGateway invocation by specific StateMachineARN which is being kept when calling any external ApiGatway: > "aws:SourceArn": ["<SourceAccountStateMachineARN>"]
- EE84M3i 5y ago
  Yeah, I understand that, but I don't understand why you can't execute a state machine in another account/region - assuming the IAM policy allows it. If this is a limitation of the startExectuion resource, I don't see it documented.

XorNot 5y ago

I am yet to find a single use case where AWS step functions look appealing. Am I right to think that if you don't want to do some "no code" weirdness, they're just not for you? Because deploying a thing I can't store in source control looks like insanity to me.

c1yd3i 5y ago
You store the State Machine using Terraform, in source control.
mcintyre1994 5y ago
I've used it before to do lightweight scheduling: you can include a timestamp and then have a "sleep until" step that waits until that timestamp before continuing. So I'd pass in timestamp + event, it'd wait and then execute the event by passing it to a lambda. For something only occasionally I think it was probably easier and less intrusive into the rest of the infrastructure than any alternative. In terms of source control, the step function definition is just JSON (IIRC). I deployed it with CDK so the structure of the step function, its metadata, permissions etc. were all in source control. AWS docs like this tend to show things done in either the web UI or the CLI, but generally you can use an infrastructure-as-code solution instead. Also the local testing experience is surprisingly good: there's a stepfunctions-local by Amazon and I think that's used by localstack too (which is what I used).
jon_adler 5y ago
It is great for large, asynchronous, parallelised processing jobs. I’ve used it to integrate and orchestrate with numerous services within AWS ecosystem. We use source control for the step function JSON and deploy with terraform.

alephnan 5y ago

Can someone please provide an example of a large-scale distributed state machine? I can’t think of any. Maybe a train / rail network?

belter 5y ago

A little friendly warning, about setting orchestrated logic with AWS Step Functions or similar third party solutions, particularly when calling Lambda functions. Don't forget to setup your AWS billing alarms using AWS Budgets plus AWS Budget Actions. A standard best practice in any case. Also enforce timeouts at the State Machine level: https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-state-machine-structure.html https://docs.aws.amazon.com/step-functions/latest/dg/amazon-... The reason being that, for a Lambda function, and independently of your execution logic, the Lambda execution in any case terminates, ( depending on configuration), in the worst case after 15 min. So also terminates the corresponding billing for that execution also. With Step Functions or similar orchestration, a logic error could expose you to create an infinite loop of continuous invocations and the corresponding never ending growing invoice accompanying it :-)

moltar 5y ago
Good advice. But a small correction. Step Functions have a limit too. I think it’s 25,000 transitions per execution.