3 ms·

A few years back, the Step Functions team was soliciting input, and the Python thing was something that came up as a suggestion. It's hard, yes, but it should

A few years back, the Step Functions team was soliciting input, and the Python thing was something that came up as a suggestion. It's hard, yes, but it should be possible to "Starlark" this and tell users that if you stick to this syntax, you can write Python and compile it down to native StepFunction syntax. Having said that, they have slightly improved the StepFunctions by adopting JSONata syntax.

anentropic 1y ago
I don't think it should need Starlark or a restricted syntax. You just want some Python code that builds up a representation of the state machine, e.g. via decorating functions the same way that Celery, Dask, Airflow, Dagster et al have done for years. Then you have some other command to take that representation and generate the actual Step Functions JSON from it (and then deploy it etc). But the missing piece is that those other tools also explicitly give you a Python execution environment, so the function you're decorating is usually the 'task' function you want to run remotely. Whereas Step Functions doesn't provide compute itself, it mostly just gives you a way to execute AWS API calls. But the non control flow tasks in my Step Functions end up mostly being Lambda invoke steps to run my Python code. I'm currently authoring Step Functions via CDK. It is clunky AF. What it needs is some moderately opinionated layer on top. Someone at AWS did have a bit of an attempt here: https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/readmelink.html#workflows https://aws-step-functions-data-science-sdk.readthedocs.io/e... but I'd really like to see something that goes further and smooths away a lot of the finickety JSON input arg/response wrangling. Also the local testing story (for Step Functions generally) is pretty meh.
- vtuulos 1y ago
  If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly. One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler

anentropic 1y ago

I don't think it should need Starlark or a restricted syntax. You just want some Python code that builds up a representation of the state machine, e.g. via decorating functions the same way that Celery, Dask, Airflow, Dagster et al have done for years. Then you have some other command to take that representation and generate the actual Step Functions JSON from it (and then deploy it etc). But the missing piece is that those other tools also explicitly give you a Python execution environment, so the function you're decorating is usually the 'task' function you want to run remotely. Whereas Step Functions doesn't provide compute itself, it mostly just gives you a way to execute AWS API calls. But the non control flow tasks in my Step Functions end up mostly being Lambda invoke steps to run my Python code. I'm currently authoring Step Functions via CDK. It is clunky AF. What it needs is some moderately opinionated layer on top. Someone at AWS did have a bit of an attempt here: https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/readmelink.html#workflows https://aws-step-functions-data-science-sdk.readthedocs.io/e... but I'd really like to see something that goes further and smooths away a lot of the finickety JSON input arg/response wrangling. Also the local testing story (for Step Functions generally) is pretty meh.

vtuulos 1y ago
If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly. One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler