10 ms·

Amazon Simple Workflow - Cloud-Based Workflow Management

104 points by jeffbarr 15y ago

deleted 15y ago
[deleted]
verelo 15y ago
This is a great idea! Actually this may have just saved me a bunch of coding. Amazon you're starting to rule the world and its a bit scary...but for now i'm ok with it.
- jeffbarr 15y ago
  Have fun with it, and let me know what you come up with.
pieter 15y ago
This really sounds too enterprisey. It's unclear to me what the benefit is of using SWF, and lost my interest when it started listing business definitions.
- flyt 15y ago
  "too enterprisey" = a service that large businesses can integrate deeply into their existing applications, Amazon can bill them for, and yet another compelling reason for companies with obscene amounts of money to dip their toes into AWS. Look at this feature next to the AWS Storage Gateway, recently announced: these are features to get businesses using portions of AWS so that they demonstrate their reliability and cost effectiveness at a small scale. Once a business is comfortable dealing with Amazon then they provide compelling reasons to expand into more of their services (EBS booting + attaching Storage Gateway volumes to EC2 instances for braindead simple cloud compute migrations away from expensive, inflexible on-premises datacenters, SES, SQS, SNS integration with SWF, etc) Never forget that Amazon is playing the long game.
- tim_sw 15y ago
  One usecase is analytics workflows. It can also be used to build more elaborate machine learning models (say over data that is too large to fit in one box, and requires distributed coordination)
- sriramk 15y ago
  Look past that. This is super useful - high scale services often have loosely coupled services which pass data from one stage to the other and act on it based on various decision factors. Almost everyone I know invents their own mechanism for gluing these things together - the SWF seems super useful there. Windows Azure (my former team) actually tried to do this with Azure Workflow Services several years ago, but that got killed around a year into it's existence.
Krish123456 15y ago
This is Amazon's attempt at next gen PaaS. You can find more details here http://www.cloudave.com/17315/remember-next-gen-paas-and-aws-here-is-the-second-piece-to-the-puzzle/ http://www.cloudave.com/17315/remember-next-gen-paas-and-aws...
- ridruejo 15y ago
  Yep, your analysis is spot on. Good article
newhouseb 15y ago
I think this is meant to be a parallel to things like celery and gearman, i.e. a task queue. A table mapping between AWS services and what components they replace in a typical DIY stack would be helpful for a lot of people (i.e. SQS = RabbitMQ, SWF = celery, ELB = nginx, DynamoDB = Cassandra(?)).
- JeffJenkins 15y ago
  Cerery doesn't do workflows with dependencies, just tasks. This is more like Storm: https://github.com/nathanmarz/storm https://github.com/nathanmarz/storm
  - kennu 15y ago
    I wonder if Celery could still use SWF as its backend in the future? Instead of having to manage your own EC2 instances and databases for the Celery daemons.
  - newhouseb 15y ago
    Well celery does have subtasks that it can wait and join on (see http://ask.github.com/celery/userguide/tasksets.html http://ask.github.com/celery/userguide/tasksets.html), but I suppose that the logic to do such things would have to occur within a running task as opposed to SWF where such things are handled by the platform itself.
chubot 15y ago
Looks cool, but the link to the API reference is missing. I'd like to see some code examples and how you write non-java workflows.
- zootm 15y ago
  From the product page: http://aws.amazon.com/documentation/swf/ http://aws.amazon.com/documentation/swf/
jbarham 15y ago
I didn't see it linked from the blog post, so here's the official SWF page: http://aws.amazon.com/swf/ http://aws.amazon.com/swf/
davidw 15y ago
Interesting. I just implemented a workflow thing for LiberWriter. It's a little unclear how the Amazon system might integrate with, say, an open ended comment system. Definitely something I'll be keeping an eye on though.
joshwa 15y ago
I'm actually in the middle of designing a Workflow-as-a-service for my current employer (based on jBPM 5). However, this isn't Workflow-with-a-capital-W. It's not even a state machine. SWF is actually a distributed-processing task queue with persistent state and execution history. Which is pretty cool.
troels 15y ago
It sounds scary to me to have something that core to your business be running on a rented platform.
- jwilliams 15y ago
  Have you mapped out what services your business consumes and the relationships? e.g. Your bank accounts, payment systems, deliveries, suppliers. I think you'd be surprised.
- ridruejo 15y ago
  Yes and no. There is a risk on building your business or app on somebody else's platform, but there is also risk in investing your time and resources on trying to recreate the same stuff (and be stuck maintaining it) rather than on your true core business. AWS is quickly becoming the standard runtime to build your application on top of (like win32 was in the 90s) and for a lot of scenarios the rewards on building on top of their platform are bigger than the potential for lock in.
nubela 15y ago
I don't get it, can someone explain what problem it is trying to solve? By means of an example?
bwarp 15y ago
At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric. It has an arguably better feature set (including extensive transaction, correlation, failover, load balancing support), is extremely robust and can be deployed on EC2 or your own infrastructure (or both at the same time - you can scale your workers up on EC2 for example). The learning curve to do something useful is about 2 days. It's pretty cool actually.
- dbattaglia 15y ago
  > At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric. I've been researching/demoing App Fabric at my job for a few weeks now to possibly use for our new SOA layer, and that was the first thing I thought when I saw the SWF email from Amazon this morning. Of course anyone not working in IIS or Azure (ie Linux or AWS) would get no benefit out of App Fabric, and this could be an excellent alternative regardless (judging by the high quality of Amazon's other services I mean).
j45 15y ago
First thought: D'oh, why didn't I think of this? New resolution: consider everything I do from a cloud based perspective. My Experience: A few years ago I considered windows workflow and built my own simplified version for a project. WF tooling at that time was still evolving. The requirements were all over the place and I thought it might be prudent in this case to build something to expect the worst. As Murphy usually does, the worst never came almost 5 years later. The workflow engine works well, and simply enough for a simple event based system. It's more of a routing system and I'm fine with it. Where ASW catches my eye is the admin tools. Like most apps the admin tools are among the most complex parts of the app. In ASW, so far I'm recognizing a certain simplicity.. in a generalized sense for a workflow system. If I tried to do the same in another workflow technology, I would almost certainly be running into more configuration upfront, and the admin tooling to create and manage workflows from a UI perspective would likely be either too basic or to overkill for what I needed. That's the crux of generalizing a solution to a problem, it risks being so much bigger than we need that it adds overhead to what we were liking to simplify. ASW initially strikes me as being much simpler. It could be useful to managing state simply across distributed applications, specifically in the cloud. Not sure how the pricing would work though, that seems like a lot of money to pay for logic gates and a state machine, but I could be trivializing. Now.. just to think up a simple fun utility or project to try this out with.. :)
redguava 15y ago
I really love this new tool. I think Amazon has some amazing technologies and services. Why don't they hire a designer and awesome UX guy and fix their terrible interfaces? I know most people use the API's, and I realise it's not really stifling their business, but I still don't see why a little more effort can't go there. I am sure there are a lot of people around the world that would be very happy for the console to have some more features and be a bit easier to use.
- spydum 15y ago
  This has puzzled me for quite a while as well. The AWS Console isn't just awkward, it's lacking some of the features the API exposes if I recall.
  - ceejayoz 15y ago
    Yep. It seems to be impossible to delete an SSL certificate for an ELB via the console, for example. You can set one up, but not remove it - for that you need the CLI tools.
amalag 15y ago
This is very cool. We are working on a website for a complex workflow. This could take all of the business logic out of the web application (done via rails state plugins) and move it to the amazon service. Then just make semi-dumb forms to accomplish each state. I think that is the goal of this, you just have to code individual pages which coordinate with your database models and hit the API as the business layer.
meanguy 15y ago
Depends on your volume, of course, but seems expensive. 1 million jobs scheduled, 4 steps per job? $200. And you're locked into their API. You can have 8 workers each polling an Amazon SQS queue once a second for $19/month. Having had huge problems with their queues (3 minute lags, spurious duplicates) I'm wary of Amazon's polling-based API. SWF's long-poll feature is smart versus SQS, however.
dbecker 15y ago
Using (and even researching) new AWS services has always been surprisingly frustrating to me. You have to sift through buzzword-filled ad copy and corporate-speak to find any explanation of what the product is or how to use it.
Skywing 15y ago
This looks very applicable to what I'm working on right now. I've built my own naive implementation of a distributed worker queue, using redis as the basic centralized queue. There are many edge cases where redis lpush and rpop will let me down, so as I build my tool, I know that I'll eventually have to move to an actual message queue. I've stayed away from RabbitMQ, or even something like ASW, while developing this project of mine because they're a lot of work to integrate, on their own. ASW looks awesome but it looks as though it'd take all of my focus to integrate it. It's something I wish I could use right now but don't think I'll be able to until I've finished most of my development. I guess I'm just saying that ASW looks complex.

104 points by jeffbarr 15y ago

deleted 15y ago

[deleted]

This is a great idea! Actually this may have just saved me a bunch of coding. Amazon you're starting to rule the world and its a bit scary...but for now i'm ok with it.

jeffbarr 15y ago
Have fun with it, and let me know what you come up with.

pieter 15y ago

This really sounds too enterprisey. It's unclear to me what the benefit is of using SWF, and lost my interest when it started listing business definitions.

flyt 15y ago
"too enterprisey" = a service that large businesses can integrate deeply into their existing applications, Amazon can bill them for, and yet another compelling reason for companies with obscene amounts of money to dip their toes into AWS. Look at this feature next to the AWS Storage Gateway, recently announced: these are features to get businesses using portions of AWS so that they demonstrate their reliability and cost effectiveness at a small scale. Once a business is comfortable dealing with Amazon then they provide compelling reasons to expand into more of their services (EBS booting + attaching Storage Gateway volumes to EC2 instances for braindead simple cloud compute migrations away from expensive, inflexible on-premises datacenters, SES, SQS, SNS integration with SWF, etc) Never forget that Amazon is playing the long game.
tim_sw 15y ago
One usecase is analytics workflows. It can also be used to build more elaborate machine learning models (say over data that is too large to fit in one box, and requires distributed coordination)
sriramk 15y ago
Look past that. This is super useful - high scale services often have loosely coupled services which pass data from one stage to the other and act on it based on various decision factors. Almost everyone I know invents their own mechanism for gluing these things together - the SWF seems super useful there. Windows Azure (my former team) actually tried to do this with Azure Workflow Services several years ago, but that got killed around a year into it's existence.

Krish123456 15y ago

This is Amazon's attempt at next gen PaaS. You can find more details here http://www.cloudave.com/17315/remember-next-gen-paas-and-aws-here-is-the-second-piece-to-the-puzzle/ http://www.cloudave.com/17315/remember-next-gen-paas-and-aws...

ridruejo 15y ago
Yep, your analysis is spot on. Good article

newhouseb 15y ago

I think this is meant to be a parallel to things like celery and gearman, i.e. a task queue. A table mapping between AWS services and what components they replace in a typical DIY stack would be helpful for a lot of people (i.e. SQS = RabbitMQ, SWF = celery, ELB = nginx, DynamoDB = Cassandra(?)).

JeffJenkins 15y ago
Cerery doesn't do workflows with dependencies, just tasks. This is more like Storm: https://github.com/nathanmarz/storm https://github.com/nathanmarz/storm
- kennu 15y ago
  I wonder if Celery could still use SWF as its backend in the future? Instead of having to manage your own EC2 instances and databases for the Celery daemons.
- newhouseb 15y ago
  Well celery does have subtasks that it can wait and join on (see http://ask.github.com/celery/userguide/tasksets.html http://ask.github.com/celery/userguide/tasksets.html), but I suppose that the logic to do such things would have to occur within a running task as opposed to SWF where such things are handled by the platform itself.

chubot 15y ago

Looks cool, but the link to the API reference is missing. I'd like to see some code examples and how you write non-java workflows.

zootm 15y ago
From the product page: http://aws.amazon.com/documentation/swf/ http://aws.amazon.com/documentation/swf/

jbarham 15y ago

I didn't see it linked from the blog post, so here's the official SWF page: http://aws.amazon.com/swf/ http://aws.amazon.com/swf/

davidw 15y ago

Interesting. I just implemented a workflow thing for LiberWriter. It's a little unclear how the Amazon system might integrate with, say, an open ended comment system. Definitely something I'll be keeping an eye on though.

joshwa 15y ago

I'm actually in the middle of designing a Workflow-as-a-service for my current employer (based on jBPM 5). However, this isn't Workflow-with-a-capital-W. It's not even a state machine. SWF is actually a distributed-processing task queue with persistent state and execution history. Which is pretty cool.

troels 15y ago

It sounds scary to me to have something that core to your business be running on a rented platform.

jwilliams 15y ago
Have you mapped out what services your business consumes and the relationships? e.g. Your bank accounts, payment systems, deliveries, suppliers. I think you'd be surprised.
ridruejo 15y ago
Yes and no. There is a risk on building your business or app on somebody else's platform, but there is also risk in investing your time and resources on trying to recreate the same stuff (and be stuck maintaining it) rather than on your true core business. AWS is quickly becoming the standard runtime to build your application on top of (like win32 was in the 90s) and for a lot of scenarios the rewards on building on top of their platform are bigger than the potential for lock in.

nubela 15y ago

I don't get it, can someone explain what problem it is trying to solve? By means of an example?

bwarp 15y ago

At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric. It has an arguably better feature set (including extensive transaction, correlation, failover, load balancing support), is extremely robust and can be deployed on EC2 or your own infrastructure (or both at the same time - you can scale your workers up on EC2 for example). The learning curve to do something useful is about 2 days. It's pretty cool actually.

dbattaglia 15y ago
> At the risk of sounding like a Microsoft shill (I'm not), you can already have your cake and eat it if you use Windows Workflow on top of AppFabric. I've been researching/demoing App Fabric at my job for a few weeks now to possibly use for our new SOA layer, and that was the first thing I thought when I saw the SWF email from Amazon this morning. Of course anyone not working in IIS or Azure (ie Linux or AWS) would get no benefit out of App Fabric, and this could be an excellent alternative regardless (judging by the high quality of Amazon's other services I mean).

j45 15y ago

First thought: D'oh, why didn't I think of this? New resolution: consider everything I do from a cloud based perspective. My Experience: A few years ago I considered windows workflow and built my own simplified version for a project. WF tooling at that time was still evolving. The requirements were all over the place and I thought it might be prudent in this case to build something to expect the worst. As Murphy usually does, the worst never came almost 5 years later. The workflow engine works well, and simply enough for a simple event based system. It's more of a routing system and I'm fine with it. Where ASW catches my eye is the admin tools. Like most apps the admin tools are among the most complex parts of the app. In ASW, so far I'm recognizing a certain simplicity.. in a generalized sense for a workflow system. If I tried to do the same in another workflow technology, I would almost certainly be running into more configuration upfront, and the admin tooling to create and manage workflows from a UI perspective would likely be either too basic or to overkill for what I needed. That's the crux of generalizing a solution to a problem, it risks being so much bigger than we need that it adds overhead to what we were liking to simplify. ASW initially strikes me as being much simpler. It could be useful to managing state simply across distributed applications, specifically in the cloud. Not sure how the pricing would work though, that seems like a lot of money to pay for logic gates and a state machine, but I could be trivializing. Now.. just to think up a simple fun utility or project to try this out with.. :)

redguava 15y ago

I really love this new tool. I think Amazon has some amazing technologies and services. Why don't they hire a designer and awesome UX guy and fix their terrible interfaces? I know most people use the API's, and I realise it's not really stifling their business, but I still don't see why a little more effort can't go there. I am sure there are a lot of people around the world that would be very happy for the console to have some more features and be a bit easier to use.

spydum 15y ago
This has puzzled me for quite a while as well. The AWS Console isn't just awkward, it's lacking some of the features the API exposes if I recall.
- ceejayoz 15y ago
  Yep. It seems to be impossible to delete an SSL certificate for an ELB via the console, for example. You can set one up, but not remove it - for that you need the CLI tools.

amalag 15y ago

This is very cool. We are working on a website for a complex workflow. This could take all of the business logic out of the web application (done via rails state plugins) and move it to the amazon service. Then just make semi-dumb forms to accomplish each state. I think that is the goal of this, you just have to code individual pages which coordinate with your database models and hit the API as the business layer.

meanguy 15y ago

Depends on your volume, of course, but seems expensive. 1 million jobs scheduled, 4 steps per job? $200. And you're locked into their API. You can have 8 workers each polling an Amazon SQS queue once a second for $19/month. Having had huge problems with their queues (3 minute lags, spurious duplicates) I'm wary of Amazon's polling-based API. SWF's long-poll feature is smart versus SQS, however.

dbecker 15y ago

Using (and even researching) new AWS services has always been surprisingly frustrating to me. You have to sift through buzzword-filled ad copy and corporate-speak to find any explanation of what the product is or how to use it.

Skywing 15y ago

This looks very applicable to what I'm working on right now. I've built my own naive implementation of a distributed worker queue, using redis as the basic centralized queue. There are many edge cases where redis lpush and rpop will let me down, so as I build my tool, I know that I'll eventually have to move to an actual message queue. I've stayed away from RabbitMQ, or even something like ASW, while developing this project of mine because they're a lot of work to integrate, on their own. ASW looks awesome but it looks as though it'd take all of my focus to integrate it. It's something I wish I could use right now but don't think I'll be able to until I've finished most of my development. I guess I'm just saying that ASW looks complex.