6 ms·

Workflows and orchestration are my jam -- that's what we're trying to simplify over at https://refinery.io https://refinery.io Conductor is a cool piece of tec

Workflows and orchestration are my jam -- that's what we're trying to simplify over at https://refinery.io https://refinery.io Conductor is a cool piece of tech, and it's a well-established player in a rapidly growing space for workflow engines. I used to work at Uber and that company had microservice-hell for a while. They built the project Cadence[0] to alleviate that. It is similar to Conductor in many ways. One project to watch out for is Argo[1] which is a CNCF-backed project. There are also some attempts[2] to standardize the workflow spec. Serverless adds a whole new can of worms to what orchestration engines have to manage, and I'm very curious to see how things evolve in the future. Kubernetes adds a whole dimension of complexity to the problem space, as well. If anybody is interested in chatting about microservice hell or complex state machines for business logic, I'd be excited to chat. I'm always looking for more real world problems to help solve (as an early stage startup founder) and more exposure to what others are struggling with is helpful! 0: https://github.com/uber/cadence https://github.com/uber/cadence 1: https://argoproj.github.io/argo/ https://argoproj.github.io/argo/ 2: https://serverlessworkflow.github.io/ https://serverlessworkflow.github.io/

Answerawake 6y ago
Not sure if this relates but I was wondering if you heard of ASCI ActiveBatch? If so what are your thoughts on it? I'd like to have some easy to set up orchestration/job scheduling engine in my team so we can help clean up the tangled mess that we are in but something like Ansible seems like too much work to setup and add more jobs over time. I tried Activebatch and I wish there was some free or cheap alternative of it.
- freeqaz 6y ago
  I hadn't heard of it before. Thanks for sharing! What are the types of problems you're hoping to untangle, more specifically? I assume you want something like... A workflow where each step maps to some provisioning process? If that guess is reasonably true, then I immediately think of Terraform. You can specify dependencies and write hooks that trigger callbacks when certain steps are called. We use Terraform a bunch under-the-hood for Refinery, and it's great. I haven't used Ansible (only read about it) so I can't contrast them very well. If you want to chat some more, I'd be curious to hear more about what you're trying to build and see if there is a way we could collaborate. Or if I can offer any recommendations for software that I know of. My email is free at refinery.io :)
devmunchies 6y ago
interesting project you started. I had used snaplogic before and its similar, although not built specifically for AWS serverless. Do you have any videos of any walkthroughs, demos, or tutorials? EDIT: nevermind, i see your getting started docs have lots of video clips.
- freeqaz 6y ago
  I hadn't heard of Snaplogic before. Had to dig in. Thanks for sharing -- this is helpful context to have. There are so many companies across different spaces that it's hard to find them all! Glad you found the docs here[0] (for anybody else who is curious). We're still iterating on Refinery a lot and trying to find product market fit. If it doesn't make sense or you're confused, that feedback is super helpful for me. The goal is to build something people actually want to use and it's an iterative process to get there! 0: https://docs.refinery.io/getting-started/ https://docs.refinery.io/getting-started/
halfmatthalfcat 6y ago
My company is slowly switching from Tekton to Argo (much to my chagrin). They seem very similar in that they’re incredibly alpha and fighting over similar parts of the stack. Argo seems more CD focused right now whereas Tekton really is a “toolbox”/make what you want.
- freeqaz 6y ago
  That's funny because the Tekton website[0] has this as their first text: "Tekton is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems." Do you know why they are switching, out of curiosity? The market is definitely very young and I don't think anybody has really nailed down every use case. There are the parts of the market targeting "business model" use cases like Camunda/Zeebe. Then there are ETL-style systems like Airflow for dealing with massive data processing. And still you've got the CI/CD side of things like Argo/Tekton for automating complex build systems/running tests. Then for systems like Netflix Conductor, Uber Cadence, and AWS Step Functions (among others), you have systems trying to abstract on top of existing complex systems (microservices, etc). That's not even including low-code spaces like Zapier or IFTTT that try to target making integrations trivial. It's a crazy world! 0: https://tekton.dev/ https://tekton.dev/
- theptip 6y ago
  I had Tekton in my mental filing cabinet under "Cloud Native CI/CD", how's the UI / DAG-visualization for that toolchain? Like the sibling comment, I'd also be interested in hearing your A/B thoughts on Argo vs. Tekton for the generic workflow management usecase.
  - halfmatthalfcat 6y ago
    Tekton's UI is very bare bones. Really all you need to see job status and logs from each job/step running. It's not _good_, it's fine to get the job done. It's not good at surfacing errors during execution though. We spend a lot of time diagnosing why pipelines failed. At this point in my experience for both (I have much, much more time with Tekton though), Argo is good for CD. It's visualization and syncing of environment charts (we use Helm) is useful. Tekton is amazing for defining highly complex build/deploy workflows (which we have). Along with Tekton's sibling project Triggers, you can have full end-to-end PR -> Build -> Deploy -> Feedback into Slack/Monitoring/Alerting/etc. If I had to make a comparison (with my current knowledge level), I'd say Argo is like Angular, batteries loaded with addons you can include if you wish. Tekton is like React, you get pieces and build them into whatever you want.
CraigJPerry 6y ago
What kind of support is present for testing workflows?
- freeqaz 6y ago
  It's fairly limited right now to testing single blocks in the editor. We recently added support for Git-based development to address testing. The idea there is that each block is just a chunk of code living in a Git repo, so you can use whatever testing tools you'd like to. It's actually pretty slick -- it's bidirectional. You can use Git to make a commit, like normal, and then refresh the editor to see it. And you can make also changes in the editor and it will commit + push to Git. If you want to play with that, I can share details on how to enable it (it's fairly hidden in the UI atm). In terms of testing workflows together as a chain, that's still TBD. It is something that we'd like to ship eventually though. Along with staging environments and canary deployments. On our immediate roadmap is to get the core of the tool open-sourced so that people can start playing with it outside of our hosted platform. We have it gated behind a credit-card form right now to fight fraud and because of some technical limitations.
- mfateev 6y ago
  Temporal (Cadence) comes with Unit testing framework. The most interesting feature of it is ability to skip time automatically when workflow is blocked waiting for something. It allows testing long running workflows in milliseconds without any need to change timeouts.
jayd16 6y ago
Hey, wait a second. Are these just a modern incarnation of the enterprise service bus? Is there a significant difference?
- extrapickles 6y ago
  Yes. The only difference from what I can tell is that each service/node on the bus is a VM/Container rather than a bespoke built machine.
  - swyx 6y ago
    dragonwriter below you says the opposite - based on a plane-level analysis. would be interested if that changes your mind.
    - extrapickles 6y ago
      The real answer is it depends on what you think what responsibilities exactly ESB or Workflow Orchestration should have. dragonwriter is of the reasonable opinion that ESBs focus should be a message plane. If your of my opinion, ESBs generally include some of the features found in workflow. devonkim is also right in that ESBs in general let the business data/process bleed into the rest of the stack, where as with workflow, its completely agnostic to the business data/process. The most accurate answer would be from comparing various ESB products to Workflow Orchestration products, as every vendor/product will have slightly different opinions on where their responsibilities are.
      - swyx 6y ago
        fair enough. appreciate the thoughts!
- zo1 6y ago
  By modern you mean using "docker", have an "io domain", and use node-js/go/rust? That seems to be the pattern these days. Take existing items, rewrite/repackage and io-ify it, maybe make it a SAAS product while at it.
faizshah 6y ago
Is there a serverless workflow solution for google cloud yet?
dap 6y ago
Thanks for the resources! It seems like a lot of these implementations expect to run as independently managed services in some microservice architecture. Are you aware of any workflow engines implemented as a library inside your application, maybe with storage backed by an external database? I think that you could still have a highly available model, provided the database supported that.
- manyxcxi 6y ago
  Funny, even though Conductor isn’t designed to be- this is exactly how we use it. In a Kotlin and Spring Boot codebase nonetheless!
- AlphaSite 6y ago
  Like celery?
- mianos 6y ago
  Prefect. It's a workflow engine in a python library.
- hukola 6y ago
  https://github.com/ing-bank/baker https://github.com/ing-bank/baker is one such library for JVM languages. The state is kept in Cassandra or in-memory. We've been running production workloads with it for the last 2.5 years. A feature comparison with alternatives can be found here https://ing-bank.github.io/baker/sections/feature-comparison/ https://ing-bank.github.io/baker/sections/feature-comparison....
Ataraxy 6y ago
The idea of a pseudo flow based programming worfklow has always appealed to me because I think having logic as components that can be used as legos makes sense. It seemed like the other stuff out there was either too overkill/obtuse or even too low level for what I had in mind. So I started writing something for personal use in JS that fits my mental model of what that would be. In a nutshell it's just a runner and uses a project called moleculer which is a microservice framework that I'm more or less just using as an rpc client to execute the tasks of the workflow. One thing I've been debating with myself is how I should work with the dataflow. Would you say it's better to have the results of every node merged into a singular context across an entire flow that is passed to every other node/block (ie. always one input, the context, one output the context), or would it be better to explicitly declare and pass in inputs/outputs. Here's one project that does it that way: https://github.com/danielduarte/flowed https://github.com/danielduarte/flowed One benefit of this seems to be that nodes can run in parallel the moment their dependencies are met without having to care about each other. I poked around on your site and I like what you have to offer. You appear to do it the singular context way which I suppose makes sense because it seems like every block is an individual lambda. This also seems like overkill for my purposes because I'm interested in being able to do granular things if I wish such as a simpe rule action. I'm not sure having an individual lambda to run "if email exists return true" would be practical. That and the warm up time/latency. The other thing is managing something like npm dependencies might be annoying across blocks. Being able to create arbitrary API endpoints is nice though. I wish there were something a little more in between than all of these enterprisey offerings.

Answerawake 6y ago

Not sure if this relates but I was wondering if you heard of ASCI ActiveBatch? If so what are your thoughts on it? I'd like to have some easy to set up orchestration/job scheduling engine in my team so we can help clean up the tangled mess that we are in but something like Ansible seems like too much work to setup and add more jobs over time. I tried Activebatch and I wish there was some free or cheap alternative of it.

freeqaz 6y ago
I hadn't heard of it before. Thanks for sharing! What are the types of problems you're hoping to untangle, more specifically? I assume you want something like... A workflow where each step maps to some provisioning process? If that guess is reasonably true, then I immediately think of Terraform. You can specify dependencies and write hooks that trigger callbacks when certain steps are called. We use Terraform a bunch under-the-hood for Refinery, and it's great. I haven't used Ansible (only read about it) so I can't contrast them very well. If you want to chat some more, I'd be curious to hear more about what you're trying to build and see if there is a way we could collaborate. Or if I can offer any recommendations for software that I know of. My email is free at refinery.io :)

devmunchies 6y ago

interesting project you started. I had used snaplogic before and its similar, although not built specifically for AWS serverless. Do you have any videos of any walkthroughs, demos, or tutorials? EDIT: nevermind, i see your getting started docs have lots of video clips.

freeqaz 6y ago
I hadn't heard of Snaplogic before. Had to dig in. Thanks for sharing -- this is helpful context to have. There are so many companies across different spaces that it's hard to find them all! Glad you found the docs here[0] (for anybody else who is curious). We're still iterating on Refinery a lot and trying to find product market fit. If it doesn't make sense or you're confused, that feedback is super helpful for me. The goal is to build something people actually want to use and it's an iterative process to get there! 0: https://docs.refinery.io/getting-started/ https://docs.refinery.io/getting-started/

halfmatthalfcat 6y ago

My company is slowly switching from Tekton to Argo (much to my chagrin). They seem very similar in that they’re incredibly alpha and fighting over similar parts of the stack. Argo seems more CD focused right now whereas Tekton really is a “toolbox”/make what you want.

freeqaz 6y ago
That's funny because the Tekton website[0] has this as their first text: "Tekton is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems." Do you know why they are switching, out of curiosity? The market is definitely very young and I don't think anybody has really nailed down every use case. There are the parts of the market targeting "business model" use cases like Camunda/Zeebe. Then there are ETL-style systems like Airflow for dealing with massive data processing. And still you've got the CI/CD side of things like Argo/Tekton for automating complex build systems/running tests. Then for systems like Netflix Conductor, Uber Cadence, and AWS Step Functions (among others), you have systems trying to abstract on top of existing complex systems (microservices, etc). That's not even including low-code spaces like Zapier or IFTTT that try to target making integrations trivial. It's a crazy world! 0: https://tekton.dev/ https://tekton.dev/
theptip 6y ago
I had Tekton in my mental filing cabinet under "Cloud Native CI/CD", how's the UI / DAG-visualization for that toolchain? Like the sibling comment, I'd also be interested in hearing your A/B thoughts on Argo vs. Tekton for the generic workflow management usecase.
- halfmatthalfcat 6y ago
  Tekton's UI is very bare bones. Really all you need to see job status and logs from each job/step running. It's not _good_, it's fine to get the job done. It's not good at surfacing errors during execution though. We spend a lot of time diagnosing why pipelines failed. At this point in my experience for both (I have much, much more time with Tekton though), Argo is good for CD. It's visualization and syncing of environment charts (we use Helm) is useful. Tekton is amazing for defining highly complex build/deploy workflows (which we have). Along with Tekton's sibling project Triggers, you can have full end-to-end PR -> Build -> Deploy -> Feedback into Slack/Monitoring/Alerting/etc. If I had to make a comparison (with my current knowledge level), I'd say Argo is like Angular, batteries loaded with addons you can include if you wish. Tekton is like React, you get pieces and build them into whatever you want.

CraigJPerry 6y ago

What kind of support is present for testing workflows?

freeqaz 6y ago
It's fairly limited right now to testing single blocks in the editor. We recently added support for Git-based development to address testing. The idea there is that each block is just a chunk of code living in a Git repo, so you can use whatever testing tools you'd like to. It's actually pretty slick -- it's bidirectional. You can use Git to make a commit, like normal, and then refresh the editor to see it. And you can make also changes in the editor and it will commit + push to Git. If you want to play with that, I can share details on how to enable it (it's fairly hidden in the UI atm). In terms of testing workflows together as a chain, that's still TBD. It is something that we'd like to ship eventually though. Along with staging environments and canary deployments. On our immediate roadmap is to get the core of the tool open-sourced so that people can start playing with it outside of our hosted platform. We have it gated behind a credit-card form right now to fight fraud and because of some technical limitations.
mfateev 6y ago
Temporal (Cadence) comes with Unit testing framework. The most interesting feature of it is ability to skip time automatically when workflow is blocked waiting for something. It allows testing long running workflows in milliseconds without any need to change timeouts.

jayd16 6y ago

Hey, wait a second. Are these just a modern incarnation of the enterprise service bus? Is there a significant difference?

extrapickles 6y ago
Yes. The only difference from what I can tell is that each service/node on the bus is a VM/Container rather than a bespoke built machine.
- swyx 6y ago
  dragonwriter below you says the opposite - based on a plane-level analysis. would be interested if that changes your mind.
  - extrapickles 6y ago
    The real answer is it depends on what you think what responsibilities exactly ESB or Workflow Orchestration should have. dragonwriter is of the reasonable opinion that ESBs focus should be a message plane. If your of my opinion, ESBs generally include some of the features found in workflow. devonkim is also right in that ESBs in general let the business data/process bleed into the rest of the stack, where as with workflow, its completely agnostic to the business data/process. The most accurate answer would be from comparing various ESB products to Workflow Orchestration products, as every vendor/product will have slightly different opinions on where their responsibilities are.
    - swyx 6y ago
      fair enough. appreciate the thoughts!
zo1 6y ago
By modern you mean using "docker", have an "io domain", and use node-js/go/rust? That seems to be the pattern these days. Take existing items, rewrite/repackage and io-ify it, maybe make it a SAAS product while at it.

faizshah 6y ago

Is there a serverless workflow solution for google cloud yet?

dap 6y ago

Thanks for the resources! It seems like a lot of these implementations expect to run as independently managed services in some microservice architecture. Are you aware of any workflow engines implemented as a library inside your application, maybe with storage backed by an external database? I think that you could still have a highly available model, provided the database supported that.

manyxcxi 6y ago
Funny, even though Conductor isn’t designed to be- this is exactly how we use it. In a Kotlin and Spring Boot codebase nonetheless!
AlphaSite 6y ago
Like celery?
mianos 6y ago
Prefect. It's a workflow engine in a python library.
hukola 6y ago
https://github.com/ing-bank/baker https://github.com/ing-bank/baker is one such library for JVM languages. The state is kept in Cassandra or in-memory. We've been running production workloads with it for the last 2.5 years. A feature comparison with alternatives can be found here https://ing-bank.github.io/baker/sections/feature-comparison/ https://ing-bank.github.io/baker/sections/feature-comparison....

Ataraxy 6y ago

The idea of a pseudo flow based programming worfklow has always appealed to me because I think having logic as components that can be used as legos makes sense. It seemed like the other stuff out there was either too overkill/obtuse or even too low level for what I had in mind. So I started writing something for personal use in JS that fits my mental model of what that would be. In a nutshell it's just a runner and uses a project called moleculer which is a microservice framework that I'm more or less just using as an rpc client to execute the tasks of the workflow. One thing I've been debating with myself is how I should work with the dataflow. Would you say it's better to have the results of every node merged into a singular context across an entire flow that is passed to every other node/block (ie. always one input, the context, one output the context), or would it be better to explicitly declare and pass in inputs/outputs. Here's one project that does it that way: https://github.com/danielduarte/flowed https://github.com/danielduarte/flowed One benefit of this seems to be that nodes can run in parallel the moment their dependencies are met without having to care about each other. I poked around on your site and I like what you have to offer. You appear to do it the singular context way which I suppose makes sense because it seems like every block is an individual lambda. This also seems like overkill for my purposes because I'm interested in being able to do granular things if I wish such as a simpe rule action. I'm not sure having an individual lambda to run "if email exists return true" would be practical. That and the warm up time/latency. The other thing is managing something like npm dependencies might be annoying across blocks. Being able to create arbitrary API endpoints is nice though. I wish there were something a little more in between than all of these enterprisey offerings.