8 ms·

Maestro: Netflix's Workflow Orchestrator

307 points by vquemener 2y ago

halamadrid 2y ago
Very nice, Netflix has a reputation of making great OSS products. I wonder where does this stand with Conductor.
- deleted 2y ago
  [deleted]
- opiniateddev 2y ago
  Maestro is a domain specific implementation for ML and data pipelines that uses Conductor as its core https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-... https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/...
iamsanteri 2y ago
So will this serve as a stand-in replacement for something like Airflow?
- makestuff 2y ago
  Yeah, also curious if this is meant as a replacement for Airflow.
- then4p 2y ago
  I'm also missing comparisons to other existing tools like airflow, dagster, mlflow...
pantsforbirds 2y ago
This is a really great-looking project. I know I've considered building (a probably worse) version of exactly this on almost every mixed ML + Data Engineering project I've ever worked on. I'm looking forward to testing it out.
oneplane 2y ago
Looks a bit like Argo Workflows combined with Argo Events. Makes sense to have so many projects and products converge around the same endstate.
indiv0 2y ago
Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.
- opiniateddev 2y ago
  Conductor was moved here: https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor Maestro uses conductor as its core. https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/... https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-...
Sparkyte 2y ago
Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.
- sjansen 2y ago
  Putting work in a queue is only the start. Most organizations start there and gradually write ad hoc logic as they discover problems like dependencies, retries, & scheduling. Dependencies: what can be done in parallel and what must be done in sequence? For example, three tasks get pushed in the queue and only after all three finish a fourth task must be run. Retries: The concept is simple. The details are killer. For example, ifa task fails, how long should the delay between retries be? Too short and you create a retry storm. Forget to add some jitter and you get thundering hoards all retrying at the same time. Scheduling: Because cron is good enough, until it isn't. A good workflow solution provides battle tested versions of all of the above. Better yet, a great workflow solution makes it easier to keep business logic separate from plumbing so that it's easier to reason about and test.
- shawabawa3 2y ago
  workflows typically involve chains of jobs with state transitions, waits, triggers, error handling etc a lot more than just e.g. celery jobs
- nijave 2y ago
  A workflow manager implements a Choreography based saga pattern https://microservices.io/patterns/data/saga.html https://microservices.io/patterns/data/saga.html
dboreham 2y ago
Interesting. My team recently built a thing for managing long running, multi-machine, restartable, cascading batch jobs in an unrelated vehicle. Had no idea it was a category.
meliora245 2y ago
why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go
- iamspoilt 2y ago
  That's also my question.
- robryan 2y ago
  Netflix also uses temporal: https://temporal.io/in-use/netflix https://temporal.io/in-use/netflix
  - tiffanyh 2y ago
    Is Temporal still alive? (website doesn't resolve for me) EDIT: I found the GitHub page https://github.com/temporalio/temporal https://github.com/temporalio/temporal
    - sjansen 2y ago
      The site loads fine for me. See also: https://downforeveryoneorjustme.com/temporal.io https://downforeveryoneorjustme.com/temporal.io
- troebr 2y ago
  Didn't they rewrite some of Temporal's core in rust?
  - sjansen 2y ago
    They (re)wrote most of the client SDKs on a Rust core, but the Temporal server is still written in Go.
- aimazon 2y ago
  isn’t Maestro an alternative to Airflow, not Temporal? Temporal isn’t a workflow orchestrator. There’s some overlap on the internals but they’re different designs for different use cases.
gtrubetskoy 2y ago
The name Maestro has already been used for a workflow orchestrator which I worked on back in 2016. That maestro is SQL-centric and infers dependencies automatically by simply examining the SQL. It's written in Go and is BigQuery-specific (but could be easily adjusted to use any SQL-based system). https://github.com/voxmedia/maestro/ https://github.com/voxmedia/maestro/
- stepanhruda 2y ago
  With all due respect, there are so many projects. They don’t care about clashing with a repo that has 12 stars and 14 commits.
  - nijave 2y ago
    Worked at a bank that named their container "cloud" platform GCP and it was in no way related to Google facepalm
    - stavros 2y ago
      Well, if you're so unimaginative as to call your cloud platform "<companyname> cloud platform", it's not the fault of the second company whose name also starts with a G.
      - nijave 2y ago
        Worse, the G was Gaia (ironically the personification of Earth in Greek mythology). They used "Gaia" as a name for all their internal cloud platforms
        jdmichal 2y ago
        Hello fellow ex-employee of that bank. I was in a segment governed by PCI, and they wouldn't even let us touch Gaia in fear of the whole thing being declared in scope
tiffanyh 2y ago
Don't see many Java projects being posted on HN.
- xyst 2y ago
  We only upvote Go or Rust projects here ;)
jekude 2y ago
Seems like they re-engineered Temporal: https://temporal.io/ https://temporal.io/
- troebr 2y ago
  They did use Temporal at Netflix, they gave a couple presentations 2 years ago. I think this is very much not-Temporal because it relies on a DSL instead of workflow as code. I don't know if it's a scale-thing, I'm not a workflow expert but this seems more in line with the map-reduce of yore, as in you get some big fat steps and you coordinate them, although you could have coarse-grained activities in Temporal workflows. I'd be curious to see what the tradeoffs are between the two and if they still have usages for Temporal. Maybe Maestro is better for less technical people? Latency? Scale?
  - swyx 2y ago
    former temporal employee here. netflix is very big, temporal-at-netflix always coexisted with other orchestration solutions including conductor > I think this is very much not-Temporal because it relies on a DSL instead of workflow as code. yup you get it. maestro defines things as json, which just inherently limits how you can write and test it with your normal app code
    - nfkuler 2y ago
      Again, another misleading comment here. Given the freedom and responsibility culture at Netflix. There is one team/person who has been using and promoting Temporal usage and all temporal conferences. Conductor is the main orchestration product at Netflix and will continue to be the case. We have increased the usage 5x in the last 6 months alone.
hintymad 2y ago
I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies like to build workflow engines. This is a not deriding comment towards anyone or Netflix in particular. To me, such observation is worth some friendly chitchat.
- sgloutnikov 2y ago
  Naming things, cache invalidation, and workflow engines? :) https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines
  - cbsmith 2y ago
    No, it's just the two things: naming things, cache invalidation, and off by one errors.
- dinobones 2y ago
  We rolled our own workflow engine and it almost crashed one of our unrelated projects for having so many bugs and being so inflexible. I’m starting to think workflow engines are somewhat of a design smell. It’s enticing to think you can build this reusable thing once and use it for a ton of different workflows, but besides requiring more than one asynchronous step, these workflows have almost nothing in common. Different data, different APIs, different feedback required from users or other systems to continue.
  - ryanianian 2y ago
    > workflow engines are somewhat of a design smell Probably so, but the real design smell seems to be thinking of a workflow engine as a panacea for sustainable business process automation. You have to really understand the business flow before you automate it. You have to continuously update your understanding of it as it changes. You have to refactor it into sub-flows or bigger/smaller units of work. You have to have tests, tracer-bullets, and well-defined user-stories that the flows represent. Else your business flow automation accumulates process debt. Just as much as a full-code-based solution accumulates technical debt. And, just like technical debt, it's much easier (or at least more interesting) to propose a rewrite or framework change than it is to propose an investment in refactoring, testing, and gradual migrations.
rednafi 2y ago
[flagged]
skywhopper 2y ago
Advice: don’t rely on any tool open-sourced by Netflix. They have a long history of dropping support for things after they’ve announced them. Someone got a checkmark on their promotion packet by getting this blog post and code sharing out the door, but don’t build your business on a solution like this.
slt2021 2y ago
I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promotion and salary/rsu bump, ofc. People need to realize that code is a liability, anything that is not the business critical stuff that earns/makes $$$ for the company is a distraction and resource sink.
- ripped_britches 2y ago
  100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.
- jefurii 2y ago
  This sounds like the beginning of a sales pitch.
- makeset 2y ago
  > anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.
- YawningAngel 2y ago
  Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff
  - slt2021 2y ago
bjourne 2y ago
What is a workflow in this context?
- ju-st 2y ago
  I had the same question and asked Claude Sonnet, see its answer below. In the context of a workflow engine like Netflix Conductor, a workflow refers to a structured sequence of tasks or activities that need to be executed to complete a specific business process or achieve a particular goal. [...] To give you a concrete example, imagine an e-commerce platform's order processing workflow: 1. Validate order 2. Check inventory 3. Process payment 4. If payment successful: a. Reserve inventory b. Initiate shipping c. Send confirmation email 5. If payment fails: a. Cancel order b. Notify customer In this workflow, each step could be a separate microservice or function. The workflow engine would orchestrate the execution of these steps, handling the flow of data between them, managing any errors or retries, and ensuring the entire process completes successfully. Would you like me to elaborate on any specific aspect of workflows in this context? Or perhaps you're curious about how they're implemented or managed in practice?
skissane 2y ago
I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.netflix.conductor:conductor-core:2.31.5` [2] – and an outdated version of it too. [0] https://github.com/Netflix/conductor https://github.com/Netflix/conductor [1] https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor [2] https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84dfb4da78cc44a5c13d3d1/dependencies.gradle#L6 https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84d...
- halamadrid 2y ago
  Yes Netflix abandoned Conductor long time ago [0]. The other repo is built and managed by Orkes after Netflix abandoned it. [0] https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-netflix-abandons-the-open-source-project/ https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-n...
  - skissane 2y ago
    I haven't touched Conductor for a few years now, but back in 2020 I did some work trying to implement it, even submitted a few PRs – https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%3Askissane https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%... My impression of the code base, is I felt like it needed a lot of work to run in a non-Netflix environment. Which is part of why the project I was working on ended up abandoning Conductor – we were going to embed Conductor in our product as a workflow engine, we ended up building our own workflow engine from scratch instead. Another team did end up using it for some internal use cases, but scalability/reliability/etc are less of a concern for internal use cases as opposed to customer-facing ones. And then Netflix abandons it – and then they open source something else which depends on an old version of it – well, I'm happy they open source anything, but it fits with my earlier impression – throwing stuff over the fence which can be a struggle to adopt in an outside environment. Still, throwing it over the fence is better than not releasing it at all.
andbberger 2y ago
slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers from massive feature creep, a bizarre specification DSL (superset of python) and blurred resource requirement abstraction boundaries.
- slt2021 2y ago
  Academia better to learn k8s and one of the k8s-native workflow orchestrators. This is as close to FAANG grade and open source as they can get, and arguably a bit better than this repo
  - andbberger 2y ago
    for better or worse slurm is the status quo for HPC. it works, every university has a slurm cluster, people already know how to use it
- torrance 2y ago
  What about Nextflow?
  - andbberger 2y ago
    I considered Nextflow before begrudgingly settling on snakemake for my current project. Didn't record why... possibly because snakemake was already a known quantity and I was under time pressure or because I felt the task DAG would be difficult to specify in WDL. It's certainly the most mature of the bunch.
  - _Wintermute 2y ago
    Nobody wants to write or debug groovy, especially scientists who are used to python. It also causes havoc on a busy SLURM scheduler with its lack of array jobs (heard this is being fixed soon).
saturn8601 2y ago
Anyone here use Activebatch? To me it is the best software I wish had an equivalent for non enterprise users. I have tried and tried to use other "competitors" but Activebatch's simplicity of just attaching a simple MS SQL DB, installing the Windows GUI and execution agent is just click, click, click and now you have a robust GUI based automation environment where you don't have to use code...or if you want, go ahead and use code in any language if you want...but you don't have to. Airflow may be robust but it is hidden behind a complexity fence that prevents most from seeing whatever its true capability may be. The same goes for other "open source" competitors. Why can't someone just develop a robust DB backed GUI first system? I have tried online services as well, they pale in comparison. I guess the cost of maintaining extensions is what kills simpler paid offerings? Its a complete shame that ActiveBatch is walled off behind a stupid enterprise sales model. This has prevented this wonderful piece of software from being picked up by the wider community. Its like a hidden secret. :/
mianos 2y ago
Interesting how complete this is. It's almost as comprehensive as prefect.io This is a critical software infrastructure I have been promoting for years yet almost everyone thinks they don't need it.
nikhilsimha 2y ago
great job on open sourcing!
willbeddow 2y ago
I'm sure this is very nice, but the article reads as if written by AI. The first thing I'd want to see is an example workflow (both code and configuration) in a realistic use case. Instead, there's a lot of "powerful and flexible" language, but the example workflow doesn't come until halfway down, and then it's just foobar
antishatter 2y ago
Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.
- ssfak 2y ago
  For Python tasks you can check Prefect, among others..
- katrotz 2y ago
  I'd recommend Kestra[1] since it can be run on a single node [1] https://kestra.io/ https://kestra.io/
- rwky 2y ago
  Windmill is pretty lightweight and easy to deploy. https://www.windmill.dev/ https://www.windmill.dev/ you can configure it to have a single worker on the same server as the ui and database.
kabes 2y ago
It says one of the big differentiators with 'traditional workflow orchestrators' is that is supports cyclic graphs. But BPMN (and the orchestrators using it) also supports loops.
febed 2y ago
Dagster is a better alternative, because of its asset first philosophy. Task based workflows are still available if you really need it.
HugoLu88 2y ago
I'm building something in the space (orchestra) so here's my take: Folks making stuff open source and building in the open is obviously brilliant, but when it comes to "orchestrators" (as this is, and identifies) there is already so much that has been before (Airflow and so on) it's quite hard to see how this actually adds anything to the space other than another option nobody is ever going to use in a commercial setting. Shameless plug: https://getorchestra.io https://getorchestra.io
rubenfiszel 2y ago
Founder of https://windmill.dev https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can package their business logic in various formats such as Docker images, notebooks, bash script, SQL, Python, and more. Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc. You could replace Maestro with Windmill here and it would be precisely correct. Their rollup is what we call the openflow state. Main differences I see: - Windmill is written in Rust instead of Java. - Maestro relies on CockroachDB for state and us Postgresql for everything (state but also queue). I can see why they would use CockroachDB, we had to rollout our own sharding algorithms to make Windmill horizontally scale on our very large scale customer instances - Maestro is Apache 2.0 vs Windmill AGPL which is less friendly - It's backed by Netflix so infinite money but although we are profitable, we are a much smaller company - Maestro doesn't have extensive docs about self-hosting on k8s or docker-compose and either there is no UI to build stuff, or the UI is not yet well surfaced in their documentation But overall, pretty cool stuff to open-source, will keep an eye on it and benchmark it asap
- rwky 2y ago
  Been using windmill for a few months and so far it's rock solid keep it up!
- ensignavenger 2y ago
  Thanks for the great comparison! While Meastro is Apache licensed, if it depends on CockroachDB, Cokroach itslef isn't even Open Source, so that isn't great. I would rather have an AGPL codebase than a non open source dependency. Of course overtime some one could add alternative DB support.
  - jamra 2y ago
monkychop 2y ago
Eduardo
monkychop 2y ago
Hooolaa
winpickup 2y ago
[dead]
petromir 2y ago
So they abandoned https://github.com/Netflix/conductor https://github.com/Netflix/conductor to create Maestro

307 points by vquemener 2y ago

halamadrid 2y ago

Very nice, Netflix has a reputation of making great OSS products. I wonder where does this stand with Conductor.

deleted 2y ago
[deleted]
opiniateddev 2y ago
Maestro is a domain specific implementation for ML and data pipelines that uses Conductor as its core https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-... https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/...

iamsanteri 2y ago

So will this serve as a stand-in replacement for something like Airflow?

makestuff 2y ago
Yeah, also curious if this is meant as a replacement for Airflow.
then4p 2y ago
I'm also missing comparisons to other existing tools like airflow, dagster, mlflow...

pantsforbirds 2y ago

This is a really great-looking project. I know I've considered building (a probably worse) version of exactly this on almost every mixed ML + Data Engineering project I've ever worked on. I'm looking forward to testing it out.

oneplane 2y ago

Looks a bit like Argo Workflows combined with Argo Events. Makes sense to have so many projects and products converge around the same endstate.

indiv0 2y ago

Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.

opiniateddev 2y ago
Conductor was moved here: https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor Maestro uses conductor as its core. https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/... https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-...

Sparkyte 2y ago

Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.

sjansen 2y ago
Putting work in a queue is only the start. Most organizations start there and gradually write ad hoc logic as they discover problems like dependencies, retries, & scheduling. Dependencies: what can be done in parallel and what must be done in sequence? For example, three tasks get pushed in the queue and only after all three finish a fourth task must be run. Retries: The concept is simple. The details are killer. For example, ifa task fails, how long should the delay between retries be? Too short and you create a retry storm. Forget to add some jitter and you get thundering hoards all retrying at the same time. Scheduling: Because cron is good enough, until it isn't. A good workflow solution provides battle tested versions of all of the above. Better yet, a great workflow solution makes it easier to keep business logic separate from plumbing so that it's easier to reason about and test.
shawabawa3 2y ago
workflows typically involve chains of jobs with state transitions, waits, triggers, error handling etc a lot more than just e.g. celery jobs
nijave 2y ago
A workflow manager implements a Choreography based saga pattern https://microservices.io/patterns/data/saga.html https://microservices.io/patterns/data/saga.html

dboreham 2y ago

Interesting. My team recently built a thing for managing long running, multi-machine, restartable, cascading batch jobs in an unrelated vehicle. Had no idea it was a category.

meliora245 2y ago

why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go

iamspoilt 2y ago
That's also my question.
robryan 2y ago
Netflix also uses temporal: https://temporal.io/in-use/netflix https://temporal.io/in-use/netflix
- tiffanyh 2y ago
  Is Temporal still alive? (website doesn't resolve for me) EDIT: I found the GitHub page https://github.com/temporalio/temporal https://github.com/temporalio/temporal
  - sjansen 2y ago
    The site loads fine for me. See also: https://downforeveryoneorjustme.com/temporal.io https://downforeveryoneorjustme.com/temporal.io
troebr 2y ago
Didn't they rewrite some of Temporal's core in rust?
- sjansen 2y ago
  They (re)wrote most of the client SDKs on a Rust core, but the Temporal server is still written in Go.
aimazon 2y ago
isn’t Maestro an alternative to Airflow, not Temporal? Temporal isn’t a workflow orchestrator. There’s some overlap on the internals but they’re different designs for different use cases.

gtrubetskoy 2y ago

The name Maestro has already been used for a workflow orchestrator which I worked on back in 2016. That maestro is SQL-centric and infers dependencies automatically by simply examining the SQL. It's written in Go and is BigQuery-specific (but could be easily adjusted to use any SQL-based system). https://github.com/voxmedia/maestro/ https://github.com/voxmedia/maestro/

stepanhruda 2y ago
With all due respect, there are so many projects. They don’t care about clashing with a repo that has 12 stars and 14 commits.
- nijave 2y ago
  Worked at a bank that named their container "cloud" platform GCP and it was in no way related to Google facepalm
  - stavros 2y ago
    Well, if you're so unimaginative as to call your cloud platform "<companyname> cloud platform", it's not the fault of the second company whose name also starts with a G.
    - nijave 2y ago
      Worse, the G was Gaia (ironically the personification of Earth in Greek mythology). They used "Gaia" as a name for all their internal cloud platforms
      - jdmichal 2y ago
        Hello fellow ex-employee of that bank. I was in a segment governed by PCI, and they wouldn't even let us touch Gaia in fear of the whole thing being declared in scope

tiffanyh 2y ago

Don't see many Java projects being posted on HN.

xyst 2y ago
We only upvote Go or Rust projects here ;)

jekude 2y ago

Seems like they re-engineered Temporal: https://temporal.io/ https://temporal.io/

troebr 2y ago
They did use Temporal at Netflix, they gave a couple presentations 2 years ago. I think this is very much not-Temporal because it relies on a DSL instead of workflow as code. I don't know if it's a scale-thing, I'm not a workflow expert but this seems more in line with the map-reduce of yore, as in you get some big fat steps and you coordinate them, although you could have coarse-grained activities in Temporal workflows. I'd be curious to see what the tradeoffs are between the two and if they still have usages for Temporal. Maybe Maestro is better for less technical people? Latency? Scale?
- swyx 2y ago
  former temporal employee here. netflix is very big, temporal-at-netflix always coexisted with other orchestration solutions including conductor > I think this is very much not-Temporal because it relies on a DSL instead of workflow as code. yup you get it. maestro defines things as json, which just inherently limits how you can write and test it with your normal app code
  - nfkuler 2y ago
    Again, another misleading comment here. Given the freedom and responsibility culture at Netflix. There is one team/person who has been using and promoting Temporal usage and all temporal conferences. Conductor is the main orchestration product at Netflix and will continue to be the case. We have increased the usage 5x in the last 6 months alone.

hintymad 2y ago

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies like to build workflow engines. This is a not deriding comment towards anyone or Netflix in particular. To me, such observation is worth some friendly chitchat.

sgloutnikov 2y ago
Naming things, cache invalidation, and workflow engines? :) https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines
- cbsmith 2y ago
  No, it's just the two things: naming things, cache invalidation, and off by one errors.
dinobones 2y ago
We rolled our own workflow engine and it almost crashed one of our unrelated projects for having so many bugs and being so inflexible. I’m starting to think workflow engines are somewhat of a design smell. It’s enticing to think you can build this reusable thing once and use it for a ton of different workflows, but besides requiring more than one asynchronous step, these workflows have almost nothing in common. Different data, different APIs, different feedback required from users or other systems to continue.
- ryanianian 2y ago
  > workflow engines are somewhat of a design smell Probably so, but the real design smell seems to be thinking of a workflow engine as a panacea for sustainable business process automation. You have to really understand the business flow before you automate it. You have to continuously update your understanding of it as it changes. You have to refactor it into sub-flows or bigger/smaller units of work. You have to have tests, tracer-bullets, and well-defined user-stories that the flows represent. Else your business flow automation accumulates process debt. Just as much as a full-code-based solution accumulates technical debt. And, just like technical debt, it's much easier (or at least more interesting) to propose a rewrite or framework change than it is to propose an investment in refactoring, testing, and gradual migrations.

rednafi 2y ago

[flagged]

skywhopper 2y ago

Advice: don’t rely on any tool open-sourced by Netflix. They have a long history of dropping support for things after they’ve announced them. Someone got a checkmark on their promotion packet by getting this blog post and code sharing out the door, but don’t build your business on a solution like this.

slt2021 2y ago

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promotion and salary/rsu bump, ofc. People need to realize that code is a liability, anything that is not the business critical stuff that earns/makes $$$ for the company is a distraction and resource sink.

ripped_britches 2y ago
100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.
jefurii 2y ago
This sounds like the beginning of a sales pitch.
makeset 2y ago
> anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.
YawningAngel 2y ago
Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff
- slt2021 2y ago

bjourne 2y ago

What is a workflow in this context?

ju-st 2y ago
I had the same question and asked Claude Sonnet, see its answer below. In the context of a workflow engine like Netflix Conductor, a workflow refers to a structured sequence of tasks or activities that need to be executed to complete a specific business process or achieve a particular goal. [...] To give you a concrete example, imagine an e-commerce platform's order processing workflow: 1. Validate order 2. Check inventory 3. Process payment 4. If payment successful: a. Reserve inventory b. Initiate shipping c. Send confirmation email 5. If payment fails: a. Cancel order b. Notify customer In this workflow, each step could be a separate microservice or function. The workflow engine would orchestrate the execution of these steps, handling the flow of data between them, managing any errors or retries, and ensuring the entire process completes successfully. Would you like me to elaborate on any specific aspect of workflows in this context? Or perhaps you're curious about how they're implemented or managed in practice?

skissane 2y ago

I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.netflix.conductor:conductor-core:2.31.5` [2] – and an outdated version of it too. [0] https://github.com/Netflix/conductor https://github.com/Netflix/conductor [1] https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor [2] https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84dfb4da78cc44a5c13d3d1/dependencies.gradle#L6 https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84d...

halamadrid 2y ago
Yes Netflix abandoned Conductor long time ago [0]. The other repo is built and managed by Orkes after Netflix abandoned it. [0] https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-netflix-abandons-the-open-source-project/ https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-n...
- skissane 2y ago
  I haven't touched Conductor for a few years now, but back in 2020 I did some work trying to implement it, even submitted a few PRs – https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%3Askissane https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%... My impression of the code base, is I felt like it needed a lot of work to run in a non-Netflix environment. Which is part of why the project I was working on ended up abandoning Conductor – we were going to embed Conductor in our product as a workflow engine, we ended up building our own workflow engine from scratch instead. Another team did end up using it for some internal use cases, but scalability/reliability/etc are less of a concern for internal use cases as opposed to customer-facing ones. And then Netflix abandons it – and then they open source something else which depends on an old version of it – well, I'm happy they open source anything, but it fits with my earlier impression – throwing stuff over the fence which can be a struggle to adopt in an outside environment. Still, throwing it over the fence is better than not releasing it at all.

andbberger 2y ago

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers from massive feature creep, a bizarre specification DSL (superset of python) and blurred resource requirement abstraction boundaries.

slt2021 2y ago
Academia better to learn k8s and one of the k8s-native workflow orchestrators. This is as close to FAANG grade and open source as they can get, and arguably a bit better than this repo
- andbberger 2y ago
  for better or worse slurm is the status quo for HPC. it works, every university has a slurm cluster, people already know how to use it
torrance 2y ago
What about Nextflow?
- andbberger 2y ago
  I considered Nextflow before begrudgingly settling on snakemake for my current project. Didn't record why... possibly because snakemake was already a known quantity and I was under time pressure or because I felt the task DAG would be difficult to specify in WDL. It's certainly the most mature of the bunch.
- _Wintermute 2y ago
  Nobody wants to write or debug groovy, especially scientists who are used to python. It also causes havoc on a busy SLURM scheduler with its lack of array jobs (heard this is being fixed soon).

saturn8601 2y ago

Anyone here use Activebatch? To me it is the best software I wish had an equivalent for non enterprise users. I have tried and tried to use other "competitors" but Activebatch's simplicity of just attaching a simple MS SQL DB, installing the Windows GUI and execution agent is just click, click, click and now you have a robust GUI based automation environment where you don't have to use code...or if you want, go ahead and use code in any language if you want...but you don't have to. Airflow may be robust but it is hidden behind a complexity fence that prevents most from seeing whatever its true capability may be. The same goes for other "open source" competitors. Why can't someone just develop a robust DB backed GUI first system? I have tried online services as well, they pale in comparison. I guess the cost of maintaining extensions is what kills simpler paid offerings? Its a complete shame that ActiveBatch is walled off behind a stupid enterprise sales model. This has prevented this wonderful piece of software from being picked up by the wider community. Its like a hidden secret. :/

mianos 2y ago

Interesting how complete this is. It's almost as comprehensive as prefect.io This is a critical software infrastructure I have been promoting for years yet almost everyone thinks they don't need it.

nikhilsimha 2y ago

great job on open sourcing!

willbeddow 2y ago

I'm sure this is very nice, but the article reads as if written by AI. The first thing I'd want to see is an example workflow (both code and configuration) in a realistic use case. Instead, there's a lot of "powerful and flexible" language, but the example workflow doesn't come until halfway down, and then it's just foobar

antishatter 2y ago

Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.

ssfak 2y ago
For Python tasks you can check Prefect, among others..
katrotz 2y ago
I'd recommend Kestra[1] since it can be run on a single node [1] https://kestra.io/ https://kestra.io/
rwky 2y ago
Windmill is pretty lightweight and easy to deploy. https://www.windmill.dev/ https://www.windmill.dev/ you can configure it to have a single worker on the same server as the ui and database.

kabes 2y ago

It says one of the big differentiators with 'traditional workflow orchestrators' is that is supports cyclic graphs. But BPMN (and the orchestrators using it) also supports loops.

febed 2y ago

Dagster is a better alternative, because of its asset first philosophy. Task based workflows are still available if you really need it.

HugoLu88 2y ago

I'm building something in the space (orchestra) so here's my take: Folks making stuff open source and building in the open is obviously brilliant, but when it comes to "orchestrators" (as this is, and identifies) there is already so much that has been before (Airflow and so on) it's quite hard to see how this actually adds anything to the space other than another option nobody is ever going to use in a commercial setting. Shameless plug: https://getorchestra.io https://getorchestra.io

rubenfiszel 2y ago

Founder of https://windmill.dev https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can package their business logic in various formats such as Docker images, notebooks, bash script, SQL, Python, and more. Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc. You could replace Maestro with Windmill here and it would be precisely correct. Their rollup is what we call the openflow state. Main differences I see: - Windmill is written in Rust instead of Java. - Maestro relies on CockroachDB for state and us Postgresql for everything (state but also queue). I can see why they would use CockroachDB, we had to rollout our own sharding algorithms to make Windmill horizontally scale on our very large scale customer instances - Maestro is Apache 2.0 vs Windmill AGPL which is less friendly - It's backed by Netflix so infinite money but although we are profitable, we are a much smaller company - Maestro doesn't have extensive docs about self-hosting on k8s or docker-compose and either there is no UI to build stuff, or the UI is not yet well surfaced in their documentation But overall, pretty cool stuff to open-source, will keep an eye on it and benchmark it asap

rwky 2y ago
Been using windmill for a few months and so far it's rock solid keep it up!
ensignavenger 2y ago
Thanks for the great comparison! While Meastro is Apache licensed, if it depends on CockroachDB, Cokroach itslef isn't even Open Source, so that isn't great. I would rather have an AGPL codebase than a non open source dependency. Of course overtime some one could add alternative DB support.
- jamra 2y ago

monkychop 2y ago

Eduardo

monkychop 2y ago

Hooolaa

winpickup 2y ago

[dead]

petromir 2y ago

So they abandoned https://github.com/Netflix/conductor https://github.com/Netflix/conductor to create Maestro