6 ms·

Netflix has open-sourced its Maestro Workflow Orchestrator

141 points by kaypee901 2y ago

vb-8448 2y ago
This appears to be the high level architecture: https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt718rSvgh2Nclv https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt.... To me seems to overengineered for most of the companies outside there, how many people do you need to manage it?
- xyst 2y ago
  In the blog, it claims: > … schedule hundreds of thousands of workflows, millions of jobs every day and operate with a strict SLO of less than 1 minute of scheduler introduced delay even when there are spikes in the traffic. I wouldn’t necessarily call it over engineered if it can handle this type of workload. But if you are running a fraction of these workloads with a much more relaxed SLO (service level objection), then yea this is overengineered. Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
  - galaxyLogic 2y ago
    Over-engineering is in the eye of the beholder
  - silisili 2y ago
    > Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits. In other words, learn it now and get it on your resume, because within a year every 200-hit-per-day service and site is going to rearchitected to use it.
- nrclark 2y ago
  That chart has real Krazam Microservices vibes to it.
  - darknavi 2y ago
    > First we call Bingo to get our Name-o.
    - earthboundkid 2y ago
      Love Galactus.
  - alex_lav 2y ago
    All time great Youtube video. Still heavily relevant today.
  - rubslopes 2y ago
    Here's a video of Primeagen, a YouTuber who uses to be a dev at Netflix, watching that video and then telling a very similar story that happened at Netflix: https://youtu.be/s-vJcOfrvi0 https://youtu.be/s-vJcOfrvi0
    - rjbwork 2y ago
      Brilliant. Currently dealing with webs of microservices myself. It's truly wild to watch something akin to the cambrian explosion with these things.
- apwell23 2y ago
  this looks like someone took their systems design interview too seriously
qntty 2y ago
I was hoping to try this out, but it doesn't build on my mac :/ Here's their tech blog post about it: https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-...
- xyst 2y ago
  [flagged]
  - razemio 2y ago
    Not sure about the others but java is just an issue, if the devs did not care about cross platform and is mostly easy to fix if native libraries are absent.
- vips7L 2y ago
  It doesn't build on windows either. It looks to be missing some dependency. > Did not resolve 'com.github.vmg.protogen:protogen-annotations:1.0.0' which is part of the dependency lock state
  - qntty 2y ago
    I get a similar error
    - vips7L 2y ago
      I couldn’t find that package on Maven Central either. Though its source is on GitHub.
- sitkack 2y ago
  Use podman desktop, or is it an Arm issue?
- _joel 2y ago
  I've been able to build and run it sucessfully via IntelliJ, on an M2 Air. Takes a few mins to get all the dependencies and build, but it definitely works. Perhaps you're missing some dependency steps before hand if you're hand cranking gradle.
rickette 2y ago
I was just thinking didn't Netflix already have its own workflow engine: https://github.com/Netflix/conductor https://github.com/Netflix/conductor, but found out it's archived/EOL so this makes more sense now.
- deleted 2y ago
  [deleted]
- opiniateddev 2y ago
  Conductor was moved here: https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor
  - instanceofdev 2y ago
    And it is supported! https://orkes-conductor.slack.com https://orkes-conductor.slack.com
- halamadrid 2y ago
  https://news.ycombinator.com/reply?id=41040680&goto=item%3Fid%3D41037745%2341040680 https://news.ycombinator.com/reply?id=41040680&goto=item%3Fi...
deleted 2y ago
[deleted]
wg0 2y ago
After almost demise of nomad, market needs another general purpose orchestrator that isn't Kubernetes or docker swarm. This is not it however.
Keyframe 2y ago
So, is this something like prefect, dagster, flyte..?
- sa-code 2y ago
  How could you forget airflow?
  - Keyframe 2y ago
    Lots of drinking and therapy.
    - chse_cake 2y ago
      this comment needs to be at the top
ChrisArchitect 2y ago
[dupe] More discussion on official post: https://news.ycombinator.com/item?id=41037745 https://news.ycombinator.com/item?id=41037745
opiniateddev 2y ago
An abstraction on top of Conductor: https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/...
- pandaswo 2y ago
  This is not true. Looks like they needed some functionalities and just used a small part of conductor.
Thaxll 2y ago
Interesting that they use CockroachDB.
pshirshov 2y ago
1) Scheduling seems to be primitive? (strategy: sequential) 2) That's seems to be just a DAG executor? No high-level frontend? 3) No executiun context? 4) No typings? 5) No concept of a stream? It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.
- pests 2y ago
  Did you miss the sections on how this allows cyclic graphs and not just DAG? It's hard to take your comment seriously when this was one of the earliest talking points.
  - pshirshov 2y ago
    > In Maestro, a workflow is a DAG (Directed acyclic graph) of individual units of job definition called Steps.
    - pests 2y ago
      The link was changed at some point, from the original Medium article: > Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc. https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78 https://netflixtechblog.com/maestro-netflixs-workflow-orches...
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

141 points by kaypee901 2y ago

This appears to be the high level architecture: https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt718rSvgh2Nclv https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt.... To me seems to overengineered for most of the companies outside there, how many people do you need to manage it?

xyst 2y ago
In the blog, it claims: > … schedule hundreds of thousands of workflows, millions of jobs every day and operate with a strict SLO of less than 1 minute of scheduler introduced delay even when there are spikes in the traffic. I wouldn’t necessarily call it over engineered if it can handle this type of workload. But if you are running a fraction of these workloads with a much more relaxed SLO (service level objection), then yea this is overengineered. Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits.
- galaxyLogic 2y ago
  Over-engineering is in the eye of the beholder
- silisili 2y ago
  > Like any solution from FAANG, or “big tech”. It works for _them_ at scale but for 90% of companies the sheer management of this service will quickly outweigh the benefits. In other words, learn it now and get it on your resume, because within a year every 200-hit-per-day service and site is going to rearchitected to use it.
nrclark 2y ago
That chart has real Krazam Microservices vibes to it.
- darknavi 2y ago
  > First we call Bingo to get our Name-o.
  - earthboundkid 2y ago
    Love Galactus.
- alex_lav 2y ago
  All time great Youtube video. Still heavily relevant today.
- rubslopes 2y ago
  Here's a video of Primeagen, a YouTuber who uses to be a dev at Netflix, watching that video and then telling a very similar story that happened at Netflix: https://youtu.be/s-vJcOfrvi0 https://youtu.be/s-vJcOfrvi0
  - rjbwork 2y ago
    Brilliant. Currently dealing with webs of microservices myself. It's truly wild to watch something akin to the cambrian explosion with these things.
apwell23 2y ago
this looks like someone took their systems design interview too seriously

qntty 2y ago

I was hoping to try this out, but it doesn't build on my mac :/ Here's their tech blog post about it: https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c https://netflixtechblog.com/orchestrating-data-ml-workflows-...

xyst 2y ago
[flagged]
- razemio 2y ago
  Not sure about the others but java is just an issue, if the devs did not care about cross platform and is mostly easy to fix if native libraries are absent.
vips7L 2y ago
It doesn't build on windows either. It looks to be missing some dependency. > Did not resolve 'com.github.vmg.protogen:protogen-annotations:1.0.0' which is part of the dependency lock state
- qntty 2y ago
  I get a similar error
  - vips7L 2y ago
    I couldn’t find that package on Maven Central either. Though its source is on GitHub.
sitkack 2y ago
Use podman desktop, or is it an Arm issue?
_joel 2y ago
I've been able to build and run it sucessfully via IntelliJ, on an M2 Air. Takes a few mins to get all the dependencies and build, but it definitely works. Perhaps you're missing some dependency steps before hand if you're hand cranking gradle.

rickette 2y ago

I was just thinking didn't Netflix already have its own workflow engine: https://github.com/Netflix/conductor https://github.com/Netflix/conductor, but found out it's archived/EOL so this makes more sense now.

deleted 2y ago
[deleted]
opiniateddev 2y ago
Conductor was moved here: https://github.com/conductor-oss/conductor https://github.com/conductor-oss/conductor
- instanceofdev 2y ago
  And it is supported! https://orkes-conductor.slack.com https://orkes-conductor.slack.com
halamadrid 2y ago
https://news.ycombinator.com/reply?id=41040680&goto=item%3Fid%3D41037745%2341040680 https://news.ycombinator.com/reply?id=41040680&goto=item%3Fi...

deleted 2y ago

[deleted]

wg0 2y ago

After almost demise of nomad, market needs another general purpose orchestrator that isn't Kubernetes or docker swarm. This is not it however.

Keyframe 2y ago

So, is this something like prefect, dagster, flyte..?

sa-code 2y ago
How could you forget airflow?
- Keyframe 2y ago
  Lots of drinking and therapy.
  - chse_cake 2y ago
    this comment needs to be at the top

ChrisArchitect 2y ago

[dupe] More discussion on official post: https://news.ycombinator.com/item?id=41037745 https://news.ycombinator.com/item?id=41037745

opiniateddev 2y ago

An abstraction on top of Conductor: https://github.com/Netflix/maestro/blob/main/maestro-engine/src/main/java/com/netflix/conductor/core/execution/MaestroWorkflowExecutor.java https://github.com/Netflix/maestro/blob/main/maestro-engine/...

pandaswo 2y ago
This is not true. Looks like they needed some functionalities and just used a small part of conductor.

Thaxll 2y ago

Interesting that they use CockroachDB.

pshirshov 2y ago

1) Scheduling seems to be primitive? (strategy: sequential) 2) That's seems to be just a DAG executor? No high-level frontend? 3) No executiun context? 4) No typings? 5) No concept of a stream? It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.

pests 2y ago
Did you miss the sections on how this allows cyclic graphs and not just DAG? It's hard to take your comment seriously when this was one of the earliest talking points.
- pshirshov 2y ago
  > In Maestro, a workflow is a DAG (Directed acyclic graph) of individual units of job definition called Steps.
  - pests 2y ago
    The link was changed at some point, from the original Medium article: > Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc. https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78 https://netflixtechblog.com/maestro-netflixs-workflow-orches...

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]