3 ms·

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around

by rb808 7y ago

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

tbrock 7y ago
The only other one I can think of off the top of my head is dagster: https://github.com/dagster-io/dagster https://github.com/dagster-io/dagster It’s made by Nick Schrock of graphql fame, among others. I’m sure there are 100s of these projects though.
- TeMPOraL 7y ago
  From the beginning of Dagster's readme: > Dagster is a system for building modern data applications. > Combining an elegant programming model and beautiful tools, Dagster allows infrastructure engineers, data engineers, and data scientists to seamlessly collaborate to process and produce the trusted, reliable data needed in today's world. Two paragraphs, communicating zero bits of information. I wish Github repositories, of all places, didn't contain such noninformative copy.
daveFNbuck 7y ago
There's also Luigi. https://github.com/spotify/luigi https://github.com/spotify/luigi
monstrado 7y ago
I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.
- jpitz 7y ago
  I'd love to see what kind of complexity you are managing there, and how.
elcritch 7y ago
I was just reading up on Broadway, written in Elixir, (https://hexdocs.pm/broadway/Broadway.html https://hexdocs.pm/broadway/Broadway.html) that provides the fundamentals of batching/job control. It’s by the creator of Elixir and is based on 7 years of libraries in the area so the fundamentals are pretty well honed.
massive 7y ago
There you go https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines
hartzell 7y ago
> Any other open source alternatives out there? Here's a big handful of them: https://github.com/pditommaso/awesome-pipeline https://github.com/pditommaso/awesome-pipeline

by rb808 7y ago

tbrock 7y ago

The only other one I can think of off the top of my head is dagster: https://github.com/dagster-io/dagster https://github.com/dagster-io/dagster It’s made by Nick Schrock of graphql fame, among others. I’m sure there are 100s of these projects though.

TeMPOraL 7y ago
From the beginning of Dagster's readme: > Dagster is a system for building modern data applications. > Combining an elegant programming model and beautiful tools, Dagster allows infrastructure engineers, data engineers, and data scientists to seamlessly collaborate to process and produce the trusted, reliable data needed in today's world. Two paragraphs, communicating zero bits of information. I wish Github repositories, of all places, didn't contain such noninformative copy.

daveFNbuck 7y ago

There's also Luigi. https://github.com/spotify/luigi https://github.com/spotify/luigi

monstrado 7y ago

I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.

jpitz 7y ago
I'd love to see what kind of complexity you are managing there, and how.

elcritch 7y ago

I was just reading up on Broadway, written in Elixir, (https://hexdocs.pm/broadway/Broadway.html https://hexdocs.pm/broadway/Broadway.html) that provides the fundamentals of batching/job control. It’s by the creator of Elixir and is based on 7 years of libraries in the area so the fundamentals are pretty well honed.

massive 7y ago

There you go https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines

hartzell 7y ago

> Any other open source alternatives out there? Here's a big handful of them: https://github.com/pditommaso/awesome-pipeline https://github.com/pditommaso/awesome-pipeline