4 ms·

What you're asking for is basically Argo Workflows, I think. Not that I recommend it. It's quite lovely in principle, but really flawed in practice. It's YAML

What you're asking for is basically Argo Workflows, I think. Not that I recommend it. It's quite lovely in principle, but really flawed in practice. It's YAML hell on top of Kubernetes hell (and I say that as someone who loves Kubernetes and uses it for everything every day). Having worked with some of these tools, what I've started to wish for is a system where pipelines are written in just plain code. I'd like to run and debug my pipeline as a normal, compiled program that I can run on my own machine using the tools I already use to build software, including things like debuggers and unit testing tools. Then, when I'm really to put it into production, I want a super scalable scheduler to take my program and run it across dozens of autoscaling nodes in Kubernetes or whatever. The only thing I've come across that uses this model is Temporal, but it's got a rather different execution model than a straightforward pipeline scheduler.

leafmeal 4y ago
Check out Toil: https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil
- atombender 4y ago
  Looks Python-specific? I don't use Python.
brylie 4y ago
Prefect might work for your use-case: https://docs.prefect.io/ https://docs.prefect.io/
- FridgeSeal 4y ago
  IMO the issue with things like Prefect (and the sinking comment suggestion of Toil) is the inextricable coupling with Python. We use Argo at my work, because we made the “unit of work” just a container that’s run by K8s. There isn’t a Python runtime around to run this stuff, and that’s a good thing. Same as with Pulumi, I’ve seen what devs do when you let them write arbitrary code anywhere-you end up with code that lives everywhere, confusingly cuts across different infrastructural layers with wild-abandon. It becomes difficult to on-board, difficult to maintain, the blast radius becomes unknown, and just because it’s Python, it will randomly break in 6 months because nobody pinned a transient dependency. YAML is awful. Argo could do with some trimming in its configs, but I’ll take it, and other “constrained” alternatives over arbitrary-code-in-Python-that-controls-resources-that-run-more-arbitrary code.
- atombender 4y ago
  I don't see the attraction of a language-specific framework. I don't use Python, and adding yet another language into the mix would just add complexity.
lmm 4y ago
That's one of the things I really like about Spark (though it's probably a lot less general than what you're after - running a pipeline of steps that are Scala code rather than arbitrary commands). I can run, test, and debug my pipeline locally in my IDE where it's just plain old Scala. Then I can change one line and run my same pipeline on the company Hadoop cluster.

leafmeal 4y ago

Check out Toil: https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil

atombender 4y ago
Looks Python-specific? I don't use Python.

brylie 4y ago

Prefect might work for your use-case: https://docs.prefect.io/ https://docs.prefect.io/

FridgeSeal 4y ago
IMO the issue with things like Prefect (and the sinking comment suggestion of Toil) is the inextricable coupling with Python. We use Argo at my work, because we made the “unit of work” just a container that’s run by K8s. There isn’t a Python runtime around to run this stuff, and that’s a good thing. Same as with Pulumi, I’ve seen what devs do when you let them write arbitrary code anywhere-you end up with code that lives everywhere, confusingly cuts across different infrastructural layers with wild-abandon. It becomes difficult to on-board, difficult to maintain, the blast radius becomes unknown, and just because it’s Python, it will randomly break in 6 months because nobody pinned a transient dependency. YAML is awful. Argo could do with some trimming in its configs, but I’ll take it, and other “constrained” alternatives over arbitrary-code-in-Python-that-controls-resources-that-run-more-arbitrary code.
atombender 4y ago
I don't see the attraction of a language-specific framework. I don't use Python, and adding yet another language into the mix would just add complexity.

lmm 4y ago

That's one of the things I really like about Spark (though it's probably a lot less general than what you're after - running a pipeline of steps that are Scala code rather than arbitrary commands). I can run, test, and debug my pipeline locally in my IDE where it's just plain old Scala. Then I can change one line and run my same pipeline on the company Hadoop cluster.