4 ms·
Where is a service that doesn't focus specifically on 1.) data pipelines / data-science 2.) cicd / build pipelines 3.) ... you name it I mean, just a servic
by valenterry 4y ago
Where is a service that doesn't focus specifically on
1.) data pipelines / data-science
2.) cicd / build pipelines
3.) ... you name it
I mean, just a service that gives me all the groundwork to build one of the above myself.
Is there something like that?
- galdor 4y agoShameless self-promotion: I built Eventline (see my profile) precisely because I just wanted to schedule and run code in various environments, manually or in reaction to various events, without the complexity that goes with specialized tools. It gives you a pretty flexible job system, identities (secrets), monitoring, notifications, a web interface, a command line tool and a HTTP API. You can build lots of things above it. Do you have specific use cases in mind ?
- lemmonii 4y agoArgo Workflows
- valenterry 4y agoMore like a SaaS that I can use without setting up my own infrastructure. I guess one could turn Argo Workflows into such an SaaS.
- rhplus 4y agoAWS Simple Workflows or Azure Logic Apps are both services that let you define S2S workflows however you like without any particular bias to CI/CD or business operations. If you want to go even lower level, a framework like DTFx lets you define long-running, distributed and resilient orchestrations in code: https://github.com/Azure/durabletask https://github.com/Azure/durabletask
- jmcnevin 4y agoI'd say give Temporal (https://temporal.io https://temporal.io) a look, but there are a lot of options (https://github.com/meirwah/awesome-workflow-engines https://github.com/meirwah/awesome-workflow-engines).
- skrtskrt 4y agoSeconding Temporal, it is definitely a second-gen workflow platform with approaches that make many Airflow frustrations nonexistent. In particular, workflows are extremely durable, they can easily just sit indefinitely waiting on a condition and you can redeploy workers whenever, and workflows will resume exactly where they were.
- valenterry 4y agoWow, that looks cool. Especially since the way of defining things in code looks pretty first-class.
- deleted 4y ago[deleted]
- samhita-alla 4y agoHave you seen https://github.com/flyteorg/flyte https://github.com/flyteorg/flyte? It's a data, ML and infrastructure orchestrator. (btw, I'm a Flyte engineer)
- valenterry 4y agoBonus points for having a Scala SDK!