4 ms·

Did you perhaps have an alternative solution you don't mind sharing? In my opinion yaml is good enough for gitops. Easy to read, understand, modify.

by Jnr 3y ago

SOLAR_FIELDS 3y ago
I’ve been using Dagger for this replacement specifically. But that is ci/cd specific to caching and workflow execution. For things like workflow automation and orchestration I would reach for something like Prefect or Dagster. The point is to be able to do something in an actual programming language so that you not only get typing, readability, reusability, unit testability, local execution and language specific tooling, but also that it doesn’t suck for end users to write, debug and maintain. This also gives end users an escape hatch when your abstraction is inevitably not going to be good enough for them Cuelang etc like siblings mentioned are decent enough but the real scalable solutions here are made available in general purpose programming languages.
- shahargl 3y ago
  the "workflows as code" is something we are thinking about too. I guess there are pros and cons for every approach and eventually we would want to support both (but need to start with something)
  - fredguth 3y ago
    Check temporal.io, they are used by Uber, Netflix, Datadog...
    - ransom1538 3y ago
      temporal is for managing events. How is it related to ci/cd? Generally curious.
      - SOLAR_FIELDS 3y ago
        It’s more generally a workflow manager and orchestration tool. It is kind of a more general version of the tools I mentioned, Dagster and Prefect. It can be used to spawn and manage CICD tasks asynchronously. Though I will say that Temporal's use case is probably not really well mapped to CI/CD - though it could be used for it (which is why I didn't mention it). It's primary strength is robust, long lived workflows with intelligent retries and the like - you typically want your CI/CD to be as fast as possible and while you want retries and resilience etc it's not as important as some other things (like being hermetic, reproducible, and cached).
        lorendsr 3y ago
        Here's a Temporal v Prefect comparison I wrote: https://community.temporal.io/t/what-are-the-pros-and-cons-of-temporal-with-respect-to-prefect/5671/3 https://community.temporal.io/t/what-are-the-pros-and-cons-o... tldr is Temporal is more general-purpose: for reliable programming in general, vs data pipelines. It supports many languages, and combining languages, has features like querying & signaling, and can do very high scale. CI/CD is a common use case for Temporal—used by HashiCorp, Flightcontrol, Netflix: https://www.youtube.com/watch?v=LliBP7YMGyA https://www.youtube.com/watch?v=LliBP7YMGyA

Did you perhaps have an alternative solution you don't mind sharing? In my opinion yaml is good enough for gitops. Easy to read, understand, modify.

by Jnr 3y ago

Did you perhaps have an alternative solution you don't mind sharing? In my opinion yaml is good enough for gitops. Easy to read, understand, modify.

SOLAR_FIELDS 3y ago

I’ve been using Dagger for this replacement specifically. But that is ci/cd specific to caching and workflow execution. For things like workflow automation and orchestration I would reach for something like Prefect or Dagster. The point is to be able to do something in an actual programming language so that you not only get typing, readability, reusability, unit testability, local execution and language specific tooling, but also that it doesn’t suck for end users to write, debug and maintain. This also gives end users an escape hatch when your abstraction is inevitably not going to be good enough for them Cuelang etc like siblings mentioned are decent enough but the real scalable solutions here are made available in general purpose programming languages.

shahargl 3y ago
the "workflows as code" is something we are thinking about too. I guess there are pros and cons for every approach and eventually we would want to support both (but need to start with something)
- fredguth 3y ago
  Check temporal.io, they are used by Uber, Netflix, Datadog...
  - ransom1538 3y ago
    temporal is for managing events. How is it related to ci/cd? Generally curious.
    - SOLAR_FIELDS 3y ago
      It’s more generally a workflow manager and orchestration tool. It is kind of a more general version of the tools I mentioned, Dagster and Prefect. It can be used to spawn and manage CICD tasks asynchronously. Though I will say that Temporal's use case is probably not really well mapped to CI/CD - though it could be used for it (which is why I didn't mention it). It's primary strength is robust, long lived workflows with intelligent retries and the like - you typically want your CI/CD to be as fast as possible and while you want retries and resilience etc it's not as important as some other things (like being hermetic, reproducible, and cached).
      - lorendsr 3y ago
        Here's a Temporal v Prefect comparison I wrote: https://community.temporal.io/t/what-are-the-pros-and-cons-of-temporal-with-respect-to-prefect/5671/3 https://community.temporal.io/t/what-are-the-pros-and-cons-o... tldr is Temporal is more general-purpose: for reliable programming in general, vs data pipelines. It supports many languages, and combining languages, has features like querying & signaling, and can do very high scale. CI/CD is a common use case for Temporal—used by HashiCorp, Flightcontrol, Netflix: https://www.youtube.com/watch?v=LliBP7YMGyA https://www.youtube.com/watch?v=LliBP7YMGyA