6 ms·
Since this is 2023 and we are releasing things that solve X and Y problems in YML I do want to take the opportunity to question whether solving problem for X or
by SOLAR_FIELDS 3y ago
Since this is 2023 and we are releasing things that solve X and Y problems in YML I do want to take the opportunity to question whether solving problem for X or Y in YML is really the thing we should be building businesses around these days. I’ve spent the greater part of the last year or so undoing the pain of “reasonably complex GHA in YML” in my organization. It’s one of those things that sounds great conceptually, and works really well simplistically, but once your use case evolves beyond even remotely simple (for example, abstracting and maintaining this code in an engineering org in the tens of people, not even hundreds), it is a slow growing cancer that ends up being a huge time suck, unmaintainable, untestable mess, and technical debt for your org.
- seperman 3y agoI have been working on a data validation tool for a while. I even tried creating an extended YAML parser for data validation. You made me realize I wasted my time with that approach. Better now than later. I would love to talk to you before I throw away more code. Can we connect?
- shahargl 3y agoHey! I've missed your reply here. Sent an email.
- doctorpangloss 3y agoYeah but people freak out when they see Gradle, Bash, Bazel, or even wacky raw Python. The real competition is, what will LLMs write better? Because I have zero interest in learning new DSLs, I just want whatever will be most text based to use through an LLM.
- lionkor 3y agoThen, honestly, you want it to write something that is statically verifyable
- SOLAR_FIELDS 3y agoYou probably want python then. I think it's been well demonstrated that is probably the language with the largest amount of effort has gone into training LLM's to work with, in multiple facets.
- SeriousM 3y agoI'm the system architect and code quality gate in my company, and I feel you... my job is to keep things sane, consistent and extendable. GHA as well as Azure Logic Apps are booth helpful in the small scale but, omg, so far away from reusable or even able to deploy the same damn thing on different stages from code. To GHA: I find the GHA just look the same as Azure DevOps Pipelines yet they GHA don't hold your hand when designing and evaluating the steps.
- SOLAR_FIELDS 3y agoUnder the hood GHA is using the same backend as Azure Devops Pipelines so it would make sense that they look the same
- shahargl 3y agoAnd what was the solution? How did you eventually address those issues? While I agree that GitHub Actions has its downsides, it's also widely used and simple to start with, which we thought was a good approach. Would you be more comfortable with 'Zapier for Monitoring' or an alternative to 'Datadog Workflow Automation'?
- xctr94 3y agoThe complaint doesn’t seem to be about GitHub Actions, but YAML. I agree 100% percent, as soon as I saw that Keep is using YAML, I closed the tab. Nope. Nope. Nope. It’s like going back to Mongo without schemas and relational checks. We have perfectly good configuration languages with schemas, checks, imports, logic, etc. YAML is unacceptable in this profession.
- shahargl 3y agoI agree with you on that as well. The YAML aspect is somewhat of a 'low-level' concern that you shouldn't have to worry about unless you need something highly customized. Now, let me reverse the question—what would make you keep the tab open?"
- SOLAR_FIELDS 3y ago“Need[ing] something highly customized” is not some uncommon occurrence for your end users. It’s an inevitability for a large portion of them. Give me some well supported libraries in common general purpose languages to do this, codegen is pretty good these days and supporting 3 or 4 languages shouldn’t be an insurmountable achievement.
- shahargl 3y agogot you, so you would imagine some typescript/python/golang sdk that let you define workflows?
- 3y ago
- Jnr 3y agoDid you perhaps have an alternative solution you don't mind sharing? In my opinion yaml is good enough for gitops. Easy to read, understand, modify.
- SOLAR_FIELDS 3y agoI’ve been using Dagger for this replacement specifically. But that is ci/cd specific to caching and workflow execution. For things like workflow automation and orchestration I would reach for something like Prefect or Dagster. The point is to be able to do something in an actual programming language so that you not only get typing, readability, reusability, unit testability, local execution and language specific tooling, but also that it doesn’t suck for end users to write, debug and maintain. This also gives end users an escape hatch when your abstraction is inevitably not going to be good enough for them Cuelang etc like siblings mentioned are decent enough but the real scalable solutions here are made available in general purpose programming languages.
- shahargl 3y agothe "workflows as code" is something we are thinking about too. I guess there are pros and cons for every approach and eventually we would want to support both (but need to start with something)
- fredguth 3y agoCheck temporal.io, they are used by Uber, Netflix, Datadog...
- ransom1538 3y agotemporal is for managing events. How is it related to ci/cd? Generally curious.
- SOLAR_FIELDS 3y agoIt’s more generally a workflow manager and orchestration tool. It is kind of a more general version of the tools I mentioned, Dagster and Prefect. It can be used to spawn and manage CICD tasks asynchronously. Though I will say that Temporal's use case is probably not really well mapped to CI/CD - though it could be used for it (which is why I didn't mention it). It's primary strength is robust, long lived workflows with intelligent retries and the like - you typically want your CI/CD to be as fast as possible and while you want retries and resilience etc it's not as important as some other things (like being hermetic, reproducible, and cached).