6 ms·

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/

by zmmmmm 6y ago

Another step in the endless cycle of configuration vs code. It's not an accident that we are in a deep cycle of constrained configuration languages (yaml/json/etc etc). People chose to go there because before that we had a cycle of using programming languages and people hated it, for all sorts of good reasons. Now I see we are on the way back into adding wrappers around the static config files, to turn them back into programming languages. This is happening all over the place - because guess what, it turns out, people hate static config files too, for all sorts of good reasons. I am not sure if we will ever reach a compromise here or if we are just going to have to put up with endless change and churn because nobody is ever happy or at least willing to just settle for things that are "ok" but not "perfect".

mixologic 6y ago
I think the main issue with using programming languages to run your CI/CD is that the whole point of CI/CD is to test and verify that your code works. Now you have code that may or may not be buggy, running your CI/CD so you'll have to set up CI/CD to test your test running code, ad nauseum. People might hate constrained configuration, but the idea is that you dont have to prove that what you're configuring is actually whats happening.
- ViVr 6y ago
  > Now you have code that may or may not be buggy Doesn't your solution of replacing the possibly buggy code with configuration leave you with configuration that may or may not be buggy?
  - zmmmmm 6y ago
    Its mainly a question of it being deterministic and reproducible I think. The presumption is that you then go and test a bunch of stuff based on whatever was built, so no, it will not be buggy. But if the code that actually goes to production is actually different to what is tested (because your rebuilt it based on a config with some runtime behavior that executed differently), you are compromising the whole thing and all bets are off. People will try to split the difference and generate a static config with code and test that as a reproducible entity, but then you have another set of tradeoffs (not least, complexity).
musingsole 6y ago
The line between configuration and code is fuzzy in mental space, so it makes sense we can't decide how to resolve it in digital space. I believe the answer is "it depends on how you're using it" and generally speaking, customer needs override any pre-existing consensus on how it's being used.
kortex 6y ago
The big difference is that config files are, at some point after interpolation and templating, static data structures composed of primitive data types. Further, these configs usually act as declarative languages, giving directives to pipelines, controllers, etc., typically with idempotent side-effects. This is just good software engineering. You have some static data structure which describes some result you want to exist in the world, and your pipeline/program makes that outcome manifest. This is the essence of boundary programming/hexagonal architecture. https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries
- pas 6y ago
  Exactly! And this is perfectly achievable with high-level expressive languages too. The important part is that the interface has to have the same semantics as a config file: atomic/transactional. For example the builder pattern does this. You can enjoy all the expressiveness and power of the high-level language, but at some point you create the final config/pipeline/job-graph, call whatever needs to be called, and the provider of the interface needs to validate that. That's why the semantics of Terraform is better than Ansible's.
  - kortex 6y ago
    Yup. My main gripe with Ansible is exactly this. Too much mutable state, too imperative, too much reliance on "gathering facts". There's no singular source of truth. It tries to be declarative, but every Ansible deployment I've maintained (granted, not many) always accretes corner cases and config drift. Maybe that's just the nature of the domain - mutating live systems will always be more imperative than (re)deploying the whole thing from scratch. I'd much rather use python/jinja/gomplate/what-have-you, do all my fancy expressive stuff with code I can check in, and get a single snapshot of plain, structured text (which I can also check in). Not to mention locking down your invariants (I can't recall how many times I've gotten bit by interpolating docker image uris with the wrong TAG). It's actually pushed me towards first Make and then later python cli tools (cause Makefile syntax is...ugh) to generate configs, rather than relying on any runtime business logic, cause it's just so much saner, traceable, auditable, etc.
encryptluks2 6y ago
Template languages are meant to change a lot less than programming languages and adapt when they do. I see companies using programming languages like this and then in a year the libraries are no longer maintained, they have some churn and their new employees then want to rewrite everything. I rarely think it is worth writing CI/CD in a programming language unless you are committed to supporting it for its lifetime. Most companies will not be able to and they'll end up back at the standard templates sooner or later.
VWWHFSfQ 6y ago
> nobody is ever happy or at least willing to just settle for things that are "ok" but not "perfect" I hope we never settle for things that are "ok" but not "perfect".
izgzhen 6y ago
One important thing about typical config language is that they are much simpler to do static analysis upon than full fledged PL. Static analysis, even just simple parsing, is a quite common need when you need to migrate or take statistics of something related to all configurations in a given code base.
GongOfFour 6y ago
In my opinion generating configuration from Python (or any language with a robust standard library) is still a win if it means less Ops and DevOps engineers are stringing together convoluted bash scripts that torture sed/awk/jq/yq to obscene lengths.
- TeeMassive 6y ago
  While it is true that Python scripts are far more superior to cryptic bash scripts, those scripts shouldn't be on the CI's side. Nothing wrong having build.py and test.py and the others, but there are no good reasons for them to be executable only by the CI. Debugging CI scripts is just asking for non-stop pager duty.
  - GongOfFour 6y ago
    Most all configuration specifications have a limit unless they are turing complete. Complex systems evolve beyond static configurations easily but the question is can you leverage configuration generation to minimize logic inside your CI? Orchestration ought not be the province of your build scripts, that should be the role of CI. If the configuration isn't capable of variadic functionality but is capable of orchestration... what do you do?
    - TeeMassive 6y ago
      What is stopping you from making a project dedicated to orchestration/configuration/deployment and using test containers to make those tests? How do you debug those "complex systems" when they fail? I've made a project in one of my old jobs dedicated to configuration and deployment tools used by CI jobs. At first it got some criticisms about "testing the CI that tests the software" and "testing the tests" but when the number of fires we have to take care of was cut by half it quickly became the norm.
- thenoblesunfish 6y ago
  You can call your own Python script from the pipeline, instead of using sed/awk/etc., right?
strogonoff 6y ago
The middle ground between flexibility of a programming language and constraints of a static config syntax appears to be right about where Dhall’s niche is.
TeeMassive 6y ago
You make a good point. As a developer who also dabbed in devops, I hate CI-side scripting with a passion. The last place I've worked a few years back had about 20 interdependent Jenkins jobs per project. During a build, a job would break another job who would break another job but not successively fast enough to intervene so the majority of the time of the devops guru was spent on making hacks and fixes everywhere, thus fueling the endless cycle of madness. Middle management were aware of the problem but they never could say no to new aggressive timelines so here we were but I digress. For me a CI server is a way to automate tasks so that 1) devs' time is saved 2) all versions are release worthy 3) human mistakes are kept to a minimum 4) a public record is kept 5) long and boring tasks such as packaging releases, generating code coverage and running the static code analyzer do not bother devs and are done periodically. What CI should not do: 1) do things that devs can't do 2) crash 3) run nondeterministic builds and tests 4) be understandable by only one guy named Brent (wink wink Phoenix Project). A typical CI script should limit itself to a simple workflow: checkout revision, get dependencies, build, test, release; and those should all be things that can be done, tested and modified and debugged on a common developing environment. Anything else is just loaning time from the future.
infogulch 6y ago
I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/ https://dhall-lang.org/
- AlphaSite 6y ago
  YTT is pretty nice too and basically glues star lark and yaml together.
  - sofixa 6y ago
    via comments! YAML is an abomination in itself ( spaces for logic?!), the last thing it needed was comments for extra logic.
    - xdennis 6y ago
      Proper spacing makes code readable. Forcing it is a good thing.
      - guipsp 6y ago
        It's a good thing, until you ask a non-technical user to edit the config file. Then they become confused because a tab is visually, but not functionally equivalent to a number of spaces.
        erinnh 6y ago
        I wouldnt ask a non-technical person to write any of these config files. If I want non-technical people to configure something I need to write a GUI for self-serving.
      - sofixa 6y ago
        Indeed, it does, when done via something like go fmt. But in the case of YAML, a pedantic level of "proper spacing" is used for logic. 2 or 3 spaces shouldn't change the meaning and order of things
hutrdvnj 6y ago
I quite like how helm charts solves the problem. You have a values.yaml which is plain yaml, but at the same time you have the helm template language, which allows you to create more complex configuration templates with functions and flow control.
mxcrossb 6y ago
I get your instinct, but there is one thing missing. I’m theory, my CI isn’t really being run in a known environment. I can’t write a shell script to power the ci, because how am I to know the CI launcher even has a shell? Of course, once an environment has been deployed, I am safe to switch to write shell scripts, calling system commands, etc. But it does make some sense to be completely static at the start and I haven’t be even picked an operating system.
rendaw 6y ago
> before that we had a cycle of using programming languages When was that?

by zmmmmm 6y ago

mixologic 6y ago

I think the main issue with using programming languages to run your CI/CD is that the whole point of CI/CD is to test and verify that your code works. Now you have code that may or may not be buggy, running your CI/CD so you'll have to set up CI/CD to test your test running code, ad nauseum. People might hate constrained configuration, but the idea is that you dont have to prove that what you're configuring is actually whats happening.

ViVr 6y ago
> Now you have code that may or may not be buggy Doesn't your solution of replacing the possibly buggy code with configuration leave you with configuration that may or may not be buggy?
- zmmmmm 6y ago
  Its mainly a question of it being deterministic and reproducible I think. The presumption is that you then go and test a bunch of stuff based on whatever was built, so no, it will not be buggy. But if the code that actually goes to production is actually different to what is tested (because your rebuilt it based on a config with some runtime behavior that executed differently), you are compromising the whole thing and all bets are off. People will try to split the difference and generate a static config with code and test that as a reproducible entity, but then you have another set of tradeoffs (not least, complexity).

musingsole 6y ago

The line between configuration and code is fuzzy in mental space, so it makes sense we can't decide how to resolve it in digital space. I believe the answer is "it depends on how you're using it" and generally speaking, customer needs override any pre-existing consensus on how it's being used.

kortex 6y ago

The big difference is that config files are, at some point after interpolation and templating, static data structures composed of primitive data types. Further, these configs usually act as declarative languages, giving directives to pipelines, controllers, etc., typically with idempotent side-effects. This is just good software engineering. You have some static data structure which describes some result you want to exist in the world, and your pipeline/program makes that outcome manifest. This is the essence of boundary programming/hexagonal architecture. https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries

pas 6y ago
Exactly! And this is perfectly achievable with high-level expressive languages too. The important part is that the interface has to have the same semantics as a config file: atomic/transactional. For example the builder pattern does this. You can enjoy all the expressiveness and power of the high-level language, but at some point you create the final config/pipeline/job-graph, call whatever needs to be called, and the provider of the interface needs to validate that. That's why the semantics of Terraform is better than Ansible's.
- kortex 6y ago
  Yup. My main gripe with Ansible is exactly this. Too much mutable state, too imperative, too much reliance on "gathering facts". There's no singular source of truth. It tries to be declarative, but every Ansible deployment I've maintained (granted, not many) always accretes corner cases and config drift. Maybe that's just the nature of the domain - mutating live systems will always be more imperative than (re)deploying the whole thing from scratch. I'd much rather use python/jinja/gomplate/what-have-you, do all my fancy expressive stuff with code I can check in, and get a single snapshot of plain, structured text (which I can also check in). Not to mention locking down your invariants (I can't recall how many times I've gotten bit by interpolating docker image uris with the wrong TAG). It's actually pushed me towards first Make and then later python cli tools (cause Makefile syntax is...ugh) to generate configs, rather than relying on any runtime business logic, cause it's just so much saner, traceable, auditable, etc.

encryptluks2 6y ago

Template languages are meant to change a lot less than programming languages and adapt when they do. I see companies using programming languages like this and then in a year the libraries are no longer maintained, they have some churn and their new employees then want to rewrite everything. I rarely think it is worth writing CI/CD in a programming language unless you are committed to supporting it for its lifetime. Most companies will not be able to and they'll end up back at the standard templates sooner or later.

VWWHFSfQ 6y ago

> nobody is ever happy or at least willing to just settle for things that are "ok" but not "perfect" I hope we never settle for things that are "ok" but not "perfect".

izgzhen 6y ago

One important thing about typical config language is that they are much simpler to do static analysis upon than full fledged PL. Static analysis, even just simple parsing, is a quite common need when you need to migrate or take statistics of something related to all configurations in a given code base.

GongOfFour 6y ago

In my opinion generating configuration from Python (or any language with a robust standard library) is still a win if it means less Ops and DevOps engineers are stringing together convoluted bash scripts that torture sed/awk/jq/yq to obscene lengths.

TeeMassive 6y ago
While it is true that Python scripts are far more superior to cryptic bash scripts, those scripts shouldn't be on the CI's side. Nothing wrong having build.py and test.py and the others, but there are no good reasons for them to be executable only by the CI. Debugging CI scripts is just asking for non-stop pager duty.
- GongOfFour 6y ago
  Most all configuration specifications have a limit unless they are turing complete. Complex systems evolve beyond static configurations easily but the question is can you leverage configuration generation to minimize logic inside your CI? Orchestration ought not be the province of your build scripts, that should be the role of CI. If the configuration isn't capable of variadic functionality but is capable of orchestration... what do you do?
  - TeeMassive 6y ago
    What is stopping you from making a project dedicated to orchestration/configuration/deployment and using test containers to make those tests? How do you debug those "complex systems" when they fail? I've made a project in one of my old jobs dedicated to configuration and deployment tools used by CI jobs. At first it got some criticisms about "testing the CI that tests the software" and "testing the tests" but when the number of fires we have to take care of was cut by half it quickly became the norm.
thenoblesunfish 6y ago
You can call your own Python script from the pipeline, instead of using sed/awk/etc., right?

strogonoff 6y ago

The middle ground between flexibility of a programming language and constraints of a static config syntax appears to be right about where Dhall’s niche is.

TeeMassive 6y ago

You make a good point. As a developer who also dabbed in devops, I hate CI-side scripting with a passion. The last place I've worked a few years back had about 20 interdependent Jenkins jobs per project. During a build, a job would break another job who would break another job but not successively fast enough to intervene so the majority of the time of the devops guru was spent on making hacks and fixes everywhere, thus fueling the endless cycle of madness. Middle management were aware of the problem but they never could say no to new aggressive timelines so here we were but I digress. For me a CI server is a way to automate tasks so that 1) devs' time is saved 2) all versions are release worthy 3) human mistakes are kept to a minimum 4) a public record is kept 5) long and boring tasks such as packaging releases, generating code coverage and running the static code analyzer do not bother devs and are done periodically. What CI should not do: 1) do things that devs can't do 2) crash 3) run nondeterministic builds and tests 4) be understandable by only one guy named Brent (wink wink Phoenix Project). A typical CI script should limit itself to a simple workflow: checkout revision, get dependencies, build, test, release; and those should all be things that can be done, tested and modified and debugged on a common developing environment. Anything else is just loaning time from the future.

infogulch 6y ago

I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/ https://dhall-lang.org/

AlphaSite 6y ago
YTT is pretty nice too and basically glues star lark and yaml together.
- sofixa 6y ago
  via comments! YAML is an abomination in itself ( spaces for logic?!), the last thing it needed was comments for extra logic.
  - xdennis 6y ago
    Proper spacing makes code readable. Forcing it is a good thing.
    - guipsp 6y ago
      It's a good thing, until you ask a non-technical user to edit the config file. Then they become confused because a tab is visually, but not functionally equivalent to a number of spaces.
      - erinnh 6y ago
        I wouldnt ask a non-technical person to write any of these config files. If I want non-technical people to configure something I need to write a GUI for self-serving.
    - sofixa 6y ago
      Indeed, it does, when done via something like go fmt. But in the case of YAML, a pedantic level of "proper spacing" is used for logic. 2 or 3 spaces shouldn't change the meaning and order of things

hutrdvnj 6y ago

I quite like how helm charts solves the problem. You have a values.yaml which is plain yaml, but at the same time you have the helm template language, which allows you to create more complex configuration templates with functions and flow control.

mxcrossb 6y ago

I get your instinct, but there is one thing missing. I’m theory, my CI isn’t really being run in a known environment. I can’t write a shell script to power the ci, because how am I to know the CI launcher even has a shell? Of course, once an environment has been deployed, I am safe to switch to write shell scripts, calling system commands, etc. But it does make some sense to be completely static at the start and I haven’t be even picked an operating system.

rendaw 6y ago

> before that we had a cycle of using programming languages When was that?