8 ms·

The best config system I've ever seen used plain old Python to generate static configs. Everyone knows Python. Python is easy to do data munging in, as demonstr

The best config system I've ever seen used plain old Python to generate static configs. Everyone knows Python. Python is easy to do data munging in, as demonstrated by it's popularity as the #1 data science tool. There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. Even Starlark is just a dialect of Python. You can spend decades building a complicated configuration language, use a bespoke functional language as Mill does, but if you're a single company that can enforce code quality and just wants to get the job done, I feel like everything else is just unnecessary and over-engineered to scratch some academic itch for a "better system" that enforces "purity" at the cost of velocity. I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.

MathMonkeyMan 2y ago
This does work well. A team I was on at a past job did exactly this. On Unix the service literally ran `std::system("python config.py >config.json")` on startup. The problem with this is that the answer to the question "what kind of configuration can I expect?" is "simulate the script and find out." If the script is written well, and is short, then the parameters that are filled in by the runtime environment are apparent. Over time, though, there is a risk that the script will not remain written well, and it almost certainly won't remain short.
- siriusfeynman 2y ago
  An approach I use is splitting my config tools into 2 stages Stage 1 creates a "explicit" config that can be exported to plaintext that contains exactly what is going to be created/modified with no abstraction/simplification Stage 2 applies the "explicit" config You get to be as clever as you want in stage 1 to avoid excessive copy pasting or not being able to know what your tool is going to do because all you have to go on is some homegrown DSL
- iforgot22 2y ago
  You run into the same problem with config DSLs, except now you're dealing with a DSL. Config is almost never going to be static.
  - MathMonkeyMan 2y ago
    True. One advantage I can imagine for a DSL is that it constrains what is possible and optimizes (syntactically) what it's supposed to be for. I think that the author of Nix justified its language that way. The counterargument is "eventually you'll need every facility provided by a programming language, so just start with a programming language." I'm not sure how I feel about it. The YAML templating situation in Kubernetes is a [shit show][1]. Then again, I did once cave into the temptation of writing a [lisp-like XML preprocessor][2] to make my configurations less verbose. It doesn't have any access to the environment, though, so it's not a general purpose configuration language, just a shorthand for static XML. [1]: https://www.davidgoffredo.com/no-string-templates https://www.davidgoffredo.com/no-string-templates [2]: https://github.com/dgoffredo/llama https://github.com/dgoffredo/llama
    - iforgot22 2y ago
      What constraints are needed? I've used DSLs that are almost Python but not quite, I think because they were hermetic and deterministic. Even those ended up being produced dynamically using some higher level config DSL or just regular code. Like once you're doing RPCs, it's general programming language territory (though there are also DSLs that do this, which is cursed). And yes, I have very bad memories of Kubernetes YAML, also YAML itself.
threeseed 2y ago
> Everyone knows Python No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc. But by having a programming language as your build tool you now make it harder for new people to onboard. As in order to build project they often need to some unique, specific to the language syntax. And in order to find this syntax they look around on Github and because it’s a programming language every project has their own unique, specific to the project approach. Versus something like Cargo.toml where it’s simple and consistent regardless of which project you look at.
- wocram 2y ago
  I think it's hard to argue that Cargo.toml is any simpler than Python. Json might be ubiquitous enough for anyone to read and understand, but if Python is foreign than toml is no better.
- emidln 2y ago
  > No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc. Sure somebody might not have Python experience, but it's pretty easy to just not hire someone who says they don't know Python and isn't willing to learn for the role. I don't know that you'd filter out many candidates out of any random 100 devs.
  - threeseed 2y ago
    I am talking about graduates and others new to programming. Of course they are willing to learn for the role but making it hard for them in the beginning can forever turn them off a language. That has been a big problem with Scala and Spark.
    - morkalork 2y ago
      I've never seen a language used for ancillary purposes be the make or break on hiring for a role, it's always just been expected that you'd pick it up as you go. And IMO, python is the least offensive compared to stuff like Perl, Ruby (for Chef) or whatever the heck Terraform is.
eptcyka 2y ago
Ah yes, the age old belief that all software is complex enough that one must first run some other bespoke turing complete program to build every single piece of software. And of all the languages to pick for this, python, with it's non-hermetic execution environment is bound to bite you in the ass, once your buildscripts start depending on libraries. Oh, you could use poetry to solve the library issue with python, or maybe it'll be setuptools, pip or whatever is the flavour of the month in python packaging. After fighting with Nix for a sufficiently long time, I think most language specific build tools are not neccesarily the best solution to the problem of automating a build for bit of software written in language X. Complex projects will eventually evolve to depend on multiple languages (unless you're the Linux kernel), at which point the specialized language build tools turn into cumbersome barriers in the build process, where different build tools are not aware of the caching, conventions and configurations of any other tool. As such, in an ideal world, any new language would come with a compiler or bundler that can be supported well by higher level build/packaging tools. And bespoke python scripts ain't that.
aidenn0 2y ago
I don't know about Python specifically, but using a language I'm familiar with to generate ninja files (+ any header/environment/&c) for the build has become my go-to way of doing builds in the past 18 months or so.
IshKebab 2y ago
Python is a terrible choice for that sort of thing. Who really wants to have to set up a venv and deal with pip nonsense just to write a config file? Hell even installing Python is sometimes difficult.
- MathMonkeyMan 2y ago
  Stick to the standard library as of an oldish version of python (3.6?) and it's pretty much zero-install zero-config.
  - threeseed 2y ago
    On a Mac, Python has always been a challenge. Up until recently Apple only included Python2 and so developers used Homebrew to install Python3. Now it’s very common to find two versions of Python3 installed on a Mac developer’s laptop that conflicts with each other. You really want to be using virtualenv.
    - tom_ 2y ago
      Hence the advice to stick to the standard library - because then it doesn't matter all that much. I'm not sure what the full set of environments I've tested my Python 3 scripts actually is, but they've run OK on all the various cloud CI systems I've tried, plus my laptops, VMs, and work PCs.
    - iforgot22 2y ago
      All a Mac user has needed to do was install from https://www.python.org/downloads/ https://www.python.org/downloads/ and then run python3 in the shell. Even if you use MacPorts or brew or conda or whatever, there's a distinct command to run Python 3 instead of 2. I get that Python's package manager situation is terrible, but like the other user said, you only need built-in packages to spit out a config json or whatever.
      - threeseed 2y ago
        So you would then you end up with three Python3 installations. And if you install from the website it doesn’t override the path. So will still be using the Apple or Homebrew one.
lmm 2y ago
Scala is hardly some obscure bespoke language. It's a top-20, maybe top-10 programming language, that's been around for 20+ years (and had far fewer breaking changes than Python over that time). Most Python translates directly into Scala, but with the benefit of a proper sound type system and full IDE support. And it's a great language for data munging.
- bdangubic 2y ago
  this “top 10” your LLM hallucinating? :)
- makeitdouble 2y ago
  The claim sounded outlandish, but Scala looks indeed to be around the top 10~20 languages in hiring for instance: https://www.devjobsscanner.com/blog/top-8-most-demanded-programming-languages/ https://www.devjobsscanner.com/blog/top-8-most-demanded-prog... Scala is only in 0.5% of the scanned job offerings, and is far far behind the major languages in numbers, but I was surprised there's more demand than Rust or even Perl to be honest.
  - iforgot22 2y ago
    I'm not surprised it's above Rust and Perl, but it's below Dart?! Ouch.
    - camdenreslink 2y ago
      Might as well just list it as Flutter (Dart).
hcarvalhoalves 2y ago
> I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one. Nothing against Python, but of all the reasons to choose a technology, whatever is more represented on the dataset of some LLM is the worst reason. This is a death spiral. There's no hope for the future of this industry if newcomers are thinking like this.
- iforgot22 2y ago
  I cared about programming languages when I was a newcomer. Stopped caring about 10 years ago. They're just tools, each with their own gotchas and different design choices I couldn't care less about. Between two tools that both work ok, I will definitely pick whichever one my team and I can learn the easiest, and that includes LLM coverage.
dkkergoog 2y ago
[dead]
asalahli 2y ago
> There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. What are some examples of a library that can limit or prevent side effects of a piece of python code? I could use one right now.

MathMonkeyMan 2y ago

This does work well. A team I was on at a past job did exactly this. On Unix the service literally ran `std::system("python config.py >config.json")` on startup. The problem with this is that the answer to the question "what kind of configuration can I expect?" is "simulate the script and find out." If the script is written well, and is short, then the parameters that are filled in by the runtime environment are apparent. Over time, though, there is a risk that the script will not remain written well, and it almost certainly won't remain short.

siriusfeynman 2y ago
An approach I use is splitting my config tools into 2 stages Stage 1 creates a "explicit" config that can be exported to plaintext that contains exactly what is going to be created/modified with no abstraction/simplification Stage 2 applies the "explicit" config You get to be as clever as you want in stage 1 to avoid excessive copy pasting or not being able to know what your tool is going to do because all you have to go on is some homegrown DSL
iforgot22 2y ago
You run into the same problem with config DSLs, except now you're dealing with a DSL. Config is almost never going to be static.
- MathMonkeyMan 2y ago
  True. One advantage I can imagine for a DSL is that it constrains what is possible and optimizes (syntactically) what it's supposed to be for. I think that the author of Nix justified its language that way. The counterargument is "eventually you'll need every facility provided by a programming language, so just start with a programming language." I'm not sure how I feel about it. The YAML templating situation in Kubernetes is a [shit show][1]. Then again, I did once cave into the temptation of writing a [lisp-like XML preprocessor][2] to make my configurations less verbose. It doesn't have any access to the environment, though, so it's not a general purpose configuration language, just a shorthand for static XML. [1]: https://www.davidgoffredo.com/no-string-templates https://www.davidgoffredo.com/no-string-templates [2]: https://github.com/dgoffredo/llama https://github.com/dgoffredo/llama
  - iforgot22 2y ago
    What constraints are needed? I've used DSLs that are almost Python but not quite, I think because they were hermetic and deterministic. Even those ended up being produced dynamically using some higher level config DSL or just regular code. Like once you're doing RPCs, it's general programming language territory (though there are also DSLs that do this, which is cursed). And yes, I have very bad memories of Kubernetes YAML, also YAML itself.

threeseed 2y ago

> Everyone knows Python No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc. But by having a programming language as your build tool you now make it harder for new people to onboard. As in order to build project they often need to some unique, specific to the language syntax. And in order to find this syntax they look around on Github and because it’s a programming language every project has their own unique, specific to the project approach. Versus something like Cargo.toml where it’s simple and consistent regardless of which project you look at.

wocram 2y ago
I think it's hard to argue that Cargo.toml is any simpler than Python. Json might be ubiquitous enough for anyone to read and understand, but if Python is foreign than toml is no better.
emidln 2y ago
> No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc. Sure somebody might not have Python experience, but it's pretty easy to just not hire someone who says they don't know Python and isn't willing to learn for the role. I don't know that you'd filter out many candidates out of any random 100 devs.
- threeseed 2y ago
  I am talking about graduates and others new to programming. Of course they are willing to learn for the role but making it hard for them in the beginning can forever turn them off a language. That has been a big problem with Scala and Spark.
  - morkalork 2y ago
    I've never seen a language used for ancillary purposes be the make or break on hiring for a role, it's always just been expected that you'd pick it up as you go. And IMO, python is the least offensive compared to stuff like Perl, Ruby (for Chef) or whatever the heck Terraform is.

eptcyka 2y ago

Ah yes, the age old belief that all software is complex enough that one must first run some other bespoke turing complete program to build every single piece of software. And of all the languages to pick for this, python, with it's non-hermetic execution environment is bound to bite you in the ass, once your buildscripts start depending on libraries. Oh, you could use poetry to solve the library issue with python, or maybe it'll be setuptools, pip or whatever is the flavour of the month in python packaging. After fighting with Nix for a sufficiently long time, I think most language specific build tools are not neccesarily the best solution to the problem of automating a build for bit of software written in language X. Complex projects will eventually evolve to depend on multiple languages (unless you're the Linux kernel), at which point the specialized language build tools turn into cumbersome barriers in the build process, where different build tools are not aware of the caching, conventions and configurations of any other tool. As such, in an ideal world, any new language would come with a compiler or bundler that can be supported well by higher level build/packaging tools. And bespoke python scripts ain't that.

aidenn0 2y ago

I don't know about Python specifically, but using a language I'm familiar with to generate ninja files (+ any header/environment/&c) for the build has become my go-to way of doing builds in the past 18 months or so.

IshKebab 2y ago

Python is a terrible choice for that sort of thing. Who really wants to have to set up a venv and deal with pip nonsense just to write a config file? Hell even installing Python is sometimes difficult.

MathMonkeyMan 2y ago
Stick to the standard library as of an oldish version of python (3.6?) and it's pretty much zero-install zero-config.
- threeseed 2y ago
  On a Mac, Python has always been a challenge. Up until recently Apple only included Python2 and so developers used Homebrew to install Python3. Now it’s very common to find two versions of Python3 installed on a Mac developer’s laptop that conflicts with each other. You really want to be using virtualenv.
  - tom_ 2y ago
    Hence the advice to stick to the standard library - because then it doesn't matter all that much. I'm not sure what the full set of environments I've tested my Python 3 scripts actually is, but they've run OK on all the various cloud CI systems I've tried, plus my laptops, VMs, and work PCs.
  - iforgot22 2y ago
    All a Mac user has needed to do was install from https://www.python.org/downloads/ https://www.python.org/downloads/ and then run python3 in the shell. Even if you use MacPorts or brew or conda or whatever, there's a distinct command to run Python 3 instead of 2. I get that Python's package manager situation is terrible, but like the other user said, you only need built-in packages to spit out a config json or whatever.
    - threeseed 2y ago
      So you would then you end up with three Python3 installations. And if you install from the website it doesn’t override the path. So will still be using the Apple or Homebrew one.

lmm 2y ago

Scala is hardly some obscure bespoke language. It's a top-20, maybe top-10 programming language, that's been around for 20+ years (and had far fewer breaking changes than Python over that time). Most Python translates directly into Scala, but with the benefit of a proper sound type system and full IDE support. And it's a great language for data munging.

bdangubic 2y ago
this “top 10” your LLM hallucinating? :)
makeitdouble 2y ago
The claim sounded outlandish, but Scala looks indeed to be around the top 10~20 languages in hiring for instance: https://www.devjobsscanner.com/blog/top-8-most-demanded-programming-languages/ https://www.devjobsscanner.com/blog/top-8-most-demanded-prog... Scala is only in 0.5% of the scanned job offerings, and is far far behind the major languages in numbers, but I was surprised there's more demand than Rust or even Perl to be honest.
- iforgot22 2y ago
  I'm not surprised it's above Rust and Perl, but it's below Dart?! Ouch.
  - camdenreslink 2y ago
    Might as well just list it as Flutter (Dart).

hcarvalhoalves 2y ago

> I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one. Nothing against Python, but of all the reasons to choose a technology, whatever is more represented on the dataset of some LLM is the worst reason. This is a death spiral. There's no hope for the future of this industry if newcomers are thinking like this.

iforgot22 2y ago
I cared about programming languages when I was a newcomer. Stopped caring about 10 years ago. They're just tools, each with their own gotchas and different design choices I couldn't care less about. Between two tools that both work ok, I will definitely pick whichever one my team and I can learn the easiest, and that includes LLM coverage.

dkkergoog 2y ago

[dead]

asalahli 2y ago

> There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. What are some examples of a library that can limit or prevent side effects of a piece of python code? I could use one right now.