16 ms·

Nickel: Better Configuration for Less

61 points by figomore 6y ago

dreamer777 6y ago
Still going to write yaml
- verdverm 6y ago
  It's spelled YamHell ;]
dashwav 6y ago
I have a very hard time getting behind these complex configuration languages. To me what makes a configuration format good is the simplicity of reading the configuration of a program, and almost all of these languages are optimizing for feature complexity over readability. I think that all of the popular config formats (yaml, json, toml, etc) have issues, but none of the major issues with them have to do with being unable to represent a fibonacci sequence in their language. To draw a direct comparison, when I look at the examples in the github repository, all I can think is "I would never want to have this be a source of truth in my codebase". While I get frustrated w/ whitespace in yaml and the difficulty of reading complex json configuration, if I need a way to programmatically load complex data I would almost always rather use those two as a base and write a 'config-loader' in the language that I am already using for my project (instead of introducing another syntax into the mix)
- liuliu 6y ago
  Starlark is OK (it is very similar to Python except removed non-deterministic part). But the part where no type-hinting kicks in is when you try to read the actual underlying macros / functions the others provided. It is much harder to do without types (which Nickel seems would like to address). Honestly, I would prefer anything that mimics popular languages to lower the bar of reading.
  - FridgeSeal 6y ago
    Have you tried Dhall? Static types and enough power to provide the tools you need, but deliberately not enough to allow full on arbitrary computation. I've played with it briefly, along with the Kubernetes plugin, and it was a nice experience.
- vinceguidry 6y ago
  Configuration isn't code isn't data. Data belongs in a database. Code belongs in a codebase. Configuration doesn't. Ideally config should be reduced down to keys and values and stored anywhere where it's easy to push to the environment where the code runs. I don't understand the immense expansion and proliferation of the config layer. I never touch code anymore. All I handle is config and tooling around it. YAML engineering. Better configuration doesn't mean more ways to treat config like code, or data like config, or god forbid, code. It means treating config like config and code like code. Gitops just makes me sad. Truth should only flow in one direction. The first time I had to write a script to utilize the GitHub API to auto-update a code repo I died a little inside.
  - pc86 6y ago
    So we try to keep these three areas separate, and config generally ends up in our deployment pipeline. The problem is what do you do when code changes necessitate config changes? Adding/removing config properties, etc. We don't want 50 developers messing around with production deployment pipelines. Doesn't something like this go a little way toward solving that problem?
    - vinceguidry 6y ago
      Manage the complexity, yeah. Solve it? Well, if you’re managing complexity, you’re not really solving it, are you? Heavy lifting needs to be done with code. If your config layer is growing, I would look for why that is and how you could push the complexity to the code or data and “boil down” the config until it can be represented with just keys and values. Growing config means there’s areas of your application that aren’t being properly encapsulated. But once something is enshrined as config then it usually never will get treated as a legitimate application concern, worthy of a data model and a UI for changing it. Devs just keep adding onto it and before you know it you need a whole team just to deal with it.
- cogman10 6y ago
  Yeah... I honestly don't see the appeal of nickel. It is sold as "You use this to generate configuration in other formats like JSON"... but why? Why would I want to use some language other than the target format to configure things? Why am I making my configuration a 2 step process? And even if I bought all of those reasons, why wouldn't I just use a general purpose language instead? Why have some esoteric language dialect whose only purpose is... making configuration files? I'd much rather use Bash, python, perl, javascript, typescript, groovy, Java, kotlin, C++, C, Rust, erlang, php, awk, pascal, go, Nim, Nix, VB, Hax, coffeescript etc. Really, take your pick. Any well established language seems like a much better approach than something like this.
  - q3k 6y ago
    Three things to address: 1) This doesn't have to be a two-step process. Specialized tools like kubecfg for Jsonnet will directly take a Jsonnet top-level config and instantiate it, traverse the tree, and apply the configuration intelligently to your Kubernetes Cluster. 2) General purpose languages are at a disadvantage, because most of them are impure. Languages that limit all filesystem imports to be local to a repository and disallow any I/O ensure that you can safely instantiate configuration on CI hosts, in production programs, etc. The fact that languages like Jsonnet also ship as a single binary (or simple library) that requires no environment setup, etc. also make them super easy to integrate to any stack. 3) Configuration languages tend to be functional, lazily evaluated and declarative, vastly simplifying building abstractions that feel more in-line with your data. This allows for progressive building of abstraction, from just a raw data representation, through removal of repeated fields, to anything you could imagine makes sense for your application. Related reading: https://landing.google.com/sre/workbook/chapters/configuration-specifics/ https://landing.google.com/sre/workbook/chapters/configurati...
    - throwaway894345 6y ago
      I don’t think they tend to be lazily evaluated (unless you mean “lazy” in some other way than I’m familiar with), but in general I agree.
- q3k 6y ago
  Conversely, I've been using quite a bit of Jsonnet in different projects for a few years now, and it's a life changer. Here's a public example - using Jsonnet to parametrize all core resources of a bare metal Kubernetes cluster: [1]. This in turn uses cluster.libsonnet [2], which sets up Calico, Metallb, Rook, Nginx-Ingress-Controller, Cert-Manager, CoreDNS, ... Note that this top-level file aims to be the _entire_ source of truth for that particular cluster. I know of people who are reusing some of the dependent lib/*libsonnet code in their own deployments, which shows that this is not just abstraction for the sake of abstraction. Jsonnet isn't perfect, but it allows for actual building of abstraction layers in configuration, guaranteed pure evaluation, and not a single line of text templated or repeated YAML. [1] - https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/k0.libsonnet#L17-L50 https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/k0.lib... [2] - https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/cluster.libsonnet https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/cluste...
  - AtlasBarfed 6y ago
    Does a "configuration language" specifically incorporate features for "overlaid" or "unified from parts" configuration? Much like layered dockerfiles, mature configuration often comes from several places: env vars, configuration appropriate for checkin to git (no secrets), secrets configuration, and of course the old environment-specific configuration. All of that merges to "The Configuration". Also, these seem close to templating languages. I've done this several times with a "stacked map" implementation (much like the JSP key lookups went through page / session / application scopes, or even more convoluted for Spring Webflow.
    - q3k 6y ago
      Answer for Jsonnet: layering/overrides from multiple sources: yes, as you define it (it's a programming language, the logic is yours to define, based on your particular usecase). But no access from environment variables, as that's inpure, and not really in scope for them.
- GordonS 6y ago
  I also have a hard time for the same reasons. I'm torn though; I want a config format that's super easy to read, and that I can easily change anywhere with nothing more than sed/vim/nano/notepad - but I also want to avoid typos and formatting problems. I'm not sure which is the lesser evil. Actually, I think (as always), that it depends. For something simple like a config file for an app, JSON/YAML is usually fine. But for something more complex, like IaC (Infrastructure as Code) definitions, I think perhaps "proper" programming languages might be more beneficial. I had a look at Pulumi just yesterday, and I very much like the idea of writing a simple C#/Typescript app to deploy my, when compared to something like HCL (HashiCorp Configuration Language) or bash scripts that wrap the Azure/AWS CLI tooling.
- throwaway894345 6y ago
  > I have a very hard time getting behind these complex configuration languages. To me what makes a configuration format good is the simplicity of reading the configuration of a program, and almost all of these languages are optimizing for feature complexity over readability. I think that all of the popular config formats (yaml, json, toml, etc) have issues, but none of the major issues with them have to do with being unable to represent a fibonacci sequence in their language. Static languages like JSON and YAML are fine for toy configurations, but they don't scale to the most basic real-world configuration tasks. Consider any reasonably sized Kubernetes project that someone wants to make available for others to install in their cluster's. The project probably has thousands of lines of complex configuration but much of it will change subtly from one installation to another. Rather than distributing a copy of the configs and detailed instructions on how to manually configure the configuration for each use case, it becomes very naturally expedient to parameterize the configuration. The most flat-footed solution involves text-based templates (a la jinja, mustache, etc) which is pretty much what Helm has done for a long time. But text-based templates are tremendously cumbersome (you have to make sure your templates always render syntactically correct and ideally also human readable, which is difficult because YAML is whitespace-sensitive and text templates aren't designed to make it easy to control whitespace). A similarly naive solution is to simply encode a programming language into the YAML. Certain YAML forms encode references (e.g., `{"Ref": "<identifier>"}` is equivalent to dereferencing a variable in source code). Another program evaluates this implicit language at runtime. This is the CloudFormation approach, and it also gives you some crude reuse while leaving much to be desired. After stumbling through a few of these silly permutations, it becomes evident that this reuse problem isn't different than the reuse problems that standard programming languages solve for; however, what is different is that we don't want our configuration to have access to system APIs including I/O and we may also want to prevent against non-halting programs (which is to say that we may not want our language to be turing complete). An expression-based configuration language becomes a natural fit. After using an expression-based configuration language, you realize that it's pretty difficult to make sure that your JSON/YAML output has the right "shape" such that it will be accepted by Kubernetes or CloudFormation or whatever your target is, so you realize the need for static type annotations and a type checker. Note that at no point are we trying to implement the fibonacci sequence, and in fact we prefer not to be able to implement it at all because we expressly prefer a language that is guaranteed to halt (though this isn't a requirement for all use cases, I believe it does satisfy the range of use-cases that we're discussing, and the principle of least power suggests that we should prefer it to turing-complete solutions).
- marcosdumay 6y ago
  The use case of those executable configuration languages is that you often need to set the same setting on different programs, maybe even on different machines, but they must all reflect the same decision in different ways (like your services server can set a port to listen, so your firewall must set that port as open for internal traffic, and your applications must set that port as their data source). That said, this one language does not look powerful enough for that. So I'm not sure where it can be used.
  - wtetzner 6y ago
    > That said, this one language does not look powerful enough for that. So I'm not sure where it can be used. I mean, it's used to configure all of NixOS, so I'm not sure if that's true.
    - marcosdumay 6y ago
      Oh, so I'm wrong. Makes more sense that way :) Let me add another postit of "try NixOS in an environment" into my TODO list...
sali0 6y ago
I have been in a deep rabbithole with Nix lately. It seems almost too good to be true. Declarative configuration, and with NixOps, declarative provisioning as well. Along with the other tools in the ecosystem, there is quite some overlap with some of the popular tools from Hashicorps and others. I am just beginning to learn the intricacies of DevOps, and Nix seems like a one-stop shop. What are the downsides of Nix?
- darthrupert 6y ago
  The downsides are complexity and dissimilarity to existing things, which lead to not many people knowing how to use it.
- aidenn0 6y ago
  I've been using it for 5 years at this point and love it. Biggest downside is that it's so different, which confuses e.g. upstream. Second biggest downside is that getting things right in a declarative manner is a bit more upfront work.
- throwaway894345 6y ago
  The Nix language is a big one, the lack of typing and comments which might help readers navigate Nix code, the arbitrary directory structure of nixpkgs, the very dynamic nature of derivations (if these keys are present on the derivation it will do one thing otherwise it will do very different things), the lack of documented escape hatches for things that you don’t care to make reproducible (in a perfect world, I would make my Python dependencies reproducible but the system package works well enough on my system and I can’t be bothered to write a recipe that builds some autotools-based C project with its own dependencies on obscure C projects scattered across the Internet, each with their own special snowflake build systems and their own dependency trees), etc. Nix is definitely the right direction (it could completely solve the monorepo/multirepo problem and dramatically simplify CI/CD and otherwise significantly improve software development), but it’s not very pragmatic. It badly needs some product leadership.
- X6S1x6Okd1st 6y ago
  Specific to NixOs, but running software that hasn't already been packaged for nix can be quite hard. The language is not simple and discovery can be quite lacking.
aidenn0 6y ago
TIL that there are people who like the nix language. I love NixOS, but hate the language.
- kalium-xyz 6y ago
  A lot of big names in the Nix community work for Tweag, including Eelco Dolstra who did the initial Nix/NixOS thesis.
- throwaway894345 6y ago
  I feel similarly. I will say that the lack of static typing compounds just about every other problem I have with Nix, so this seems like a marked improvement, but still it would be nice to have something that is more familiar and intuitive to programmers generally.
HelloNurse 6y ago
Documentation is terribly lacking, but in addition examples are perplexing. Where is the "Url" type, which should be a union of a constrained subset of "Str" (as built by mylib.makeURL) and a three component record (as used in the example)? Where can I specify that the "urls" list in the configuration must be present and nonempty, and that "host" and "port" are mandatory (or not)? And that the type of "urls" is a list of "Url", the type of "port" is "port", etc.? How can a general purpose port type have a meaningful default value, given that duplicates are generally fatal and a configuration can contain multiple ports? Only individual port usages (e.g. ftpPort and telnetPort in a commonNetworkServices record) should have defaults. Where is the body of numToStr? Maybe in some unmentioned standard library? Is there some kind of enforceable separation between the configuration file (untrusted and supplied by the user) and a schema it must satisfy (trusted and supplied by the application)? What forbids the configuration file to, say, redefine makeURL as Str -> Str -> Num -> Str -> Str -> Str (to add a HTTP path and query string) and make the application's URL parsing fail?
- yagoham 6y ago
  Hi, blog post author here. As stated at the beginning, the project is still WIP, and I didn't expect it to end on HN. Sorry for the lack of documentation and convincing example for now ! The post is intended to present the ideas and let people look at the repo, but while the core language is rather complete, a lot of small but important things are still missing to make it usable right now. I sidestepped this issue to still be able to illustrate some points with simple examples. To answer your questions: - numToStr would be either in a standard library, or implemented here using standard library functions. It does not exist currently - the fact that the urls are non empty could be either enforced by the contract itself (ie use a NonEmptyUrl contract). Or you could combine a NonEmptyStr and a Url contract via merging. By default, things with a contract without a default must be present. - I'm not sure I fully understand your last point, but I don't think a configuration file could "redefine" a function in any meaningful way. Say you write a program "main.ncl", which import some external downloaded source, check it against a contract you've written, and generate a config accordingly. Since the language is pure and variable are lexically scoped, nothing from an import could redefine anything in your current scope, be it your contracts or your functions.
throwaway894345 6y ago
There is definitely a need for this class of tools (statically typed expression-based config languages), but it’s super weird to praise Nix as very simple and easy to understand and then to gripe that Dhall is complex. I’ve been trying to learn Nix on and off for years and the language is often one of the things that trips me up. Dhall on the other hand looks simple even though it insists on a Haskell-esque syntax (which I find to be difficult to read). All I really want is Starlark with type annotations or something similar, but Dhall is pretty close.
galkk 6y ago
I don't want to interpret my configuration. I don't want it to act as a source code. Many places at Google use text representation of protobuffers as config files and I like it a lot. It's typed. It's readable. The schema and text are source controlled. You have code generators for many languages. (Work at Google, opinion is mine)
- q3k 6y ago
  Text protos are decent (although a lack of a public spec limits their adoption outside of Google). But that's just a serialization format, only a piece of the configuration puzzle. You may argue that all configuration should be fully static, but having gone from writing GCL/BCL at Google to having to deal with static, text-templated, copy-pasted YAMLs in the Real World - I want my GCL back. :)
  - sa46 6y ago
    Ironically, nickel was the name of the Google-internal, ill-fated replacement for GCL. I think it just lives on in blueprint files. I also miss GCL, as quirky as it was. YAML composes poorly.
  - verdverm 6y ago
    GCL also came up in the Cue slack channel today, and why Cue is intentionally different https://cuelang.org/community https://cuelang.org/community
  - ithkuil 6y ago
    If you want GCL back, here you have it: https://jsonnet.org/ https://jsonnet.org/ A bit cleaned up (withour "up" and without weird scoping rules)
saurabhnanda 6y ago
How different is this gradual typing from TypeScript?
LukeBMM 6y ago
Yet Another Yet Another Markup Language?
- jackric 6y ago
  It's got functions so not a markup, it's Yet Another Fucking Config Language
vlovich123 6y ago
I love using cap'n'proto to store configurations. It's really slick. + Type safe since you define the schema for every part of your configuration. + Ridiculously easy to read/understand (writing has a small ramp curve to build the muscle memory). + Type system is very flexible & rich (lists, maps, unions), supports generics (for built-in & custom types) & everything is neatly composable (including constants that reference constants). + The config compiles down to a minimal binary file. + You can have arbitrary configurations in 1 file (you compile a specific constant to a file so each configuration you want is just 1 constant value you define in the schema). + The binary file can be converted back to text using standard cap'n'proto tools so it's easy to double-check the config you're deploying. + Perfect backward/forward support for the configuration as long as you follow the rules (similar to protobuf/flatbuffers) since you have to define the schema for your constants. + Loading the config file from any data source (disk, network) is trivial and for on-disk usages you could mmap the struct to get even better performance (only the parts of the config you access would get paged in). + Supports a variety of languages. While the RPC stuff has a bit less adoption, the parts needed for configuration should be available in the most popular languages (I think the only missing language generator is Swift). The only negatives are: - You do have to define the schema for your config - There's a single upfront cost to integrate cap'n'proto into your build system if you're not using CMake. Neither feel like prohibitive negatives though. If you're looking at strong typing, why not go all the way & make sure that your entire config has a strongly typed structure to it? Not just that some field is an int also that there's certain specifically named fields & that accessing these fields in a strongly-typed languages will result in build errors if you forgot to change something.
olodus 6y ago
I see so many complain about these config languages. I think it is a bit black and white to think of a perfect separation between code and config. In a perfect world and easy situations yes. But I think you are all missing the point of these languages. In my eyes they are templating languages. Have any of you guys seen the Helm charts some of us have to write daily? Mustache coding is terrible! I would love any of these languages instead of that. The only thing worse would be to have to write all the yaml yourself every time instead of any templating at all. To that end, you could use any programming language to generate the yaml for your config. But then some of these has some nice features. I really like that Dhall is only a total lang and it does some nice things with importing dhall files. That is not to say you should unnecessarily complicate your config. Keep it easy and separate from code as much as possible.
aldanor 6y ago
Opened the link wondering what's a "better configuration for `less`"... oh well.
X6S1x6Okd1st 6y ago
> All in all, the Nix language is a lazy JSON with functions. That's news to me! I have been slowly learning the language, only enough to configure and package stuff as needed for my personal use, but it has not just been a lazy JSON with functions...
verdverm 6y ago
Timely, we are talking about TC in the Cue slack today. Nickel calls out its Cue inspiration while maintaining TC. https://github.com/tweag/nickel#related-projects-and-inspirations https://github.com/tweag/nickel#related-projects-and-inspira... https://cuelang.org/community https://cuelang.org/community
anentropic 6y ago
What is the selling point for this vs say Dhall, which looks superficially similar (Haskell-ey config lang)? The examples for Nickel currently look a lot more like programming than config. Why not just use a general purpose programming language that you already know and has many libraries and mature tooling? Dhall advertises a lack of Turing-completeness as a feature. One of the Nickel examples shows an implementation of fibonacci function... (I have no idea if this implies Nickel is TC, nor why I would need such power in my config) > Dhall features a powerful type system that is able to type a wide range of idioms. But it is complex, requiring some experience to become fluent in. > Gradual types also lets us keep the type system simple: even in statically typed code if you want to write a component that the type checker doesn’t know how to verify, you don’t have to type-check that part. > Complementary to the static type system, Nickel offers contracts. Contracts offer precise and accurate dynamic type error reporting, even in the presence of function types. It's not clear to me that gradual typing as a mix of statically typed code + untyped code with "contracts" is necessarily simpler than just one or the other. It's not clear to me that a specialised config language combining code + schemas is necessarily better than an existing general purpose language + some language-agnostic schema DSL. I don't know, maybe this is all great, I feel the benefits need to be more clearly explained and demonstrated though.

61 points by figomore 6y ago

dreamer777 6y ago

Still going to write yaml

verdverm 6y ago
It's spelled YamHell ;]

dashwav 6y ago

I have a very hard time getting behind these complex configuration languages. To me what makes a configuration format good is the simplicity of reading the configuration of a program, and almost all of these languages are optimizing for feature complexity over readability. I think that all of the popular config formats (yaml, json, toml, etc) have issues, but none of the major issues with them have to do with being unable to represent a fibonacci sequence in their language. To draw a direct comparison, when I look at the examples in the github repository, all I can think is "I would never want to have this be a source of truth in my codebase". While I get frustrated w/ whitespace in yaml and the difficulty of reading complex json configuration, if I need a way to programmatically load complex data I would almost always rather use those two as a base and write a 'config-loader' in the language that I am already using for my project (instead of introducing another syntax into the mix)

liuliu 6y ago
Starlark is OK (it is very similar to Python except removed non-deterministic part). But the part where no type-hinting kicks in is when you try to read the actual underlying macros / functions the others provided. It is much harder to do without types (which Nickel seems would like to address). Honestly, I would prefer anything that mimics popular languages to lower the bar of reading.
- FridgeSeal 6y ago
  Have you tried Dhall? Static types and enough power to provide the tools you need, but deliberately not enough to allow full on arbitrary computation. I've played with it briefly, along with the Kubernetes plugin, and it was a nice experience.
vinceguidry 6y ago
Configuration isn't code isn't data. Data belongs in a database. Code belongs in a codebase. Configuration doesn't. Ideally config should be reduced down to keys and values and stored anywhere where it's easy to push to the environment where the code runs. I don't understand the immense expansion and proliferation of the config layer. I never touch code anymore. All I handle is config and tooling around it. YAML engineering. Better configuration doesn't mean more ways to treat config like code, or data like config, or god forbid, code. It means treating config like config and code like code. Gitops just makes me sad. Truth should only flow in one direction. The first time I had to write a script to utilize the GitHub API to auto-update a code repo I died a little inside.
- pc86 6y ago
  So we try to keep these three areas separate, and config generally ends up in our deployment pipeline. The problem is what do you do when code changes necessitate config changes? Adding/removing config properties, etc. We don't want 50 developers messing around with production deployment pipelines. Doesn't something like this go a little way toward solving that problem?
  - vinceguidry 6y ago
    Manage the complexity, yeah. Solve it? Well, if you’re managing complexity, you’re not really solving it, are you? Heavy lifting needs to be done with code. If your config layer is growing, I would look for why that is and how you could push the complexity to the code or data and “boil down” the config until it can be represented with just keys and values. Growing config means there’s areas of your application that aren’t being properly encapsulated. But once something is enshrined as config then it usually never will get treated as a legitimate application concern, worthy of a data model and a UI for changing it. Devs just keep adding onto it and before you know it you need a whole team just to deal with it.
cogman10 6y ago
Yeah... I honestly don't see the appeal of nickel. It is sold as "You use this to generate configuration in other formats like JSON"... but why? Why would I want to use some language other than the target format to configure things? Why am I making my configuration a 2 step process? And even if I bought all of those reasons, why wouldn't I just use a general purpose language instead? Why have some esoteric language dialect whose only purpose is... making configuration files? I'd much rather use Bash, python, perl, javascript, typescript, groovy, Java, kotlin, C++, C, Rust, erlang, php, awk, pascal, go, Nim, Nix, VB, Hax, coffeescript etc. Really, take your pick. Any well established language seems like a much better approach than something like this.
- q3k 6y ago
  Three things to address: 1) This doesn't have to be a two-step process. Specialized tools like kubecfg for Jsonnet will directly take a Jsonnet top-level config and instantiate it, traverse the tree, and apply the configuration intelligently to your Kubernetes Cluster. 2) General purpose languages are at a disadvantage, because most of them are impure. Languages that limit all filesystem imports to be local to a repository and disallow any I/O ensure that you can safely instantiate configuration on CI hosts, in production programs, etc. The fact that languages like Jsonnet also ship as a single binary (or simple library) that requires no environment setup, etc. also make them super easy to integrate to any stack. 3) Configuration languages tend to be functional, lazily evaluated and declarative, vastly simplifying building abstractions that feel more in-line with your data. This allows for progressive building of abstraction, from just a raw data representation, through removal of repeated fields, to anything you could imagine makes sense for your application. Related reading: https://landing.google.com/sre/workbook/chapters/configuration-specifics/ https://landing.google.com/sre/workbook/chapters/configurati...
  - throwaway894345 6y ago
    I don’t think they tend to be lazily evaluated (unless you mean “lazy” in some other way than I’m familiar with), but in general I agree.
q3k 6y ago
Conversely, I've been using quite a bit of Jsonnet in different projects for a few years now, and it's a life changer. Here's a public example - using Jsonnet to parametrize all core resources of a bare metal Kubernetes cluster: [1]. This in turn uses cluster.libsonnet [2], which sets up Calico, Metallb, Rook, Nginx-Ingress-Controller, Cert-Manager, CoreDNS, ... Note that this top-level file aims to be the _entire_ source of truth for that particular cluster. I know of people who are reusing some of the dependent lib/*libsonnet code in their own deployments, which shows that this is not just abstraction for the sake of abstraction. Jsonnet isn't perfect, but it allows for actual building of abstraction layers in configuration, guaranteed pure evaluation, and not a single line of text templated or repeated YAML. [1] - https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/k0.libsonnet#L17-L50 https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/k0.lib... [2] - https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/cluster.libsonnet https://cs.hackerspace.pl/hscloud/-/blob/cluster/kube/cluste...
- AtlasBarfed 6y ago
  Does a "configuration language" specifically incorporate features for "overlaid" or "unified from parts" configuration? Much like layered dockerfiles, mature configuration often comes from several places: env vars, configuration appropriate for checkin to git (no secrets), secrets configuration, and of course the old environment-specific configuration. All of that merges to "The Configuration". Also, these seem close to templating languages. I've done this several times with a "stacked map" implementation (much like the JSP key lookups went through page / session / application scopes, or even more convoluted for Spring Webflow.
  - q3k 6y ago
    Answer for Jsonnet: layering/overrides from multiple sources: yes, as you define it (it's a programming language, the logic is yours to define, based on your particular usecase). But no access from environment variables, as that's inpure, and not really in scope for them.
GordonS 6y ago
I also have a hard time for the same reasons. I'm torn though; I want a config format that's super easy to read, and that I can easily change anywhere with nothing more than sed/vim/nano/notepad - but I also want to avoid typos and formatting problems. I'm not sure which is the lesser evil. Actually, I think (as always), that it depends. For something simple like a config file for an app, JSON/YAML is usually fine. But for something more complex, like IaC (Infrastructure as Code) definitions, I think perhaps "proper" programming languages might be more beneficial. I had a look at Pulumi just yesterday, and I very much like the idea of writing a simple C#/Typescript app to deploy my, when compared to something like HCL (HashiCorp Configuration Language) or bash scripts that wrap the Azure/AWS CLI tooling.
throwaway894345 6y ago
> I have a very hard time getting behind these complex configuration languages. To me what makes a configuration format good is the simplicity of reading the configuration of a program, and almost all of these languages are optimizing for feature complexity over readability. I think that all of the popular config formats (yaml, json, toml, etc) have issues, but none of the major issues with them have to do with being unable to represent a fibonacci sequence in their language. Static languages like JSON and YAML are fine for toy configurations, but they don't scale to the most basic real-world configuration tasks. Consider any reasonably sized Kubernetes project that someone wants to make available for others to install in their cluster's. The project probably has thousands of lines of complex configuration but much of it will change subtly from one installation to another. Rather than distributing a copy of the configs and detailed instructions on how to manually configure the configuration for each use case, it becomes very naturally expedient to parameterize the configuration. The most flat-footed solution involves text-based templates (a la jinja, mustache, etc) which is pretty much what Helm has done for a long time. But text-based templates are tremendously cumbersome (you have to make sure your templates always render syntactically correct and ideally also human readable, which is difficult because YAML is whitespace-sensitive and text templates aren't designed to make it easy to control whitespace). A similarly naive solution is to simply encode a programming language into the YAML. Certain YAML forms encode references (e.g., `{"Ref": "<identifier>"}` is equivalent to dereferencing a variable in source code). Another program evaluates this implicit language at runtime. This is the CloudFormation approach, and it also gives you some crude reuse while leaving much to be desired. After stumbling through a few of these silly permutations, it becomes evident that this reuse problem isn't different than the reuse problems that standard programming languages solve for; however, what is different is that we don't want our configuration to have access to system APIs including I/O and we may also want to prevent against non-halting programs (which is to say that we may not want our language to be turing complete). An expression-based configuration language becomes a natural fit. After using an expression-based configuration language, you realize that it's pretty difficult to make sure that your JSON/YAML output has the right "shape" such that it will be accepted by Kubernetes or CloudFormation or whatever your target is, so you realize the need for static type annotations and a type checker. Note that at no point are we trying to implement the fibonacci sequence, and in fact we prefer not to be able to implement it at all because we expressly prefer a language that is guaranteed to halt (though this isn't a requirement for all use cases, I believe it does satisfy the range of use-cases that we're discussing, and the principle of least power suggests that we should prefer it to turing-complete solutions).
marcosdumay 6y ago
The use case of those executable configuration languages is that you often need to set the same setting on different programs, maybe even on different machines, but they must all reflect the same decision in different ways (like your services server can set a port to listen, so your firewall must set that port as open for internal traffic, and your applications must set that port as their data source). That said, this one language does not look powerful enough for that. So I'm not sure where it can be used.
- wtetzner 6y ago
  > That said, this one language does not look powerful enough for that. So I'm not sure where it can be used. I mean, it's used to configure all of NixOS, so I'm not sure if that's true.
  - marcosdumay 6y ago
    Oh, so I'm wrong. Makes more sense that way :) Let me add another postit of "try NixOS in an environment" into my TODO list...

sali0 6y ago

I have been in a deep rabbithole with Nix lately. It seems almost too good to be true. Declarative configuration, and with NixOps, declarative provisioning as well. Along with the other tools in the ecosystem, there is quite some overlap with some of the popular tools from Hashicorps and others. I am just beginning to learn the intricacies of DevOps, and Nix seems like a one-stop shop. What are the downsides of Nix?

darthrupert 6y ago
The downsides are complexity and dissimilarity to existing things, which lead to not many people knowing how to use it.
aidenn0 6y ago
I've been using it for 5 years at this point and love it. Biggest downside is that it's so different, which confuses e.g. upstream. Second biggest downside is that getting things right in a declarative manner is a bit more upfront work.
throwaway894345 6y ago
The Nix language is a big one, the lack of typing and comments which might help readers navigate Nix code, the arbitrary directory structure of nixpkgs, the very dynamic nature of derivations (if these keys are present on the derivation it will do one thing otherwise it will do very different things), the lack of documented escape hatches for things that you don’t care to make reproducible (in a perfect world, I would make my Python dependencies reproducible but the system package works well enough on my system and I can’t be bothered to write a recipe that builds some autotools-based C project with its own dependencies on obscure C projects scattered across the Internet, each with their own special snowflake build systems and their own dependency trees), etc. Nix is definitely the right direction (it could completely solve the monorepo/multirepo problem and dramatically simplify CI/CD and otherwise significantly improve software development), but it’s not very pragmatic. It badly needs some product leadership.
X6S1x6Okd1st 6y ago
Specific to NixOs, but running software that hasn't already been packaged for nix can be quite hard. The language is not simple and discovery can be quite lacking.

aidenn0 6y ago

TIL that there are people who like the nix language. I love NixOS, but hate the language.

kalium-xyz 6y ago
A lot of big names in the Nix community work for Tweag, including Eelco Dolstra who did the initial Nix/NixOS thesis.
throwaway894345 6y ago
I feel similarly. I will say that the lack of static typing compounds just about every other problem I have with Nix, so this seems like a marked improvement, but still it would be nice to have something that is more familiar and intuitive to programmers generally.

HelloNurse 6y ago

Documentation is terribly lacking, but in addition examples are perplexing. Where is the "Url" type, which should be a union of a constrained subset of "Str" (as built by mylib.makeURL) and a three component record (as used in the example)? Where can I specify that the "urls" list in the configuration must be present and nonempty, and that "host" and "port" are mandatory (or not)? And that the type of "urls" is a list of "Url", the type of "port" is "port", etc.? How can a general purpose port type have a meaningful default value, given that duplicates are generally fatal and a configuration can contain multiple ports? Only individual port usages (e.g. ftpPort and telnetPort in a commonNetworkServices record) should have defaults. Where is the body of numToStr? Maybe in some unmentioned standard library? Is there some kind of enforceable separation between the configuration file (untrusted and supplied by the user) and a schema it must satisfy (trusted and supplied by the application)? What forbids the configuration file to, say, redefine makeURL as Str -> Str -> Num -> Str -> Str -> Str (to add a HTTP path and query string) and make the application's URL parsing fail?

yagoham 6y ago
Hi, blog post author here. As stated at the beginning, the project is still WIP, and I didn't expect it to end on HN. Sorry for the lack of documentation and convincing example for now ! The post is intended to present the ideas and let people look at the repo, but while the core language is rather complete, a lot of small but important things are still missing to make it usable right now. I sidestepped this issue to still be able to illustrate some points with simple examples. To answer your questions: - numToStr would be either in a standard library, or implemented here using standard library functions. It does not exist currently - the fact that the urls are non empty could be either enforced by the contract itself (ie use a NonEmptyUrl contract). Or you could combine a NonEmptyStr and a Url contract via merging. By default, things with a contract without a default must be present. - I'm not sure I fully understand your last point, but I don't think a configuration file could "redefine" a function in any meaningful way. Say you write a program "main.ncl", which import some external downloaded source, check it against a contract you've written, and generate a config accordingly. Since the language is pure and variable are lexically scoped, nothing from an import could redefine anything in your current scope, be it your contracts or your functions.

throwaway894345 6y ago

There is definitely a need for this class of tools (statically typed expression-based config languages), but it’s super weird to praise Nix as very simple and easy to understand and then to gripe that Dhall is complex. I’ve been trying to learn Nix on and off for years and the language is often one of the things that trips me up. Dhall on the other hand looks simple even though it insists on a Haskell-esque syntax (which I find to be difficult to read). All I really want is Starlark with type annotations or something similar, but Dhall is pretty close.

galkk 6y ago

I don't want to interpret my configuration. I don't want it to act as a source code. Many places at Google use text representation of protobuffers as config files and I like it a lot. It's typed. It's readable. The schema and text are source controlled. You have code generators for many languages. (Work at Google, opinion is mine)

q3k 6y ago
Text protos are decent (although a lack of a public spec limits their adoption outside of Google). But that's just a serialization format, only a piece of the configuration puzzle. You may argue that all configuration should be fully static, but having gone from writing GCL/BCL at Google to having to deal with static, text-templated, copy-pasted YAMLs in the Real World - I want my GCL back. :)
- sa46 6y ago
  Ironically, nickel was the name of the Google-internal, ill-fated replacement for GCL. I think it just lives on in blueprint files. I also miss GCL, as quirky as it was. YAML composes poorly.
- verdverm 6y ago
  GCL also came up in the Cue slack channel today, and why Cue is intentionally different https://cuelang.org/community https://cuelang.org/community
- ithkuil 6y ago
  If you want GCL back, here you have it: https://jsonnet.org/ https://jsonnet.org/ A bit cleaned up (withour "up" and without weird scoping rules)

saurabhnanda 6y ago

How different is this gradual typing from TypeScript?

LukeBMM 6y ago

Yet Another Yet Another Markup Language?

jackric 6y ago
It's got functions so not a markup, it's Yet Another Fucking Config Language

vlovich123 6y ago

I love using cap'n'proto to store configurations. It's really slick. + Type safe since you define the schema for every part of your configuration. + Ridiculously easy to read/understand (writing has a small ramp curve to build the muscle memory). + Type system is very flexible & rich (lists, maps, unions), supports generics (for built-in & custom types) & everything is neatly composable (including constants that reference constants). + The config compiles down to a minimal binary file. + You can have arbitrary configurations in 1 file (you compile a specific constant to a file so each configuration you want is just 1 constant value you define in the schema). + The binary file can be converted back to text using standard cap'n'proto tools so it's easy to double-check the config you're deploying. + Perfect backward/forward support for the configuration as long as you follow the rules (similar to protobuf/flatbuffers) since you have to define the schema for your constants. + Loading the config file from any data source (disk, network) is trivial and for on-disk usages you could mmap the struct to get even better performance (only the parts of the config you access would get paged in). + Supports a variety of languages. While the RPC stuff has a bit less adoption, the parts needed for configuration should be available in the most popular languages (I think the only missing language generator is Swift). The only negatives are: - You do have to define the schema for your config - There's a single upfront cost to integrate cap'n'proto into your build system if you're not using CMake. Neither feel like prohibitive negatives though. If you're looking at strong typing, why not go all the way & make sure that your entire config has a strongly typed structure to it? Not just that some field is an int also that there's certain specifically named fields & that accessing these fields in a strongly-typed languages will result in build errors if you forgot to change something.

olodus 6y ago

I see so many complain about these config languages. I think it is a bit black and white to think of a perfect separation between code and config. In a perfect world and easy situations yes. But I think you are all missing the point of these languages. In my eyes they are templating languages. Have any of you guys seen the Helm charts some of us have to write daily? Mustache coding is terrible! I would love any of these languages instead of that. The only thing worse would be to have to write all the yaml yourself every time instead of any templating at all. To that end, you could use any programming language to generate the yaml for your config. But then some of these has some nice features. I really like that Dhall is only a total lang and it does some nice things with importing dhall files. That is not to say you should unnecessarily complicate your config. Keep it easy and separate from code as much as possible.

aldanor 6y ago

Opened the link wondering what's a "better configuration for `less`"... oh well.

X6S1x6Okd1st 6y ago

> All in all, the Nix language is a lazy JSON with functions. That's news to me! I have been slowly learning the language, only enough to configure and package stuff as needed for my personal use, but it has not just been a lazy JSON with functions...

verdverm 6y ago

Timely, we are talking about TC in the Cue slack today. Nickel calls out its Cue inspiration while maintaining TC. https://github.com/tweag/nickel#related-projects-and-inspirations https://github.com/tweag/nickel#related-projects-and-inspira... https://cuelang.org/community https://cuelang.org/community

anentropic 6y ago

What is the selling point for this vs say Dhall, which looks superficially similar (Haskell-ey config lang)? The examples for Nickel currently look a lot more like programming than config. Why not just use a general purpose programming language that you already know and has many libraries and mature tooling? Dhall advertises a lack of Turing-completeness as a feature. One of the Nickel examples shows an implementation of fibonacci function... (I have no idea if this implies Nickel is TC, nor why I would need such power in my config) > Dhall features a powerful type system that is able to type a wide range of idioms. But it is complex, requiring some experience to become fluent in. > Gradual types also lets us keep the type system simple: even in statically typed code if you want to write a component that the type checker doesn’t know how to verify, you don’t have to type-check that part. > Complementary to the static type system, Nickel offers contracts. Contracts offer precise and accurate dynamic type error reporting, even in the presence of function types. It's not clear to me that gradual typing as a mix of statically typed code + untyped code with "contracts" is necessarily simpler than just one or the other. It's not clear to me that a specialised config language combining code + schemas is necessarily better than an existing general purpose language + some language-agnostic schema DSL. I don't know, maybe this is all great, I feel the benefits need to be more clearly explained and demonstrated though.