3 ms·

We use tons of configuration at work and this has come up time and time again. The biggest pain is when inevitably you need to "parameterize" your JSON files. T

We use tons of configuration at work and this has come up time and time again. The biggest pain is when inevitably you need to "parameterize" your JSON files. The most common route I've seen is to turn your configs into templates e.g. Jinja and go from there. Welcome to hell. My solution? Write a DSL with Typescript, and "compile" your configs down to JSON. Define the structure of your configs as TS types. Write functions with free variables to work as templates, or define snippets as composable fragments. Writing the end result to a JSON file results in transparency (what you see is what you get) and compatibility.

unrealhoang 4y ago
jsonnet[1] might suit your use-case better, it was created to do exactly that. [1]https://jsonnet.org https://jsonnet.org
- Matheus28 4y ago
  Not OP. Looks interesting, but I'd bet that typescript's type system would enforce constraints a lot better
  - whacked_new 4y ago
    in my experience, for things like static configs, jsonnet has been superior to TS for constraints, if you pair the configs with a json schema (also generated from jsonnet, of course). TS's types are easier to use at write-time, but json schema includes a lot of batteries that save a lot of time when you need them, but become annoying in TS, like patternProperties in dictionaries, length constraints in keys. Of course, there are situations like key1 XOR key2 where you will need custom logic one way or another, but given json schema's evolution over the years, I think it's pretty solid. An added benefit is that if you stay in json schemas, you're almost guaranteed a validator exists in $other_language, it _almost_ feels like a first-class construct everywhere.
    - Matheus28 4y ago
      That's a valid point. I think that TS might also lack constraints for object keys and values (like matching them against a regex). After thinking for a while, I think that json schema might be the best approach to it (possibly generated from jsonnet if complicated enough).
- whacked_new 4y ago
  jsonnet is my go-to language for anything related to configuration, after having tried json, yaml, TS, edn, and tasting dhall and toml. It addresses all problems in the article and more. the composition model strikes a good balance between data extensibility / language expressiveness / ease of use. the generated json leads to easy-to-understand and portable data, and if you write jsonschemas from jsonnet, tools like json-schema-to-typescript [1] make it easy to import a consistent interface, and almost every language has a reasonably up-to-date validation library. [1] https://github.com/bcherny/json-schema-to-typescript https://github.com/bcherny/json-schema-to-typescript
campers 4y ago
+1 for TypeScript for config. I've been using Pulumi with TS to create a simple DSL to define all our GitLab config and alerting policies in GCP. It makes it easy to have concise definitions and quickly refactor the DSL with confidence. Having a proper type system removes so many possibilities of inevitable accidental configuration mistakes. One of our other types of deployment projects has a yaml configuration. One time a section had the wrong tab indenting, took a little while to debug through why a table wasn't being provisioned.
- OliverGilan 4y ago
  I’d love to learn more about how you do this. Do you have any helpful resources that I can look at to understand the process of creating a DSL with TS + Pulumi?
  - eurasiantiger 4y ago
    Just create types for all objects in your configuration as well as the configuration object itself. The ”DSL” is the typings, and the rest is nothing more than a simple TS program which calls JSON.stringify() with the type-checked config object and writes the resulting string to disk.
  - campers 4y ago
    Calling it a DSL is maybe bit of a stretch, its just creating a concise abstraction above the provided API. So instead of having 20 lines of code for every alert policy like at https://www.pulumi.com/registry/packages/gcp/api-docs/monitoring/alertpolicy/#monitoring-alert-policy-basic https://www.pulumi.com/registry/packages/gcp/api-docs/monito... you create a function that generates all the standardised alerts from concise strongly typed config such as cloudSqlAlerts({ instanceId: 'sql-main', highCpu: { percent: 70, duration: _5min }, highMemory: { percent: 80, duration: _5min }, highDisk: { percent: 80, duration: _10min }, deadlockDetection: true }, projectNotificationChannels)
valbaca 4y ago
Sounds like what AWS CDK basically does: uses TypeScript to write JSON (Cloudformation).
AtlasBarfed 4y ago
Configuration is a rabbit hole that starts with a single innocent file (while it is not total bikeshedding arguing about the serialization format, it is just the tip of the iceberg), and descends to many complications: https://www.blogger.com/blog/post/edit/9049876254685009339/1545856458424837048 https://www.blogger.com/blog/post/edit/9049876254685009339/1... The author of the post is already signalling descent into config hell, they call JSON a language, which it TECHNICALLY is, but it shows their strong desire for more power and features and a descent into config hell.

unrealhoang 4y ago

jsonnet[1] might suit your use-case better, it was created to do exactly that. [1]https://jsonnet.org https://jsonnet.org

Matheus28 4y ago
Not OP. Looks interesting, but I'd bet that typescript's type system would enforce constraints a lot better
- whacked_new 4y ago
  in my experience, for things like static configs, jsonnet has been superior to TS for constraints, if you pair the configs with a json schema (also generated from jsonnet, of course). TS's types are easier to use at write-time, but json schema includes a lot of batteries that save a lot of time when you need them, but become annoying in TS, like patternProperties in dictionaries, length constraints in keys. Of course, there are situations like key1 XOR key2 where you will need custom logic one way or another, but given json schema's evolution over the years, I think it's pretty solid. An added benefit is that if you stay in json schemas, you're almost guaranteed a validator exists in $other_language, it _almost_ feels like a first-class construct everywhere.
  - Matheus28 4y ago
    That's a valid point. I think that TS might also lack constraints for object keys and values (like matching them against a regex). After thinking for a while, I think that json schema might be the best approach to it (possibly generated from jsonnet if complicated enough).
whacked_new 4y ago
jsonnet is my go-to language for anything related to configuration, after having tried json, yaml, TS, edn, and tasting dhall and toml. It addresses all problems in the article and more. the composition model strikes a good balance between data extensibility / language expressiveness / ease of use. the generated json leads to easy-to-understand and portable data, and if you write jsonschemas from jsonnet, tools like json-schema-to-typescript [1] make it easy to import a consistent interface, and almost every language has a reasonably up-to-date validation library. [1] https://github.com/bcherny/json-schema-to-typescript https://github.com/bcherny/json-schema-to-typescript

campers 4y ago

+1 for TypeScript for config. I've been using Pulumi with TS to create a simple DSL to define all our GitLab config and alerting policies in GCP. It makes it easy to have concise definitions and quickly refactor the DSL with confidence. Having a proper type system removes so many possibilities of inevitable accidental configuration mistakes. One of our other types of deployment projects has a yaml configuration. One time a section had the wrong tab indenting, took a little while to debug through why a table wasn't being provisioned.

OliverGilan 4y ago
I’d love to learn more about how you do this. Do you have any helpful resources that I can look at to understand the process of creating a DSL with TS + Pulumi?
- eurasiantiger 4y ago
  Just create types for all objects in your configuration as well as the configuration object itself. The ”DSL” is the typings, and the rest is nothing more than a simple TS program which calls JSON.stringify() with the type-checked config object and writes the resulting string to disk.
- campers 4y ago
  Calling it a DSL is maybe bit of a stretch, its just creating a concise abstraction above the provided API. So instead of having 20 lines of code for every alert policy like at https://www.pulumi.com/registry/packages/gcp/api-docs/monitoring/alertpolicy/#monitoring-alert-policy-basic https://www.pulumi.com/registry/packages/gcp/api-docs/monito... you create a function that generates all the standardised alerts from concise strongly typed config such as cloudSqlAlerts({ instanceId: 'sql-main', highCpu: { percent: 70, duration: _5min }, highMemory: { percent: 80, duration: _5min }, highDisk: { percent: 80, duration: _10min }, deadlockDetection: true }, projectNotificationChannels)

valbaca 4y ago

Sounds like what AWS CDK basically does: uses TypeScript to write JSON (Cloudformation).

AtlasBarfed 4y ago

Configuration is a rabbit hole that starts with a single innocent file (while it is not total bikeshedding arguing about the serialization format, it is just the tip of the iceberg), and descends to many complications: https://www.blogger.com/blog/post/edit/9049876254685009339/1545856458424837048 https://www.blogger.com/blog/post/edit/9049876254685009339/1... The author of the post is already signalling descent into config hell, they call JSON a language, which it TECHNICALLY is, but it shows their strong desire for more power and features and a descent into config hell.