13 ms·

The perfect configuration format? Try TypeScript

136 points by tmcneal 5y ago

taylodl 5y ago
You know what we don't need? Yet Another Configuration Format! Also, don't tell me I can't add comments to JSON because I sure can - add it as a field! If a comment is that important then it's important enough to transfer along with the dataset and durably persist.
- cout 5y ago
  I've done that before and found that my comment got reordered by a tool that thought the order of objects didn't matter.
emodendroket 5y ago
OK, so we're coming back around to the "configuration as code" trend again then.
enriquto 5y ago
Typescript has a lot of punctuation and thus it's not really human-editable. I'd prefer a ini-style format or dockerfile style (with one directive per line of text, clearly identified by a single uppercase word). Moreover, these formats are so easy to parse that you do not even need a library to do so.
- ComputerGuru 5y ago
  Except there’s the ini where strings need to be quoted and the one where they don’t. The one where white space is allowed after the equal sign and the one where it’s not. The one where keys are case sensitive and the one where they’re not. TOML cleans up most of that, though.
- tomtheelder 5y ago
  I'm puzzled what you mean by this. Do you mean it's not easily editable by non-engineers?
  - enriquto 5y ago
    It's annoying to edit by everybody. I guess engineers will especially abhor the unnecessary complexity. This is the proposal of TFA: export const config: MyAppConfig = { locale: 'en-US', timezone: 'America/New_York', logLevel: 'info', environment: 'local', } And this is how simple it could be: LOCALE en-US TIMEZONE America/New_York LOGLEVEL info ENVIRONMENT local The first version has 19 useless punctuation symbols and a few spurious words that are not related to the configuration itself, but are constraints of the typescript implementation. The second version is much simpler, easier to write and to parse using trivial string processing, without depending on any third-party library. It is beyond me how could any serious engineer honestly find any advantage in the first version.
    - aldonius 5y ago
      Well, if you want nesting on your map then you need something slightly more complicated than your second version, but I will grant that you could do something like: A.B value C value D.E.F value D.E.G value
the_arun 5y ago
How to add comments in json or TypeScript config files? Commenting is very much needed for any manually written code or config we keep in repositories.
strogonoff 5y ago
TypeScript is limiting for describing schema in my experience. Try requiring an array of certain length, for example. Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage. To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
- throwaway894345 5y ago
  Dhall is too unfamiliar for most developers, IMO. Developers don’t want to have to grok an entirely new syntax just for config. I’m personally surprised no one has added type annotations to Starlark yet.
  - strogonoff 5y ago
    Everything is unfamiliar, until one adopts it.
    - deleted 5y ago
      [deleted]
    - throwaway894345 5y ago
      I didn't think it needed to be explained, but not all things are equally unfamiliar nor equally easily adopted.
  - mattw1810 5y ago
    The Rust implementation has it as an experimental extension (https://github.com/facebookexperimental/starlark-rust/blob/main/docs/types.md https://github.com/facebookexperimental/starlark-rust/blob/m...)
    - throwaway894345 5y ago
      Oooh, very cool! I'll check it out and maybe port to Go.
ravenstine 5y ago
My only real criticism is that I believe it's generally an anti-pattern to have configuration that necessitates DRY. It has essentially the same problems as OOP inheritance. Because configuration isn't and shouldn't be an application in and of itself, avoiding potential issues and misdirection from avoiding DRY in this case seems like the right tradeoff to having to use multi-select to change the config format in your IDE.
- dqpb 5y ago
  Cuelang enables DRY while forbidding inheritance.
intellix 5y ago
YAML is beautiful when it's simple and horrible when you need to dynamically generate it. The indentation...
pramodbiligiri 5y ago
Anyone have experiences to share about Dhall (https://dhall-lang.org https://dhall-lang.org)? On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"
- junon 5y ago
  Sorta. I evaluated it a long time ago and decided against ever touching it due to its remote import functionality. Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
  - strogonoff 5y ago
    > Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests. Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
- Reitet00 5y ago
  Yeah, I've used that a year ago or so. Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc. Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
  - ReleaseCandidat 5y ago
    > no bindings to various languages. There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.html#language-support https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...
ReleaseCandidat 5y ago
Great idea. And because you need many constants that often change in such a configuration file you could put that all in another file and include that in the 'real' configuration file. Why not call that, hmm, `config_config.txt`. But as plain text is a bit hard to parse, we could give that file some structure, so that it is like, well, another markup language, just without markup and easier to read and write.
sbr464 5y ago
I use TypeScript and like the ecosystem. For broader use, I would like to see native support or an official extension for creating dependent types or specific constraints. Things like number ranges, or the basics from JSON Schema, etc.
tantalor 5y ago
Using programming languages for configuration is a bad idea.
- bryanrasmussen 5y ago
  why? It can't be because then if something goes wrong you need to debug your configuration, because you need to do that often enough with a non-programming language format for even less power offered.
  - krapp 5y ago
    If your config is so complex that it needs "debugging" then it's too complex. The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability. If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum. And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
- taeric 5y ago
  Emacs would like a word. :)
- latchkey 5y ago
  Static declarations without a reason are equally bad.
- inoffensivename 5y ago
  gradle has entered the chat
  - 5y ago
sirtimbly 5y ago
The line between configuration and code is always a little blurry. There are some projects where using a .js file for the "configuration" of another operation is incredibly useful. TS makes even more sense in many of those situations. Lots of configuration files require pre-processing already, like merging shell env vars in with textutil or something like that. So, I wouldn't dismiss this opinion outright.
jayd16 5y ago
Kind of interesting but now I need to run arbitrary code to read a config?
- taeric 5y ago
  To be fair, you already have to execute arbitrary code to understand what config is doing. Where execute is to sequence a series of effects that are described in a file.
  - jayd16 5y ago
    >To be fair, you already have to execute arbitrary code to understand what config is doing. That's not arbitrary code. That's my compiled, tested and verified code.
    - taeric 5y ago
      All code is arbitrary code if you are paranoid enough. :)
richeyryan 5y ago
I feel like the AWS CDK is a great example of this working well. Rather than puzzling over what config options are valid, I get autocomplete and compiler errors when I insist on using an incorrect option. You also get stuff like deprecation warnings showing up in your editor. It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
- Lucasoato 5y ago
  This is so cool, I feel like other similar projects, like k8cdk, are going in the same direction for that reason. The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.
Shadonototra 5y ago
I can't believe what i am reading...
- crate_barre 5y ago
  JS people gonna JS.
  - npsimons 5y ago
    > JS people gonna JS. Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
realrocker 5y ago
Great little writeup ! After mangling YAML, HCL, JSON for years as an ops engineer, I have come to the same realisation. In fact, I have put this into practice in production pipelines by using: jkcfg[1] for the last couple of years. Two data points: 1. Zero developer support contract rate around YAML syntax and templating issues 2. High number of contributions in our private typescript configuration library from developers. Using typescript as an ops frontend has made operations a lot more approachable to folks. Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration. [1] https://jkcfg.github.io/#/ https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg https://github.com/dxcfg/dxcfg
nsm 5y ago
I’m not on board with using a language that only has a single implementation and is complex enough that implementing others is difficult. Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
- pbalau 5y ago
  If you can afford to wear only one hat, then sure, use one special language for configuration. But I can't, I am the entire infra department and I still need to do product and framework work. Not having to switch context, language wise, is a very nice thing to have.
- tn890 5y ago
  Honorable mention for Cue[0]. [0] https://cuelang.org/ https://cuelang.org/
  - Kinrany 5y ago
    CUE does look simple enough to be implemented in every language. I wonder if there's anything preventing that. Perhaps there's space for a CUE-like language with an even simpler spec.
- throwaway894345 5y ago
  TypeScript has at least 2 implementations: tsc and Deno, but I take your point and agree. That said, most languages probably have a JS runtime, so a basic implementation would be ts->js->interpreter. The ts->js is presumably pretty easy.
  - junon 5y ago
    I don't think Deno actually re-implements typescript, does it? Seems like a lot of wasted effort. They don't even use Typescript internally.
eyelidlessness 5y ago
I have used TypeScript for config in a few projects, with node-config[1], and it’s substantially better IME than JSON/YAML/etc. Yeah it’s unconventional to write configs in a general purpose language, and I understand why that would be undesirable. But having configs validated at compile time is great at preventing bugs. I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic. 1: https://github.com/lorenwest/node-config https://github.com/lorenwest/node-config
pphysch 5y ago
My money is on https://cuelang.org https://cuelang.org in this space. As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
s_gourichon 5y ago
Dhall seems to gather some praises, and not fall into the pits mentioned here. https://news.ycombinator.com/item?id=17523623 https://news.ycombinator.com/item?id=17523623
- wyager 5y ago
  Dhall is awesome. Only downside is that the included formatter doesn't preserve comments. Super well-designed and with interesting features like semantic hashing.
zffr 5y ago
Apple takes a similar approach for swift. Package.swift files run real swift code and export an object that specifies the package’s configuration. In practice I think it works fairly well.
progre 5y ago
TypeScript needs to be compiled, right? So what use is a configuration file if I need to run it through the build pipeline? The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.
- klodolph 5y ago
  The fact that it is compiled does not mean you need to run it through an additional step in your build process. In practice, it would just mean that you include TypeScript in your package, which can either be used as an executable or as a library. It can be completely transparent to the user. You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
- nverno 5y ago
  > To use this approach today requires that your project also be written in Typescript. It also means you’ll need to forgo a separate flat-file for your configuration. You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
- dragonwriter 5y ago
  > TypeScript needs to be compiled, right? Yes. > So what use is a configuration file if I need to run it through the build pipeline? Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library. > The point of having configuration is to gain the ability to configure software without rebuilding in my mind. There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
- handrous 5y ago
  I think we got here because everyone's mixed up describing a state with describing the process to describe a state. Or, configuration versus scripts that apply configuration. Most of the popularly config-as-code systems mix the two up, including Ansible, my favorite one. You end up with "config files" that are, all at once in the same file: 1) configuration, 2) templates for configuration, and 3) programs that modify config when they're run. I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh. So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
pstch 5y ago
Is evaluation lazy in TypeScript ? For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules. Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function. I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
- richeyryan 5y ago
  Typescript is definitely eagerly evaluated unless you were to use a first class function to defer execution
ThePhysicist 5y ago
I think parsing YAML or JSON into typed structures is the easier way to go. I e.g. do that in Golang using a little form validation and coercion library I've written. The end result is a nested, strongly typed data structure. Here's an example: https://github.com/iris-connect/eps/blob/master/settings.go https://github.com/iris-connect/eps/blob/master/settings.go (the accompanying form validation configuration: https://github.com/iris-connect/eps/blob/master/forms/settings.go#L78 https://github.com/iris-connect/eps/blob/master/forms/settin...). Hashicorp has some libraries that do similar stuff, I wrote my own though since I need it quite often and wanted to have full control over it. The library is just a few hundred lines of Golang code (https://github.com/kiprotect/go-helpers/tree/master/forms https://github.com/kiprotect/go-helpers/tree/master/forms). Undocumented as of now as it's nothing I want to explicitly share with the world. In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
melolife 5y ago
I've done a similar thing with Scala in the past - ship the compiler jar and then compile your configuration (which is just an abstract class implementation) as part of the startup process.
steeve 5y ago
Starlark or Cue, but not a general purpose language please. Side effects in a configuration language is an atrocity...
- IshKebab 5y ago
  It's possible to sandbox most languages, and with some work you can probably make them deterministic too. Here's an example: https://github.com/jkcfg/jk https://github.com/jkcfg/jk That beats having to learn an entirely new language.
fay59 5y ago
Probably important to mention that at the moment, TypeScript as a configuration format means anyone who writes a configuration blob for you has arbitrary code execution. This is rarely an issue in practice, though.
- cphoover 5y ago
  It also means your configuration must be transpiled into JS first before execution unless run through a typescript runtime interpreter.
- mejutoco 5y ago
  This could be solved by having some kind of sandbox (https://github.com/patriksimek/vm2 https://github.com/patriksimek/vm2), but I agree it complicates it. It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
  - realrocker 5y ago
    One of such sandboxes is Deno.
    - cormacrelf 5y ago
      Not even Deno can solve the halting problem, unfortunately.
      - Zababa 5y ago
        No need to solve the halting problem, just put a timeout.
        cormacrelf 5y ago
        If you use it by running a V8 isolate that transforms the exported config into JSON, then sure. But TS-based config is largely for people using TS elsewhere. And in that case: // this returns in 0 ms export const config: any = { get value() { while (true) {} } } as const; // accessing config.value runs forever console.log(config.value); For a less unrealistic perspective, if you write enough configuration in a Turing-complete language, you will eventually be tempted to use the full power of the language. You may find yourself having to write tests for the config, which might be e.g. parsing other files, making HTTP requests, querying the filesystem/OS and more. If you're doing that, you are no longer really writing configuration. The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format. ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
nly 5y ago
Protocol Buffers text format ?
- jonny_eh 5y ago
  Protobuffs remain very underrated outside of Google.
- joshuamorton 5y ago
  Composability in configuration is really important beyond a small scale. If you're running a replicated server in n locations, it's far better to keep one config file that has some form of conditional logic based on location than to maintain n textprotos that need to be kept synced.
deleted 5y ago
[deleted]
11235813213455 5y ago
or json-schema with jsonc or js files
ljm 5y ago
I've only worked with cdk8s, but the experience of that alone was enough for me to regret it. It's not that cdk8s is bad, per-se, it's that it turned an otherwise declarative kubernetes config into something that was much harder to grok in the first instance. One of the arguments for TS in the post is that it can encourage DRY. It was DRY that made this TS-based kubernetes config so hard to follow, because simple repetitive statements got abstracted into parameterised functions, or classes to inherit from, or other things that are resolved dynamically at runtime. Not to mention that you're also going to start mixing in random dependencies from NPM to help abstract things further. This is great for application development but not so great for pure configuration. A language designed for config, like Dhall or CUE, can give you much of what a typed language offers without letting you go crazy with a full-blown language runtime.
- tetha 5y ago
  Those were my concerns as well. Your config file can stay simple, sure. But I'm pretty sure it won't. And from some years of experience with configuration management systems, configuration creation can accrue a surprising amount of complexity over time. Mostly because unlike the code they support, a config management system has to support many versions of the code and multiple versions of a configuration at the same time.
- dogleash 5y ago
  Sounds like the Configuration Complexity Clock http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html http://mikehadlow.blogspot.com/2012/05/configuration-complex...
  - photojosh 5y ago
    Oh I enjoyed that, thanks. Shared around at work. Likely controversial, but I really like how Django does its settings. I think because Python code can start off just looking like a .ini file with variables and assignments, and get more complex only as required. On the other end of my work, I'm a fan of having embedded firmware apply to as many devices as possible, and then specialise with configuration settings from EEPROM.
escherize 5y ago
Good to see people coming to the same conclusions we have had for a while using Clojure. There's an up and coming runtime data validation framework called Malli that I use for these sorts of files. You can get many of the benefits but also run custom validations (dictate how two keys in a map relate to eachother, or force a be of a certain shape instead of just a string). Here's the example from the script: [0] [0] - https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%20%3Atimezone%20%22America%2FNew_York%22%0A%20%20%3Alog-level%20%22info%22%0A%20%20%3Aenvironment%20%22local%22%7D&schema=%5B%3Amap%20%0A%20%5B%3Alocale%20string%3F%5D%0A%20%5B%3Atimezone%20string%3F%5D%0A%20%5B%3Alog-level%20%5B%3Aenum%20%22trace%22%20%22debug%22%20%22info%22%20%22warn%22%20%22error%22%20%22fatal%22%5D%5D%0A%20%5B%3Aenvironment%20%5B%3Aenum%20%22local%22%20%22dev%22%20%22staging%22%20%22production%22%5D%5D%5D https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%...
SavantIdiot 5y ago
Except .... (...No one asked, but here are my $0.02 ;-) You need typescript in your environment to use typescript! The four formats the OP listed can be read by anything from Ada to EXCEL. But typescript can only be ready by a TS build flow. In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.[1] FOR GENERAL PURPOSE: big no. BUT: If this is purely for TS projects where the tooling already exists, then OK. I'd use it. EDIT [1] I'm referring to the article's early suggestion to run JSON through JSMin to support comments. So transpiling is more apt, not compiling.
- __ryan__ 5y ago
  > In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler. Does JSON need a compiler?
  - a_lost_needle 5y ago
    It does not. Easily implemented with combinators the whole way down.
    - SavantIdiot 5y ago
      Correct, but the very beginning of the original article proposes running JSON through JSMin to support comments. So in this context, a "compiler" is being proposed (or transpiler to more modern). My comment isn't clear.
      - a_lost_needle 5y ago
        Still easily implemented with combinators the whole way down. It’s building an ast, that’s it. Once you have a tree, you can do whatever you want with the result before passing it to a compiler for a language. Here though, just serialization stuff.
        SavantIdiot 5y ago
strifey 5y ago
I'm curious what folks who agree with this think about setup.py then? There's been a push to move to setup.cfg to specifically avoid the fact that python libraries run arbitrary code on install.
- nonameiguess 5y ago
  The problem with setup.py is different. Because it can run arbitrary code, package writers go crazy and import extra dependencies, but you specify your package dependencies in setup.py, so without some out-of-band way to tell your users that the setup.py file in itself also has dependencies, package installation on machines other than your own might fail unless by sheer luck your users already had the packages being used in setup.py. That problem doesn't apply to arbitrary configuration, but only to configuration files that represent package build and installation metadata. You do see this issue with things like Gradle, though, where your build.gradle file can represent the dependencies of the package being built, but the build file itself may have dependencies since you can run arbitrary Groovy code and import any package that can be contained in a jar file. It gets around this because Gradle itself can just download those dependencies before trying to execute the build script, which would be perfectly possible for a real Python build system, but not when setup.py is just being executed by the Python interpreter, which doesn't do dependency management and expects imports to already be available in the system import path.
  - nonameiguess 5y ago
    Heck, for what it's worth, even Make has this issue, as people quite often write Makefiles assuming you have a bunch of tools installed that may or may not be part of the POSIX specification. I wish I could find it now, but somewhere in the GNU Make guide book is a list of more or less "approved" utilities you're supposed to be able to safely assume are installed on any GNU based system that you can use in a Makefile, and you're not supposed to go beyond that.
whirlwin 5y ago
The idea of allowing logic from a programming language in a config format is.... at best very horrifying...
vander_elst 5y ago
What would be the problem with using something like protobufs? To write the configuration, you can use whatever language you want, what matters are the outputs anyway. The outputs are stored by automation in text format so that they can be better inspected. The programs that use such configuration have the advantage of knowing the schema of the configuration and directly parse it avoiding possible typing problems. The nice thing is that you can use the same language to write the programs and the configuration.
- mikojan 5y ago
  protobuf may very well be total overkill. What you wanted to do was strongly type your configuration. What you get is a complete RPC layer. This might not even be warranted when inter-process communication is a primary concern. After all, there's nothing inherently wrong with using whatever data transfer mechanism your standard lib provides. It's not just protobuf though. Almost every data format for serializing structured data suffers from pushing a jungle on you when all you asked for was a banana. The least invasive format I know of is FlatBuffers[0]. Doesn't push RPC on you. Allows you to parse a message even if you do not know this specific message's data structure in advance (like JSON). [0]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
  - vander_elst 5y ago
    1. As said in the original comment, it doesn't necessarily have to be protobuf, but just a data serialization system. 2. Protobuf per se is just a serialization system, it does not push RPC "Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler." [0]. It can be used in combination with RPC systems, but does not have to. [0] https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers
beebeepka 5y ago
Typescript is perfect for stuff like this because it's so much less verbose than most things out there As a bonus, you might not need validation and schemas, too.
shadowgovt 5y ago
I like the idea of the IDE assisting with the schema-correctness of the config (which you get for free with TypeScript coupled with any competent IDE configured to use the typechecker to validate the code). ... but as a rule of thumb, if your config language is Turing complete, you're setting yourself up for future pain as complexity of config grows with time and someone makes the bad decision to take advantage of that Turing completeness...
harunurhan 5y ago
Not typed support but I find jsonnet [0] pretty useful when you have tons of configuration files. [0] - https://jsonnet.org/ https://jsonnet.org/
eb0la 5y ago
Really interesting but, sincerely, I would rather use sqlite. I know: It's a binary format. And you need to add a library to read it. BUT - Your configuration _has_ data types. - You can have a copy with an old config within the file easily (create table backup_config as select * from actual_config; ). - The file is small. - Encryption is built-in if you need it. - And it's programable
fourseventy 5y ago
Personally I wouldn't use Typescript for config. It breaks my #1 dev rule which is KISS. Keep it simple stupid. Simplicity is orders of magnitude more important than other things like DRY or config schema.
Slasher1337 5y ago
I've really come to prefer json5 as a config format: https://json5.org/ https://json5.org/ It's basically Json with comments, trailing commas, and unquoted object keys (and some other minor things). Much nicer than real json as a config format, but still simple and declarative, not requiring an interpreter or being turing-complete.
yayr 5y ago
Should this be a future ECMAScript enhancement or is the better route for browsers to natively support TypeScript?
veidr 5y ago
I think this is kind of a no-brainer, if your project is already based on TypeScript. Not just for configuration, but for things like test fixture data, and even data assets like i18n translations, or basically anything that projects might commonly use JSON for, internally. It's just so much easier to create, edit, manage, and deal with in TypeScript. But it is hard to imagine TypeScript-as-config taking root in projects that don't already use TypeScript.
vendiddy 5y ago
I think typescript as a config format alone may get too complex. But json backed by a typescript interface definition could be a useful middle ground. You're still writing json but you get nice code completion and realtime validation of the config file.

136 points by tmcneal 5y ago

You know what we don't need? Yet Another Configuration Format! Also, don't tell me I can't add comments to JSON because I sure can - add it as a field! If a comment is that important then it's important enough to transfer along with the dataset and durably persist.

cout 5y ago
I've done that before and found that my comment got reordered by a tool that thought the order of objects didn't matter.

emodendroket 5y ago

OK, so we're coming back around to the "configuration as code" trend again then.

enriquto 5y ago

Typescript has a lot of punctuation and thus it's not really human-editable. I'd prefer a ini-style format or dockerfile style (with one directive per line of text, clearly identified by a single uppercase word). Moreover, these formats are so easy to parse that you do not even need a library to do so.

ComputerGuru 5y ago
Except there’s the ini where strings need to be quoted and the one where they don’t. The one where white space is allowed after the equal sign and the one where it’s not. The one where keys are case sensitive and the one where they’re not. TOML cleans up most of that, though.
tomtheelder 5y ago
I'm puzzled what you mean by this. Do you mean it's not easily editable by non-engineers?
- enriquto 5y ago
  It's annoying to edit by everybody. I guess engineers will especially abhor the unnecessary complexity. This is the proposal of TFA: export const config: MyAppConfig = { locale: 'en-US', timezone: 'America/New_York', logLevel: 'info', environment: 'local', } And this is how simple it could be: LOCALE en-US TIMEZONE America/New_York LOGLEVEL info ENVIRONMENT local The first version has 19 useless punctuation symbols and a few spurious words that are not related to the configuration itself, but are constraints of the typescript implementation. The second version is much simpler, easier to write and to parse using trivial string processing, without depending on any third-party library. It is beyond me how could any serious engineer honestly find any advantage in the first version.
  - aldonius 5y ago
    Well, if you want nesting on your map then you need something slightly more complicated than your second version, but I will grant that you could do something like: A.B value C value D.E.F value D.E.G value

the_arun 5y ago

How to add comments in json or TypeScript config files? Commenting is very much needed for any manually written code or config we keep in repositories.

strogonoff 5y ago

TypeScript is limiting for describing schema in my experience. Try requiring an array of certain length, for example. Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage. To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.

throwaway894345 5y ago
Dhall is too unfamiliar for most developers, IMO. Developers don’t want to have to grok an entirely new syntax just for config. I’m personally surprised no one has added type annotations to Starlark yet.
- strogonoff 5y ago
  Everything is unfamiliar, until one adopts it.
  - deleted 5y ago
    [deleted]
  - throwaway894345 5y ago
    I didn't think it needed to be explained, but not all things are equally unfamiliar nor equally easily adopted.
- mattw1810 5y ago
  The Rust implementation has it as an experimental extension (https://github.com/facebookexperimental/starlark-rust/blob/main/docs/types.md https://github.com/facebookexperimental/starlark-rust/blob/m...)
  - throwaway894345 5y ago
    Oooh, very cool! I'll check it out and maybe port to Go.

ravenstine 5y ago

My only real criticism is that I believe it's generally an anti-pattern to have configuration that necessitates DRY. It has essentially the same problems as OOP inheritance. Because configuration isn't and shouldn't be an application in and of itself, avoiding potential issues and misdirection from avoiding DRY in this case seems like the right tradeoff to having to use multi-select to change the config format in your IDE.

dqpb 5y ago
Cuelang enables DRY while forbidding inheritance.

intellix 5y ago

YAML is beautiful when it's simple and horrible when you need to dynamically generate it. The indentation...

pramodbiligiri 5y ago

Anyone have experiences to share about Dhall (https://dhall-lang.org https://dhall-lang.org)? On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"

junon 5y ago
Sorta. I evaluated it a long time ago and decided against ever touching it due to its remote import functionality. Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests.
- strogonoff 5y ago
  > Apparently it's crazy to think that your configuration files shouldn't be issuing HTTP requests. Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
Reitet00 5y ago
Yeah, I've used that a year ago or so. Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc. Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
- ReleaseCandidat 5y ago
  > no bindings to various languages. There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.html#language-support https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...

ReleaseCandidat 5y ago

Great idea. And because you need many constants that often change in such a configuration file you could put that all in another file and include that in the 'real' configuration file. Why not call that, hmm, `config_config.txt`. But as plain text is a bit hard to parse, we could give that file some structure, so that it is like, well, another markup language, just without markup and easier to read and write.

sbr464 5y ago

I use TypeScript and like the ecosystem. For broader use, I would like to see native support or an official extension for creating dependent types or specific constraints. Things like number ranges, or the basics from JSON Schema, etc.

tantalor 5y ago

Using programming languages for configuration is a bad idea.

bryanrasmussen 5y ago
why? It can't be because then if something goes wrong you need to debug your configuration, because you need to do that often enough with a non-programming language format for even less power offered.
- krapp 5y ago
  If your config is so complex that it needs "debugging" then it's too complex. The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability. If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum. And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
taeric 5y ago
Emacs would like a word. :)
latchkey 5y ago
Static declarations without a reason are equally bad.
inoffensivename 5y ago
gradle has entered the chat
- 5y ago

sirtimbly 5y ago

The line between configuration and code is always a little blurry. There are some projects where using a .js file for the "configuration" of another operation is incredibly useful. TS makes even more sense in many of those situations. Lots of configuration files require pre-processing already, like merging shell env vars in with textutil or something like that. So, I wouldn't dismiss this opinion outright.

jayd16 5y ago

Kind of interesting but now I need to run arbitrary code to read a config?

taeric 5y ago
To be fair, you already have to execute arbitrary code to understand what config is doing. Where execute is to sequence a series of effects that are described in a file.
- jayd16 5y ago
  >To be fair, you already have to execute arbitrary code to understand what config is doing. That's not arbitrary code. That's my compiled, tested and verified code.
  - taeric 5y ago
    All code is arbitrary code if you are paranoid enough. :)

richeyryan 5y ago

I feel like the AWS CDK is a great example of this working well. Rather than puzzling over what config options are valid, I get autocomplete and compiler errors when I insist on using an incorrect option. You also get stuff like deprecation warnings showing up in your editor. It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.

Lucasoato 5y ago
This is so cool, I feel like other similar projects, like k8cdk, are going in the same direction for that reason. The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.

Shadonototra 5y ago

I can't believe what i am reading...

crate_barre 5y ago
JS people gonna JS.
- npsimons 5y ago
  > JS people gonna JS. Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.

realrocker 5y ago

Great little writeup ! After mangling YAML, HCL, JSON for years as an ops engineer, I have come to the same realisation. In fact, I have put this into practice in production pipelines by using: jkcfg[1] for the last couple of years. Two data points: 1. Zero developer support contract rate around YAML syntax and templating issues 2. High number of contributions in our private typescript configuration library from developers. Using typescript as an ops frontend has made operations a lot more approachable to folks. Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration. [1] https://jkcfg.github.io/#/ https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg https://github.com/dxcfg/dxcfg

nsm 5y ago

I’m not on board with using a language that only has a single implementation and is complex enough that implementing others is difficult. Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.

pbalau 5y ago
If you can afford to wear only one hat, then sure, use one special language for configuration. But I can't, I am the entire infra department and I still need to do product and framework work. Not having to switch context, language wise, is a very nice thing to have.
tn890 5y ago
Honorable mention for Cue[0]. [0] https://cuelang.org/ https://cuelang.org/
- Kinrany 5y ago
  CUE does look simple enough to be implemented in every language. I wonder if there's anything preventing that. Perhaps there's space for a CUE-like language with an even simpler spec.
throwaway894345 5y ago
TypeScript has at least 2 implementations: tsc and Deno, but I take your point and agree. That said, most languages probably have a JS runtime, so a basic implementation would be ts->js->interpreter. The ts->js is presumably pretty easy.
- junon 5y ago
  I don't think Deno actually re-implements typescript, does it? Seems like a lot of wasted effort. They don't even use Typescript internally.

eyelidlessness 5y ago

I have used TypeScript for config in a few projects, with node-config[1], and it’s substantially better IME than JSON/YAML/etc. Yeah it’s unconventional to write configs in a general purpose language, and I understand why that would be undesirable. But having configs validated at compile time is great at preventing bugs. I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic. 1: https://github.com/lorenwest/node-config https://github.com/lorenwest/node-config

pphysch 5y ago

My money is on https://cuelang.org https://cuelang.org in this space. As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.

s_gourichon 5y ago

Dhall seems to gather some praises, and not fall into the pits mentioned here. https://news.ycombinator.com/item?id=17523623 https://news.ycombinator.com/item?id=17523623

wyager 5y ago
Dhall is awesome. Only downside is that the included formatter doesn't preserve comments. Super well-designed and with interesting features like semantic hashing.

zffr 5y ago

Apple takes a similar approach for swift. Package.swift files run real swift code and export an object that specifies the package’s configuration. In practice I think it works fairly well.

progre 5y ago

TypeScript needs to be compiled, right? So what use is a configuration file if I need to run it through the build pipeline? The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.

klodolph 5y ago
The fact that it is compiled does not mean you need to run it through an additional step in your build process. In practice, it would just mean that you include TypeScript in your package, which can either be used as an executable or as a library. It can be completely transparent to the user. You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
nverno 5y ago
> To use this approach today requires that your project also be written in Typescript. It also means you’ll need to forgo a separate flat-file for your configuration. You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
dragonwriter 5y ago
> TypeScript needs to be compiled, right? Yes. > So what use is a configuration file if I need to run it through the build pipeline? Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library. > The point of having configuration is to gain the ability to configure software without rebuilding in my mind. There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
handrous 5y ago
I think we got here because everyone's mixed up describing a state with describing the process to describe a state. Or, configuration versus scripts that apply configuration. Most of the popularly config-as-code systems mix the two up, including Ansible, my favorite one. You end up with "config files" that are, all at once in the same file: 1) configuration, 2) templates for configuration, and 3) programs that modify config when they're run. I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh. So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.

pstch 5y ago

Is evaluation lazy in TypeScript ? For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules. Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function. I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.

richeyryan 5y ago
Typescript is definitely eagerly evaluated unless you were to use a first class function to defer execution

ThePhysicist 5y ago

I think parsing YAML or JSON into typed structures is the easier way to go. I e.g. do that in Golang using a little form validation and coercion library I've written. The end result is a nested, strongly typed data structure. Here's an example: https://github.com/iris-connect/eps/blob/master/settings.go https://github.com/iris-connect/eps/blob/master/settings.go (the accompanying form validation configuration: https://github.com/iris-connect/eps/blob/master/forms/settings.go#L78 https://github.com/iris-connect/eps/blob/master/forms/settin...). Hashicorp has some libraries that do similar stuff, I wrote my own though since I need it quite often and wanted to have full control over it. The library is just a few hundred lines of Golang code (https://github.com/kiprotect/go-helpers/tree/master/forms https://github.com/kiprotect/go-helpers/tree/master/forms). Undocumented as of now as it's nothing I want to explicitly share with the world. In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.

melolife 5y ago

I've done a similar thing with Scala in the past - ship the compiler jar and then compile your configuration (which is just an abstract class implementation) as part of the startup process.

steeve 5y ago

Starlark or Cue, but not a general purpose language please. Side effects in a configuration language is an atrocity...

IshKebab 5y ago
It's possible to sandbox most languages, and with some work you can probably make them deterministic too. Here's an example: https://github.com/jkcfg/jk https://github.com/jkcfg/jk That beats having to learn an entirely new language.

fay59 5y ago

Probably important to mention that at the moment, TypeScript as a configuration format means anyone who writes a configuration blob for you has arbitrary code execution. This is rarely an issue in practice, though.

cphoover 5y ago
It also means your configuration must be transpiled into JS first before execution unless run through a typescript runtime interpreter.
mejutoco 5y ago
This could be solved by having some kind of sandbox (https://github.com/patriksimek/vm2 https://github.com/patriksimek/vm2), but I agree it complicates it. It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
- realrocker 5y ago
  One of such sandboxes is Deno.
  - cormacrelf 5y ago
    Not even Deno can solve the halting problem, unfortunately.
    - Zababa 5y ago
      No need to solve the halting problem, just put a timeout.
      - cormacrelf 5y ago
        If you use it by running a V8 isolate that transforms the exported config into JSON, then sure. But TS-based config is largely for people using TS elsewhere. And in that case: // this returns in 0 ms export const config: any = { get value() { while (true) {} } } as const; // accessing config.value runs forever console.log(config.value); For a less unrealistic perspective, if you write enough configuration in a Turing-complete language, you will eventually be tempted to use the full power of the language. You may find yourself having to write tests for the config, which might be e.g. parsing other files, making HTTP requests, querying the filesystem/OS and more. If you're doing that, you are no longer really writing configuration. The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format. ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.

nly 5y ago

Protocol Buffers text format ?

jonny_eh 5y ago
Protobuffs remain very underrated outside of Google.
joshuamorton 5y ago
Composability in configuration is really important beyond a small scale. If you're running a replicated server in n locations, it's far better to keep one config file that has some form of conditional logic based on location than to maintain n textprotos that need to be kept synced.

deleted 5y ago

[deleted]

11235813213455 5y ago

or json-schema with jsonc or js files

ljm 5y ago

I've only worked with cdk8s, but the experience of that alone was enough for me to regret it. It's not that cdk8s is bad, per-se, it's that it turned an otherwise declarative kubernetes config into something that was much harder to grok in the first instance. One of the arguments for TS in the post is that it can encourage DRY. It was DRY that made this TS-based kubernetes config so hard to follow, because simple repetitive statements got abstracted into parameterised functions, or classes to inherit from, or other things that are resolved dynamically at runtime. Not to mention that you're also going to start mixing in random dependencies from NPM to help abstract things further. This is great for application development but not so great for pure configuration. A language designed for config, like Dhall or CUE, can give you much of what a typed language offers without letting you go crazy with a full-blown language runtime.

tetha 5y ago
Those were my concerns as well. Your config file can stay simple, sure. But I'm pretty sure it won't. And from some years of experience with configuration management systems, configuration creation can accrue a surprising amount of complexity over time. Mostly because unlike the code they support, a config management system has to support many versions of the code and multiple versions of a configuration at the same time.
dogleash 5y ago
Sounds like the Configuration Complexity Clock http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html http://mikehadlow.blogspot.com/2012/05/configuration-complex...
- photojosh 5y ago
  Oh I enjoyed that, thanks. Shared around at work. Likely controversial, but I really like how Django does its settings. I think because Python code can start off just looking like a .ini file with variables and assignments, and get more complex only as required. On the other end of my work, I'm a fan of having embedded firmware apply to as many devices as possible, and then specialise with configuration settings from EEPROM.

escherize 5y ago

Good to see people coming to the same conclusions we have had for a while using Clojure. There's an up and coming runtime data validation framework called Malli that I use for these sorts of files. You can get many of the benefits but also run custom validations (dictate how two keys in a map relate to eachother, or force a be of a certain shape instead of just a string). Here's the example from the script: [0] [0] - https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%20%3Atimezone%20%22America%2FNew_York%22%0A%20%20%3Alog-level%20%22info%22%0A%20%20%3Aenvironment%20%22local%22%7D&schema=%5B%3Amap%20%0A%20%5B%3Alocale%20string%3F%5D%0A%20%5B%3Atimezone%20string%3F%5D%0A%20%5B%3Alog-level%20%5B%3Aenum%20%22trace%22%20%22debug%22%20%22info%22%20%22warn%22%20%22error%22%20%22fatal%22%5D%5D%0A%20%5B%3Aenvironment%20%5B%3Aenum%20%22local%22%20%22dev%22%20%22staging%22%20%22production%22%5D%5D%5D https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%...

SavantIdiot 5y ago

Except .... (...No one asked, but here are my $0.02 ;-) You need typescript in your environment to use typescript! The four formats the OP listed can be read by anything from Ada to EXCEL. But typescript can only be ready by a TS build flow. In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.[1] FOR GENERAL PURPOSE: big no. BUT: If this is purely for TS projects where the tooling already exists, then OK. I'd use it. EDIT [1] I'm referring to the article's early suggestion to run JSON through JSMin to support comments. So transpiling is more apt, not compiling.

__ryan__ 5y ago
> In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler. Does JSON need a compiler?
- a_lost_needle 5y ago
  It does not. Easily implemented with combinators the whole way down.
  - SavantIdiot 5y ago
    Correct, but the very beginning of the original article proposes running JSON through JSMin to support comments. So in this context, a "compiler" is being proposed (or transpiler to more modern). My comment isn't clear.
    - a_lost_needle 5y ago
      Still easily implemented with combinators the whole way down. It’s building an ast, that’s it. Once you have a tree, you can do whatever you want with the result before passing it to a compiler for a language. Here though, just serialization stuff.
      - SavantIdiot 5y ago

strifey 5y ago

I'm curious what folks who agree with this think about setup.py then? There's been a push to move to setup.cfg to specifically avoid the fact that python libraries run arbitrary code on install.

nonameiguess 5y ago
The problem with setup.py is different. Because it can run arbitrary code, package writers go crazy and import extra dependencies, but you specify your package dependencies in setup.py, so without some out-of-band way to tell your users that the setup.py file in itself also has dependencies, package installation on machines other than your own might fail unless by sheer luck your users already had the packages being used in setup.py. That problem doesn't apply to arbitrary configuration, but only to configuration files that represent package build and installation metadata. You do see this issue with things like Gradle, though, where your build.gradle file can represent the dependencies of the package being built, but the build file itself may have dependencies since you can run arbitrary Groovy code and import any package that can be contained in a jar file. It gets around this because Gradle itself can just download those dependencies before trying to execute the build script, which would be perfectly possible for a real Python build system, but not when setup.py is just being executed by the Python interpreter, which doesn't do dependency management and expects imports to already be available in the system import path.
- nonameiguess 5y ago
  Heck, for what it's worth, even Make has this issue, as people quite often write Makefiles assuming you have a bunch of tools installed that may or may not be part of the POSIX specification. I wish I could find it now, but somewhere in the GNU Make guide book is a list of more or less "approved" utilities you're supposed to be able to safely assume are installed on any GNU based system that you can use in a Makefile, and you're not supposed to go beyond that.

whirlwin 5y ago

The idea of allowing logic from a programming language in a config format is.... at best very horrifying...

vander_elst 5y ago

What would be the problem with using something like protobufs? To write the configuration, you can use whatever language you want, what matters are the outputs anyway. The outputs are stored by automation in text format so that they can be better inspected. The programs that use such configuration have the advantage of knowing the schema of the configuration and directly parse it avoiding possible typing problems. The nice thing is that you can use the same language to write the programs and the configuration.

mikojan 5y ago
protobuf may very well be total overkill. What you wanted to do was strongly type your configuration. What you get is a complete RPC layer. This might not even be warranted when inter-process communication is a primary concern. After all, there's nothing inherently wrong with using whatever data transfer mechanism your standard lib provides. It's not just protobuf though. Almost every data format for serializing structured data suffers from pushing a jungle on you when all you asked for was a banana. The least invasive format I know of is FlatBuffers[0]. Doesn't push RPC on you. Allows you to parse a message even if you do not know this specific message's data structure in advance (like JSON). [0]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- vander_elst 5y ago
  1. As said in the original comment, it doesn't necessarily have to be protobuf, but just a data serialization system. 2. Protobuf per se is just a serialization system, it does not push RPC "Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler." [0]. It can be used in combination with RPC systems, but does not have to. [0] https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers

beebeepka 5y ago

Typescript is perfect for stuff like this because it's so much less verbose than most things out there As a bonus, you might not need validation and schemas, too.

shadowgovt 5y ago

I like the idea of the IDE assisting with the schema-correctness of the config (which you get for free with TypeScript coupled with any competent IDE configured to use the typechecker to validate the code). ... but as a rule of thumb, if your config language is Turing complete, you're setting yourself up for future pain as complexity of config grows with time and someone makes the bad decision to take advantage of that Turing completeness...

harunurhan 5y ago

Not typed support but I find jsonnet [0] pretty useful when you have tons of configuration files. [0] - https://jsonnet.org/ https://jsonnet.org/

eb0la 5y ago

Really interesting but, sincerely, I would rather use sqlite. I know: It's a binary format. And you need to add a library to read it. BUT - Your configuration _has_ data types. - You can have a copy with an old config within the file easily (create table backup_config as select * from actual_config; ). - The file is small. - Encryption is built-in if you need it. - And it's programable

fourseventy 5y ago

Personally I wouldn't use Typescript for config. It breaks my #1 dev rule which is KISS. Keep it simple stupid. Simplicity is orders of magnitude more important than other things like DRY or config schema.

Slasher1337 5y ago

I've really come to prefer json5 as a config format: https://json5.org/ https://json5.org/ It's basically Json with comments, trailing commas, and unquoted object keys (and some other minor things). Much nicer than real json as a config format, but still simple and declarative, not requiring an interpreter or being turing-complete.

yayr 5y ago

Should this be a future ECMAScript enhancement or is the better route for browsers to natively support TypeScript?

veidr 5y ago

I think this is kind of a no-brainer, if your project is already based on TypeScript. Not just for configuration, but for things like test fixture data, and even data assets like i18n translations, or basically anything that projects might commonly use JSON for, internally. It's just so much easier to create, edit, manage, and deal with in TypeScript. But it is hard to imagine TypeScript-as-config taking root in projects that don't already use TypeScript.

vendiddy 5y ago

I think typescript as a config format alone may get too complex. But json backed by a typescript interface definition could be a useful middle ground. You're still writing json but you get nice code completion and realtime validation of the config file.