14 ms·
What is wrong with TOML? (2019)
- tipiirai 3y agoAgree with the author. I have used TOML when configuring Netlify and it's super awkward.
- baq 3y agoEagerly waiting for a Yet Another Human Readable and Writable Graph (Which Is Mostly a Tree But Not Always) Serialization Language With Built In Schemas and Pure Functions, Maybe made with <3 for humans. Actually, that, but without sarcasm. YAML is crap. TOML is crap. JSON is crap. INIs are actually fine for flat lists but major crap otherwise. Anything Turing-complete is completely inadequate crap for the purpose.
- ducktective 3y agoNickel[1] maybe? Though I'm not sure about "for humans" part :) [1]: https://github.com/tweag/nickel https://github.com/tweag/nickel
- RedNifre 3y agoTry edn: https://learnxinyminutes.com/docs/edn/ https://learnxinyminutes.com/docs/edn/
- masklinn 3y agoI like edn but I’m not convinced it’s great for configuration. First off, it shares the desire for extensibility with yaml (in the form of tags), which is a giant trap. And then it’s quite syntactically noisy.
- dkersten 3y agoI love EDN (especially the integrant extensions) for configuration of the stack — that is, dependency injection and so on, the developer-focused configuration. For user-facing configuration I still favour TOML. I think it’s a bit application dependent, sone applications (eg nginx) have complex configuration needs and for that it makes sense to use something more sophisticated, but for many user-facing config settings, a simpler-TOML would be a great fit. Basically just some basic key-value pairs that can be collected into groups. As the article states, the parsing types should perhaps be enforced by the parser not the written config.
- jmkr 3y agoWhy do you love edn? Specifically I also like edn, but I always felt like my use of it was glorified json. I did like integrant, at least I felt like I understood it, unlike component. There were two things about edn that made it seem better than json to me, tagged elements (and readers), and symbols. I don't remember exactly my use case but I used symbols in edn to something like namespace-resolution for multimethods. It was something like including a file in a classpath or loading a file, dev vs prod kind of config.
- dkersten 3y agoI suppose mostly superficial reasons such as liking the keyword syntax better that using string keys in json, but also the richer set of data types (like sets), although that mostly doesn’t matter in practice, but it is very nice when using Clojure since the Clojure types just work. As an interchange or storage format it’s also nice to be able to store Clojure types and get them back the way they were out in, which if using json you would lose some information. Mainly it’s just personal taste and no deep reasons.
- dkersten 3y agoI especially love the Integrant[1] version which uses the E from EDN (extensible) to add references: {:adapter/jetty {:port 8080, :handler #ig/ref :handler/greet} :handler/greet {:name "Alice"}} [1] https://github.com/weavejester/integrant https://github.com/weavejester/integrant
- eternityforest 3y agoToo many features that can't be represented in standard programming language constructs.. JSON-alikes are opinionated. They don't let you do stuff that requires a plugin for the parser to work with
- conradludgate 3y agoMaybe kdl[0]? It's a document language somewhere in between xml and yaml without all the crap of either IMO [0]: https://kdl.dev/ https://kdl.dev/
- alpaca128 3y agoI use KDL in a (not too complicated, yet) config file of a project and I like it a lot. Tree structure with attributes like XML but with less syntax than JSON. Nothing redundant but has basics like comments.
- jmaker 3y agoDhall is superb, https://dhall-lang.org https://dhall-lang.org
- 37469920away 3y agoThanks for that, it is interesting. Is this actually something I want to see in a configuration script? Why not just use a scripting langauge and be done with it? I wonder if the safety features can't be replicated with rigorous testing of say python as config scripts instead of learning yet another programming language? https://prelude.dhall-lang.org/Text/concatSep.dhall https://prelude.dhall-lang.org/Text/concatSep.dhall I think this is the key fulcrum for me: "config is code", sure, but not the same kind of "code". It -is- compelling to argue for statistically deterministic config code but my practical objection here is 'can we arrive at same safety using testing with a known language?' Writing this has made consider whether configuration should be conceptually looked at as a database instead of "code". How many people even know how e.g. postgres stores its tables and why would modulo some performance niche would you care anyway? It seems configuration management is a graph db query and update matter. Standardize on configuration query language (if necessary) and stop worrying how the damn thing is represented by the config management tool.
- jmaker 3y agoDepends on your requirements. If your config complexity is getting beyond manageable, the benefit of something more reliable is apparent. Type safety clears lots of common bugs no test suite would be certain to filter out. How complex should your tests get? Do you want to take on that responsibility or rather delegate it to something that provides you certain guarantees? It’s all very subjective in the end. I run my configs mostly as YAML in Consul and Vault, sometimes in Spring Cloud Config with a git backend. This way I have dynamic config evolution. But I prefer to generate those yaml files from Dhall to avoid unnecessary bugs. After years with Haskell, the syntax is very natural, too. As for Postgres internals, they do matter if your data set keeps growing. Xkcd covered standardization. YAML and JSON ASTs are graphs, YAML not necessarily a tree. JSON extensions also support references. As for the ops side, YAML has become a de facto standard, HCL is used with the HashiCorp tools. Nix has its own language. It’s not about how it’s represented but how you express dependencies across config key nodes. It’s good to avoid repetition and have a syntax linter, a compiler even better. Small static configs are amenable to querying and writing. But you need to separate the writing from the querying the configs. With Dhall you write code to generate the actual config, whether as a Dhall AST or exported to YAML or JSON, with certain correctness guarantees upfront.
- larschdk 3y agoINI-files are crap too, as there is no standard. Every program has it's own dialect, and automating around them can be a pain.
- rapsey 3y agoINI files are toml
- incrudible 3y agoExactly, so just use JSON for configuration. This will upset the people that have not yet learned to get over it, but you really need people that got over it.
- BiteCode_dev 3y agoI do this, but regularly get annoyed with the date, comment, trailing comma, mandatory key quotes, etc
- retzkek 3y agoMaybe you'd like jsonnet: https://jsonnet.org/ https://jsonnet.org/ I find it particularly useful for configurations that often have repeated boilerplate, like ansible playbooks or deploying a bunch of "similar-but" services to kubernetes (with https://tanka.dev https://tanka.dev). Dhall is also quite interesting, with some tradeoffs: https://dhall-lang.org/ https://dhall-lang.org/ A few years ago I did a small comparison by re-implementing one of my simpler ansible playbooks: https://github.com/retzkek/ansible-dhall-jsonnet https://github.com/retzkek/ansible-dhall-jsonnet
- paulddraper 3y agojsonnet is turing complete. It's configuration with a build step. Which is an option, but that's pretty different than JSON, YAML, TOML, etc.
- BiteCode_dev 3y agoIf I have to use a better format, I will use toml or cue. The whole point of the parent is that JSON is the Pareto solution, which I agree with. But it does grind my gear.
- baq 3y agoJSON is the runner up on the top crap list right below Windows Registry.
- remram 3y ago
- BiteCode_dev 3y agoLet'add another one: CUE https://cuelang.org https://cuelang.org
- ianburrell 3y agoI think it is important to distinguish between human-readable serialization format and human-readable config language. I wish the distinction was made more explicit. JSON is the champion serialization format. But it is hard to edit config files with missing comments and strict quotes and commas. JSON is great for the API, but configs should be converted from something nicer to JSON. I wonder if it would be good to make the conversion explicit so any format could be used.
- twic 3y agoProperties files have been fine, i guess since 1995.
- paulddraper 3y agoHOCON (But extrapolating, I'm guessing it goes into your crap category.)
- WorldMaker 3y agoI'm surprised no one has thrown in raw S-Expressions, yet, in this thread. That's an HN perennial favorite. Great at trees, decent at graphs. You can easily go Turing complete or not on the whim of any Lisp at hand and the hammer and forge of macros until your heart's content.
- tikhonj 3y agoI interned at a company that used an s-expression format as an alternative to JSON and it was great—much better than any mainstream format for config files and human-readable representations for API data. It was great at representing structured data (including tagged variants, so full algebraic data types) and even pretty good for text markup. It also has best-in-class editor support thanks to paredit :) The only real downside is the hassle of convincing people to use something weird and non-standard, which is really more a problem with people than with the format.
- eternityforest 3y agoINI is really nice. Anything you can't represent in it, is getting towards not quite just configuration. I think my ideal format would be INI, with tagged headings so you could do [md:README] and stick a multi line string section in there, or use something like [comment:Module Attributes] to make an embedded docs section. Either that, or just some other method of embedding multi line keys. Maybe HTML-like tags, so that inside a heading you can do <my-key> value </my-key>
- PennRobotics 3y agoI have a beef with YAML/TOML/JSON as an inline front matter (header) format for SSG (static site generator) posts, but that's because if I want to save a mostly Markdown SSG post on Github... 1. the post is previewed with Github's dialect of Markdown and not the SSG's (and definitely with none of the inline configuration applied) 2. the preview is still a Markdown document, so you get no benefits of syntax highlighting or auto-formatting w.r.t. the config header (short of opening in Vim and explicitly declaring the filetype or temporarily changing the extension in Github's editor) 3. you HAVE to put trailing spaces in the YAML/TOML/JSON or the preview pane crams everything into an unbroken paragraph 4. there's not a quick preview of how the configuration will parse, just specific workflows (live update, compile single page) that you can test and then modify as needed. This is either in the rich online editor or your own machine and will require console commands and a browser window 5. I still have to know all of the modifiable attributes as well as defaults, which will be in a separate document and probably not in a Ctrl+Space dropdown ----- For point 5, it would be nice if configuration formats had completions for common editors and/or their own scripts: * "generate big config file with all possible keys and default values" * "condense modified config file so it only contains non-defaults (and hope the schema doesnt change hahaha)" * "suggest a valid fix for a currently invalid config file" ----- Sure, this all isn't a direct criticism of TOML, but inlining configuration is a great-to-okay idea that is simply poorly executed. It is extremely unfriendly to non-technical users; I can fully understand why someone would pay a few hundred a year for a WYSIWYG templated website builder to just handle everything.
- globular-toast 3y agoLiterally every config file format is terrible in some way or another. The best configs are executable and loaded into a dynamic runtime. Emacs and Airflow are good examples of this. But I definitely strongly prefer YAML to TOML. It's just makes a lot more sense to me and it's a huge shame that PyPA went with TOML which is so un-Pythonic. I preferred setup.py. StrictYAML is a really good development that I wasn't aware of, though.
- baq 3y agoFlat is better than nested. I'd argue that's enough for TOML to be more pythonic than YAML.
- globular-toast 3y agoBoth TOML and YAML support nesting. TOML simply looks flat even when it's not, so it's the worst of both worlds. In any case, it's not the format that is nested or flat, it's the content. Python itself supports nesting. The zen of Python merely says you shouldn't use the nesting when flat is an option.
- tomohawk 3y agoAgree with author. Use yamllint with YAML to avoid commonly complained about problems.
- bschwindHN 3y agoI'm okay with TOML for something like Cargo configuration, I don't enjoy it for much else though. I always comment the same thing on these sorts of discussions - JSON5 has been really nice to work with if you can fully control all consumer applications of it (since there aren't great libraries for json5 in every language). Certainly nicer than the hellscape that is YAML.
- masklinn 3y ago> I'm okay with TOML for something like Cargo configuration, I don't enjoy it for much else though. Which seems fine given TOML was designed as a stricter and more reliable INI, not a half-assed programming language. Can’t say I’m unhappy to know that when I see a toml file, it’s probably going to be pretty simple (exactly the opposite of the dread I feel when I see a yaml extension).
- paulgb 3y agoI think that’s exactly it. TOML is bad at deep hierarchies, but that’s a good thing; it prevents people from trying to model a whole damn AST in it, which is the direction that YAML seems to go.
- bryanlarsen 3y agoToo bad this isn't a high-level comment that can be upvoted and float to the top.
- eviks 3y agoThat Cargo config with [Lib] But also [[bin]]
- goku12 3y agoIt really depends on what you're configuring. I like TOML for most cases since it has the right balance of simplicity and expressiveness. JSON could in theory support everything that TOML can - but it can be jarring to look a large config file with just brackets/parenthesis to demarcate blocks. Even YAML finds its niche applications. I don't think JSON or TOML are good substitutes for YAML in case of Kubernetes. Kubernetes needs a single format that can act as a config language (in case of spec) and as a serialization format (in case of state). YAML fulfills that without being overly verbose. We probably shouldn't get too attached to a config language.
- boredumb 3y agoI use TOML for my cargo files quite a bit and to be frank about it, I've never even considered using TOML for anything else. With that said, I'm using YAML for kubernetes and I couldn't imagine TOML being any more obnoxious.
- arp242 3y agoThe (upcoming) TOML 1.1 will alleviate some of this; that example document could then be written as: [params] profile = { name = "Gareth", tagline = "..", } contact = { enable = true, list = [ {class = "email", icon = "fa-envelope"}, {class = "phone"}, ] } The whole business with the syntax typing has no "one correct way" to do it. No matter what you do it will cause problems and headaches for someone at some point somewhere. > Dates and times, as many more experienced programmers are probably aware is an unexpectedly deep rabbit hole of complications and quirky, unexpected, headache and bug inducing edge cases. TOML experiences many of these edge cases because of this. Eh? In the original text it links to three issues to back this up: That first issue it links to is "failed to parse long floats like x = 0.1234567891234567891" and the third is a feature request for hex values (v = 0xff, not even a bug report). That has nothing to do with dates? The second issue did relate to dates, but was just a simple bug, not an "unexpectedly deep rabbit hole of complications and quirky, unexpected, headache and bug inducing edge cases". This just seems repeating a tautology. I maintain a TOML implementation that sees some reasonable use. Dates have not been a huge source of bugs, confusion, or other issues. All you need is to be able to parse RFC 3339 style dates (and some things derived from that), which is usually just calling strftime() or whatever your language has for this. I do think TOML had some bits I wouldn't have added (not dates though), but the feature sets and complexity of TOML and YAML and not even comparable; it's like comparing Iceland (pop: ~300k) to Ireland (pop: ~5M). Yes, they're both islands and both are small countries, yet the scale of their "smallness" is just completely different.
- hddqsb 3y agoThe inline table syntax is awesome! Details: https://github.com/toml-lang/toml/blob/main/toml.md#inline-table https://github.com/toml-lang/toml/blob/main/toml.md#inline-t... https://github.com/toml-lang/toml/pull/235 https://github.com/toml-lang/toml/pull/235 https://github.com/toml-lang/toml/issues/516 https://github.com/toml-lang/toml/issues/516
- mardifoufs 3y agoWill python support the new version for pyproject.toml?
- NoboruWataya 3y agoIt is the least bad configuration format I have found. Granted I have only ever used it for fairly simple projects. But every config format is plagued with issues. A bit like programming languages, the fundamental problem is that they need to be easily understandable by both humans and computers which is an impossible problem to truly solve for any non-trivial use case. For example, TOML is criticised for verbosity but a lot of the abstractions that are used to implement DRY in a programming context may make the configuration confusing and unintuitive for non-programmers.
- AndyKluger 3y agoHave you looked at NestedText?
- IshKebab 3y agoJsonnet or JSON5 are much better than TOML or YAML. Both are much easier to read, and don't have the footguns of YAML or StrictYAML. I would generally say JSON5 is more appropriate because it is simpler, but Jsonnet does have some neat features and its IDE support is much better.
- hot_gril 3y agoAt least general programming languages like Python or JS are well-understood by many programmers. More than makes up for not being as specialized as a DSL.
- pgt 3y agoEDN (Extensible Data Notation) is a subset of Clojure: https://github.com/edn-format/edn https://github.com/edn-format/edn It is: - Streamable - Extensible - Whitespace-insensitive, but there are formatting conventions for readability
- sundarurfriend 3y agoThe first and the last ones, at least, are tradeoffs where TOML made the right decision for most users. Not being DRY is a good thing in a config file - it makes it much easier to understand and work with just one section of the file (which is what you most often want to do), because the context information is right there without having to jump around and figure things out. And whatever the downsides of syntactic typing are, requiring a schema file to go along with your config file is far more of a downside; it's one more point of potential failure, another thing to maintain and sync up and keep in your head, and not worth it for most use cases. And that's the crux of it: it all depends on what you need from your markup language, what your use case is, today and through the lifetime of your project. "What's wrong with TOML" makes much less sense as a question than "What's wrong with TOML(/JSON/YAML/etc.) for _this project and its needs_".
- hitchstory 3y agoHi, I'm the author of this piece. Thanks for your comments. >Not being DRY is a good thing in a config file - it makes it much easier to understand and work with just one section of the file (which is what you most often want to do), because the context information is right there without having to jump around and figure things out. If the contextual information is relevant that's true. However, syntactic noise of the form of lots of [s, ]s and equal signs isn't necessarily relevant. A YAML file can exhibit identical information with fewer characters and that makes the files easier to read and maintain - especially if they are information dense. >And whatever the downsides of syntactic typing are, requiring a schema file to go along with your config file is far more of a downside; it's one more point of potential failure Schemas aren't technically required by strictyaml provided you're happy mapping everything to a dict, list and string, but they're recommended because they make it much easier to prevent something from going wrong and it means you can directly use the types you were expecting. Schemas in config files are equivalent to static types that generate compiler errors in a program. If you can use them, it's an easy way to get your program to fail fast on invalid input and save on debugging time. If you don't have a schema and some invalid data gets put into your config file, instead of getting an error that says "didn't expect key "ip addresses" on line 14" you tend to get a really cryptic error a bit later on when your program tries to get a key from a dictionary that doesn't exist. This is an example of the principle of https://en.wikipedia.org/wiki/Fail-fast https://en.wikipedia.org/wiki/Fail-fast design.
- dagw 3y agoOne of the more interesting config formats I've come across was an application that used an Excel file. Once you get over the horror of such a terrible decision, it was actually a quite interesting choice that allowed a fair few advantages. First of all each config subcategory was on a separate sheet making easy to navigate and find what you where looking for. You could use formulas to relate different config options (If you wanted A to be 20% of B you just set that in a formula). You could use drop-down for fields where there were only a limited number of valid values. You could include as much comments and documentation as you wanted (including diagrams and images) as long you only wrote in unused cells. And finally, my favourite, when configuring colours, instead of typing in RGB or hex values you simply changed the colour of the cell to the colour you wanted. Now I would obviously never ever recommend doing this, but it was certainly an interesting and eyeopening experience.
- btreecat 3y agoThat's just begging to be an actual database. Imagine if you said everything with SQLite instead of Excel, and all of a sudden your just talking about structured config in a database. Not new, not crazy, and generally a decent practice.
- michaelbuckbee 3y agoNow I just want a spreadsheet style front end to SQLite and I'm nerd sniped into trying to figure out how to do formulas (triggers I suppose).
- speed_spread 3y agoCongratulations, you're about to reinvent MS Access / DBase / FoxPro... 30 years later.
- michaelbuckbee 3y agoThat's an excellent point.
- Ygg2 3y agoSo replace TOML with YAML? Or strictYAML? That's like having an engine problem and replacing engine with Shuttle Thruster. Or Space X rocket engine. YAML imo will never beat JSON. The spec is so complex and full of edge cases that it will never reach simplicity and speed of JSON. I implemented a spec compliant parser and it gets around 20 MB/s. In JSON I can get to at least five times that.
- bemusedthrow75 3y ago> YAML imo will never beat JSON. The spec is so complex and full of edge cases that it will never reach simplicity and speed of JSON. Ironically, all JSON documents are valid YAML. YAML is actually a superset of JSON.
- Ygg2 3y agoIronically? Fact it's a superset means it has more states and can't be as optimized as JSON. To compound the misery YAML is whitespace indented with some extra bonkers rules. So even your JSON subset in YAML needs to obey it. Mix of flow and block makes both modes worse. However it also makes a lot of sense.
- Sankozi 3y agoYAML is mainly for configuration files. JSON sucks at being configuration file format, it is mainly used for serialization in HTTP requests. Those 2 have different use cases.
- grose 3y agoI think syntax typing is a good idea. There are a million ways to format dates and standardizing that saves a lot of pain. Having explicit syntax for types means it's easier to handle arbitrary data or data of an unknown schema, and without it you are forced to assume everything is a string or rely on heuristics which leads to divergent implementations. No semantic whitespace please. Can't tell you how many times I've seen something in GitHub Actions or similar get messed up because someone forgot a space. I don't think TOML is a perfect format and agree it's not great at hierarchies.
- hprotagonist 3y ago> StrictYAML, by contrast, was designed to be a language to write readable 'story' tests where there will be many files per project with more complex hierarchies, a use case where TOML starts to really suck. I can’t be the only one who feels this way; isn’t it this use case the thing that sucks? “plain text but not really” config formats that aren’t “code” but have special syntax but lack a debugger and handy IDE tools and you’re never really sure of what you’re doing … isn’t that the thing that sucks?
- masklinn 3y agoThat was also my first reaction opening the example, I’m perfectly fine with a half-assed programming language (embedding more programming languages) not working in TOML, I don’t think it works in yaml in the first place, and it’s exactly the sort of usual mess which makes me recoil at the sight of a yaml extension. This specific example is like somebody saw cucumber and went “I think this should be a lot worse, and in an ecosystem which doesn’t want to come anywhere near that too”. This would probably be half the size and actually comprehensible if it were just a pytest test file.
- hitchstory 3y agoHi, I'm the author of hitchstory. With the stories in YAML two new use cases (which are demonstrated in the example projects) are enabled: * Automatically updated how-to docs. I used to write these types of docs manually and if they existed at all they would ALWAYS get out of sync with the code and were painful to maintain manually. Now I have YAML files and a simple jinja2 template I can push out new markdown how-to docs on each new build - with snippets of JSON, screenshots from the app, whatever. * Tests that rewrite themselves. E.g. if I have a REST API test of the form "call API x and expect y blob of json", I don't have to actually write that blob of json into the test, I just write the code that produces it and run the test in rewrite mode so it updates the "expected JSON" field with actual json. I can then eyeball it and 20 seconds later it's part of the test and part of the docs. The productivity improvements from doing both of these things means that writing tests is cheaper so I do more of them. Having how-to docs for all scenarios is way cheaper so I now always have them. These use cases are impossible with pytest. They are impossible with cucumber. They would be too painful to maintain with regular (i.e. not type-safe) YAML and those stories have enough indents that they would be an epic unreadable mess of syntactic noise if they were built in something like TOML or JSON.
- rewmie 3y agoI think this is a glass-half-full view o TOML, when TOML itself was the one that added water to the glass to begin with. The main value proposition of TOML is to provide a concrete specification of a INI-type config language. INI is ubiquitous, but it lacked a spec, which led to a lot of wheels being reinvented. TOML fixed that. If a project needs convoluted config files, I'd argue the project is already broken. If TOML doesn't fit your needs, that's hardly TOML's fault.
- dystroy 3y agoI previously argued that TOML wasn't good enough in this blog post https://dystroy.org/blog/hjson-in-broot/ https://dystroy.org/blog/hjson-in-broot/ where I show an example of problem which frequently hurts my users and leaves them lost without even understanding that the problem is in how they wrote their TOML. I moved the configuration of several of my programs to Hjson. There are still problems but they're less puzzling. Hjson isn't ideal either but might still be the best configuration format we have today.
- throwawee 3y agoThank you for introducing me to Hjson. I've been using simple colon delimited lists which seem to be, hilariously enough, already valid Hjson.
- dystroy 3y agoA lot of formats are Hjson compatible, notably JSON, and also what users wrote thinking it was JSON but they forgot some quotes or had a trailing comma so the JSON parser refuses it while the Hjson one is perfectly happy.
- eviks 3y agohjson is indeed more H vs json You've mentioned in the blog that ": it's meant to be written by humans, read and modified by other humans, then read by programs", but is it possible for apps to (roundtrip)-edit those configs preserving all the human syntax intact? It's rather common for apps to e.g. have font size changed, but unfortunately also common to destroy human formatting in the process
- lr4444lr 3y agoI can't be the only person who thinks the monumental effort spent on config formats is bike-shedding. JSON is good enough for anything I've done. Not perfect, but no serious flaw that can't be fixed by just adding a simple app-specific post-process step that I will inevitably do for any other format anyway. JSONSchema gives us some typing sanity. Can we just move on already to more interesting problems? It's not like git fulfills every VCS wish I've ever had either, but I have to move on. Projects and libs that introduce new config formats that continually remake the wheel, whose quirks have to be learned, are not helping my net productivity. </rant>
- ihateolives 3y ago> JSON is good enough for anything I've done. I want comments in config files.
- skrebbel 3y agoJSONC is exactly that, JSON with comments. Works fine, eg typescript’s tsconfig file is JSONC and I’ve yet to find a problem with it.
- ok_computer 3y agoA horrible workaround I use is a blank redundant key and a leading // in my string to draw my eye to it. This only preserves the last comment in my python dictionary but I only use comments to work in the json file. { “”:”// comment here”, “Entry”:[-1,0,2], “”:”// next comment”, “Flag”:true }
- timmytokyo 3y agoAnd trailing commas on final list elements and object properties.
- bobbylarrybobby 3y agohttps://json5.org https://json5.org
- hddqsb 3y ago
- bemusedthrow75 3y agoI'm not sure it's clear from the article -- only the URL -- that the writer is also the author/maintainer of StrictYAML. This article has been written in a way that (most likely inadvertently) implies a measure of distance from StrictYAML.
- gorelyy 3y agoI also wasn't happy with TOML as an hierarchical ini alternative, so I made my own config format https://shoal.eelnet.org https://shoal.eelnet.org Don't judge me, as I don't use it anywhere besides personal projects)
- wink 3y agoThose examples are nuts. Who would ever write a config like this?
- anuraaga 3y agoWill go ahead and start off the config wars as undoubtedly the OP intends. FWIW my best config experience has been with HOCON via typesafe|lightbend/config in Java. The ability to compose environment specific defaults in a reasonable way just felt good. Of course /internal/config to dump the config was a necessity, but trivial, so I tend to be less sympathetic to DRY is not necessarily good arguments. Have been missing it in Go, unfortunately Java's ability to publish files within packages (that can be imported in config) was a key part of the UX that is missing in any compiled language I've seen.
- mike_hearn 3y ago+1 for HOCON, if you're on the JVM at least. We use it in our product with some extensions. Like you, I find it just feels good. For those who haven't encountered it before, HOCON is a superset of JSON so all valid JSON is also valid HOCON. Then it starts adding syntax sugar and useful features specifically for configuration files (the "H" stands for Human). We wrote a tutorial for our product, it has a slider you can move to see how JSON evolves into HOCON along the way: https://conveyor.hydraulic.dev/11.2/configs/hocon/ https://conveyor.hydraulic.dev/11.2/configs/hocon/ It's got some nice features. There's no "syntax typing", programs that use HOCON are thus very forgiving. Conveyor takes that even further, for example, anywhere you would normally need to specify a list of strings you can also specify just one string, it'll be wrapped automatically. There is a formal spec. It supports substitution, inclusions and it defines the semantics of duplicate keys which allows for refactoring of configs out to separate re-usable files. It has a nice clean look that gets out of your way. You can not only include files but also URLs. On top of that we add a few more features. If you need to express a list of strings then brace expansion is supported, i.e. foo = "bar-{1,2,3}" is equivalent to foo = [ bar-1, bar-2, bar-3 ] But probably the most important feature is hashbang includes. These allow you to include the output of arbitrary external programs: include "#!program --flags" This lets you get the best of all worlds - the fast loading, simplicity and IDE sympathy of a declarative JSON-based config syntax, but if you hit the limits and need to programmatically generate some config you can do so whilst restricting the imperative logic only to the part of the file where it's needed. The rest remains declarative. All this works pretty well. At some point I want to package this up into a native library using GraalVM so it's available to anything that can load native libraries. Being Java it's accessible to any language that can run on the JVM which is pretty good already, but to use it from Go would require bindings. There are some downsides. It's not as well known as other syntaxes so syntax highlighting is sometimes missing from things like docsite generators. IntelliJ has a plugin for it but for other editors you might not get good support. It doesn't really have a schema language either, although you could of course just use JSON schema.
- nemoniac 3y agoIt's one of 12 "Why not use...?" subsections of that document but there's no subsection on "Why not use s-expressions?" Perhaps that was because they didn't consider them or maybe they did consider them but could not think of any reasons not to use them?
- drewcoo 3y ago> there's no subsection on "Why not use s-expressions?" It's all about config files. Are there people using s-exps in config files?
- kstrauser 3y agoI use Emacs, so yeah. I’d rather write some TOML/YAML/JSON for simple things where there are a handful of key/value pairs. In more complex cases where you want specific typing, or inheritance, or to solve any of the problems in this article, oh how I wish I could use something like a Guile parser to evaluate s-exprs. It would make life so much simpler.
- nemoniac 3y agoEmacs and Guix are standout examples. You could also include anything built on Clojure EDN. And, in the same way as Elisp for Emacs, Guile is an s-expression based language used not only for configuration but also for extension of GnuCash, GIMP, LilyPond and Pidgin.
- lapsed_lisper 3y agoS-expressions only really address a smallish part of what people seem to want out of configuration file syntaxes: a simple, recursive syntax. Among the things that S-expressions don't address per se are the interpretation of tokens (e.g., numeric vs non-numeric token syntax), how non-numeric tokens may be interpreted as booleans, symbols, timestamps, etc.; whether to use alists vs. plists for associations and the semantics of any duplicate keys within an associative construct; how to specify the schema for a configuration object (required vs. optional elements & the types of each, etc.) IOW, merely saying "use S-expressions" over-emphasizes syntax while under-emphasizing semantics.
- kunley 3y agoRelated: just few days ago I crafted together some ideas i had couple of years already for a configuration language, syntactically like HCL but without HashiCorp's idiosyncrasies. Here it goes, BCL (_Basic_ Configuration Language, for a lack of better name yet), Go prototype, I can code Python port and possibly several other as well.. https://github.com/wkhere/bcl https://github.com/wkhere/bcl PS. It was a pleasure to code, esp. getting the parser & the reflection right; the latter enables so elegant api. And yes, together with Russ Cox I am in a camp claiming that yacc is alive and well
- stmblast 3y agoCool article! I can see why TOML's verboseness can definitely be an issue once you start working on larger and larger TOML files. I'm assuming StrictYAML is your favourite out of all the different config files you could have currently?
- bemusedthrow75 3y agoA safe assumption. The author wrote/maintains StrictYAML.
- Izkata 3y ago> Guido van Rossum came across subtle bugs where the indentation disagreed with the syntactic grouping. Meaningful indentation fixed this class of bug. Since there are no begin/end brackets there cannot be a disagreement between grouping perceived by the parser and the human reader. Significant indentation alone doesn't eliminate this class of bugs. I'm pretty sure I've only seen this once ever, and can't remember the exact combination that caused it, but I have encountered a file that mixed tabs and spaces in a way python didn't barf over, that resulted in visual indentation being different from the programmatic indentation. I can definitely say it was python 2, not 3, so that particular instance may error nowadays.
- mmarx 3y agoMixing tabs and spaces is a TabError in Python 3, so this would definitely error today.
- bobbylarrybobby 3y agoWeird that his solution to this problem was to make a language where whitespace is significant (which makes it quite challenging to write) instead of just building a linter into Python.
- panzi 3y agoProbably a hot take, but I'd like an extended JSON syntax where you can do this: foo = "something to use later", bar = { "a": 1.1, "b": 2.2 }, baz = [ 1n, 999999999999n, ], /* comments! */ { [foo]: `\ multiline template string ${foo}`, ...bar, /* spread! */ "baz": [ ...baz, 1234n ], "more": my_extension("something like !foo in YAML, but just JavaScript function syntax") }
- ghkbrew 3y agoHow is this meaningfully different than using Javascript?
- panzi 3y agoIt doesn't run actual code. The function call syntax is just for extensions like `!vault` in Ansible YAML. And yes, it is supposed to be a new JavaScript subset and JSON superset.
- moritzwarhier 3y agoI'm confused, apart from the question where you'd like to limit JS, fundamentelly with your mix of statements and and an object literal, how would you even extract a key from this? E.g., how to retrieve that nameless object literal? Maybe defining the format as one big comma expression [^1] and disallowing functions+eval would have it's merit... but then, what to do about circular references? Very creative though :) [1] So that way, the whole config would evaluate to the expression behind the last top-level comma
- LordDragonfang 3y agoI assume the nameless object literal is, to steal a term from XML, the "root element", the same as with normal JSON.
- panzi 3y agoExactly.
- ursuscamp 3y agoIf JSON just originally had comments there wouldn't be such a proliferation of redundant configuration language, since JSON serves perfectly well 99% of cases.
- m000 3y agoComments, and allowing trailing commas after the last element of an array/dictionary.
- enriquto 3y agoAnd standard floating point numbers.
- svieira 3y agoMultiple bases would be better than standardizing on how to represent floating point numbers, since floating point still results in errors with unrepresentable values (e. g. 1/3 in base 10).
- harles 3y agoBase 10 is sufficient. Any base 2 number can be represented exactly in base 10. If you need rational numbers, you need something other than floating point.
- jerf 3y agoComments, trailing commas, and multiline strings. Comments, trailing commas, multiline strings, and a real date type. Comments, trailing commas, multiline strings, a real date type, and dropping the quotes on keys. Comments, trailing commas, ... JSON really isn't just the "simple tweak" away from being the perfect config language it is often presented as. There's this fun sort of error that only a group of people can make, where someone can stand up and make a statement like "JSON just needs a couple of tweaks to be the perfect config format!" and everyone can individually nod in agreement, and the group thinks it is in agreement. But it turns out every individual actually had a completely different interpretation of the statement and they don't agree at all. When you get a group of developers together to be clear about the changes to JSON you will discover not everybody has the same tweaks in mind. (& I wrote this before seeing the sibling comment from enriquto who gives another thing necessary for perfection. I don't even 100% know what enriquto means by "standard" floating point numbers versus what is already in the spec, but I bet it's another thing that multiple people could nod in agreement to but in fact mean something quite different about if you nail them down to the spec level!)
- Tainnor 3y agoI feel that as long as you use it for some simple configuration, you can use JSON, YAML, .properties files, TOML, whatever. The problem IMHO is that we're using "configuration" files for things that aren't configuration. The "story" example from the blog post illustrates this. I find it hard to read YAML files that are dozens or hundreds of lines long. Furthermore, once your "configuration" starts being so long, there's usually going to be enough repetition that you want to extract some duplication. YAML does have some facilities for this (anchors), unlike some other formats, but they're extremely limited. So what happens is that different tools using YAML all start designing their own mechanism for sharing behaviour. It's all usually very ad-hoc, has edge cases and may not do things in the way you expect them. It also forces you to learn the specific rules for these facilities instead of allowing you to reuse your general programming knowledge. On top of it all, YAML is essentially just a structureless key/value data structure. You can add schemas, but as far as I know, this isn't really standardised and editor support is... variable. In the worst case, you don't get any indication that you've configured something wrong. This is also part of the reason why I think that significant whitespace is OK for a programming language (still not a fan of it though), but bad for a configuration format, because bad indentation in a program either won't parse or will lead to obvious runtime errors, whereas bad indentation in a YAML file might just mean that a key isn't being set even though you think it should be. For authors of tools that consume YAML, this means writing a lot of custom validation logic instead of relying on standard techniques like type systems. I think we're on the wrong track and essentially just repeating XML's mistakes (just slightly less verbose, but also without schemas). We should rather use the programming constructs we know, e.g. by leveraging internal DSLs (I think that's part of the reason why Ruby was popular for tools like Chef for a short period, why Jenkins uses Groovy and Gradle now uses Groovy or Kotlin - these languages make internal DSLs easy). If we're worried about Turing completeness, maybe Dhall or something like it is the answer. But 400 line long YAML files with custom "!reference" tags that my editor doesn't understand doesn't seem like the solution.
- seanhunter 3y agoThis exactly. When the author is complaining that the configuration syntax doesn't support DRY you know something has gone wrong and configuration isn't really to blame.
- bowsamic 3y agoThese seem like major nitpicks
- martypitt 3y agoI really like the way the author writes. The article is great, and so are a few other links. Particularly enjoyed this[0] on the same site - The Norway Problem, which discusses parsing challenges in YAML. [0] https://hitchdev.com/strictyaml/why/implicit-typing-removed/ https://hitchdev.com/strictyaml/why/implicit-typing-removed/
- WirelessGigabit 3y agoThe Norway problem is an example of trying to fix a problem that was never there. If we kept strings double quoted as they were forever it wouldn't have been a problem. That's the easiest way. * TRUE / True / true & FALSE / False / false = parse as boolean * "something quoted" = parse as string * ... Same stuff happened with the first version of Angular. For some reason they parsed "no" as false, even though JavaScript coerces it to true.
- donatj 3y ago> Having symbols delimiting blocks and indentation violates the DRY principle. This is a very interesting hot take if I have ever heard one. I don’t want to agree, I don’t like significant whitespace, but I might have to agree. I think by the same line of reasoning, spaces for indentation rather than tabs violates DRY as well though? I could be wrong.
- lolinder 3y agoHonestly, all of these arguments feel pretty subjective to me. This is the major problem with most comparisons of config file formats: the actual semantic domain of a config file format is extremely limited, which means the main thing left to disagree over is syntax, which is highly subjective and extremely difficult to get people to agree on. Add too many syntactic features and a lot of people will disavow you for being too complicated. Add too few and you'll be missing someone's pet feature. Make white space significant and you'll frustrate people. Require extra characters to delineate and you'll frustrate another group. It's worth noting that this article is primarily talking about TOML in the context of the Python ecosystem, and I think that's a healthier way to talk about config file formats: How well suited are their syntactic choices to the community they're targeting?
- HideousKojima 3y ago>Make white space significant and you'll frustrate people. Or worse: make whitespace significant and strike a blow in the eternal tabs vs. spaces holy war at the same time, like YAML.
- deleted 3y ago[deleted]
- gabereiser 3y agoIsn’t the whole premise behind this article that, coming from Python where indentation is program structure, that TOML confuses the reader with syntax foreign to the reader? Like a C++ developer crying foul because inheritance doesn’t exist in YAML.
- omginternets 3y agoThe more applications I write, the more I tend to avoid typical config files in favor of using direnv [0] to set environment variables. [0] https://direnv.net/ https://direnv.net/
- goku12 3y agoThat's fine as long as there aren't too many variants of configurations. For example, I had a case where multiple ML training runs had to be configured with slight variations in parameters. That too, on the cloud. It can be managed with environment variables - but it becomes messy soon. With TOML, it was just a matter of switching those files around. There are better solutions these days for this particular use-case. However, it should give an idea where environment variables are not a better choice than TOML files.
- omginternets 3y agoAgreed, wholeheartedly in fact. My (poorly articulated) point is that the solution is generally to be found in simplifying configuration, rather than in the configuration language itself. This is something that microservice architectures are actually pretty good at doing.
- IshKebab 3y agoEnvironment variables are awful for config. You get no validation at all - typo in your variable name? It'll just silently ignore it. They can be injected by any parent process and are invisible to all child processes. Also they can be accessed by a simple getenv() anywhere in the program which eventually leads to your config being undocumented and spread all through your code. Environment variables are generally a hack and should be avoided where possible.
- omginternets 3y agoI use https://pkg.go.dev/github.com/urfave/cli/v3 https://pkg.go.dev/github.com/urfave/cli/v3 in my Go programs, and it does enough validation that this has never been a problem in practice, especially when paired with a simple config scheme (see my other comment).
- dorkrawk 3y agoNot really related to the article, but I always thought it was kind of quaint that TOML is "Tom's Obvious Minimal Language" (made by Tom Preston-Werner, one of the founders of GitHub).
- JohnMakin 3y agoIf you think this format is bad, you probably have never had to work with a fluentd config file.
- thayne 3y ago> It's very verbose. It's not DRY. It's syntactically noisy. I don't completely disagree with this. However, in most cases TOML is used, it isn't that much of a problem. And I actually like that the full key is repeated. When you have several layers of nested mappings, it can be hard to determine exactly where the current value is in the hierarchy. Especially if the top level key is above the current screen of text. It can also make it easier to search for a specific key. IMO, this is a case where more verbosity and repetition makes it more readable. That said, it seems a little arbitrary to me that inline tables don't allow newlines within them. If they did, then if you didn't like repeating the keys, you could use inline tables. > TOML's hierarchies are difficult to infer from syntax alone This is a little subjective, and depends on the actual data represented in the config. But in general, my experience is that when you have several layers of nesting, and the only indication of the hierarchy is indentation, it can be a little hard to follow where a specific value fits in the hierarchy. See above. And I disagree that meaningful indentation is "generally considered a good idea". I won't enumerate the pros and cons here, as it has been discussed a lot elsewhere, but it is definitely controversial, and subjective. > Overcomplication: Like YAML, TOML has too many features This section lists exactly one feature that it thinks TOML shouldn't have. Maybe dates shouldn't have been included, but it isn't anywhere close to the complexity of YAML.
- deleted 3y ago[deleted]
- paulddraper 3y ago> One way it does this is by trying to include date and time parsing which imports all of the inherent complications associated with dates and times. IDK, having date/time as first class seems very good. It's so common,
- mekster 3y agoI’ve quit using any format for projects because no matter how good they’re, nothing can bring auto completion as no IDE will parse and understand an external file format. Instead, I’m using PHP array which gives me nice auto completion and I see no problem with its format. When the config file needs to be consumed by a third party, then I’d use TOML.
- networked 3y agoI have read this critique, and I think it is a good idea to review it before you adopt TOML. That said, I went with TOML for a group of projects recently. My list projects on GitHub had become difficult to manage as plain Markdown. (They had too many items. The sections were easy to forget to sort. Items in multiple sections had to be kept in sync manually.) I decided to generate the Markdown from a template and serialized data. I evaluated YAML and TOML for storing the data and ended up choosing TOML. What I prefer about TOML for the data I have (dictionaries of dictionaries, no deep nesting) is that the textual representation is flat. I find it easier to read and edit. For comparison, here is one project's TOML data (formatted using https://github.com/tamasfe/taplo https://github.com/tamasfe/taplo with added comments): https://raw.githubusercontent.com/dbohdan/structured-text-tools/ce11203a432bf71bca35bc090e558844500fd7cf/data/projects.toml https://raw.githubusercontent.com/dbohdan/structured-text-to.... Here is the same data converted to YAML with unlimited line width and formatted using https://github.com/google/yamlfmt https://github.com/google/yamlfmt: https://paste.dbohdan.com/projects.1694621084.yaml.txt https://paste.dbohdan.com/projects.1694621084.yaml.txt.
- 0x457 3y agoI agree that TOML is frustrating when configuration gets large and complicated. But those issues seem like "As a Python user I don't like TOML because I like StrictYAML and TOML looks weird to my eyes". Unless there are deeply nested large maps/dictionaries, TOML is fine. I prefer UCL, though. IMO, the most sane and usable format for configuration.
- deleted 3y ago[deleted]
- stavros 3y agoI haven't had the same experience. Given that TOML is basically an INI file, I use it for data that's basically key-value with some categories. It seems fairly obvious, at least to me, that anything beyond that is going to be really ugly to represent. If I want a key-value file (for some simple preferences), TOML is by far the best out there. For anything more than that, I use YAML (though I should probably reach for a simpler subset of it, as YAML is crazy).
- jiehong 3y agoPerhaps there is kind of a way to improve JSON without changing JSON: use the gron format [0] as config. Tools can parse it as JSON and that’s it. [0]: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- continuational 3y agoCompared to JSON, TOML is more complicated. That means it's harder to learn, that it will be supported by fewer tools, and that the tools that do support it will have more bugs.
- mikece 3y agoWhile more verbose, XML is pretty clear. In what environment(s) are people messing with config files so much that the format is a productivity issue?
- hot_gril 3y agoMy patience for these config languages ran out a while ago. JSON is readable enough for simpler configs. Instead of inventing a new DSL for advanced configs, a simple .js or .py file can generate the JSON dynamically in a way that any programmer should be able to follow, which is common in the JS world. Text protobufs are also a decent alternative if you want something more than JSON but not dynamic. Heck, even an Excel spreadsheet (as mentioned elsewhere in the comments) can make sense for non-technical users. All of these options are easier to understand than some config-specific language with unexpected behaviors.
- rgreekguy 3y agoSad to find out it's not about Tales of Maj' Eyal.
- gavinhoward 3y agoAlright, I'll throw my hat in. I wrote a config file format. I took JSON and added comments, strict typing (using one character for any type that needs a marker), the ability to split items with newlines instead of commas, trailing commas, explicit binary data (as base64), and a new type I call "symbols" for things like enums or references. Then I removed the need to surround keys with quotes if they have only a certain set of typical characters. It looks like [1]. It turns out that JSON was very close. It just needed a few more things. Edit: one thing about config file formats that I strongly believe is that they must be purely data, no code. If you need code, supplement them with a separate thing, and perhaps use an established language, like Lua. [1]: https://git.yzena.com/Yzena/Yc/src/branch/master/build.gaml https://git.yzena.com/Yzena/Yc/src/branch/master/build.gaml
- Foomf 3y agoLua was originally designed as a config language.
- gavinhoward 3y agoSure, but I would say that anything that is Turing-complete is not a good config language. Lua pivoted to plugins instead.
- jokethrowaway 3y agoI'm just another datapoint but I like TOML way more - even in the complicated example I can look at any TOML file and I can see exactly which incredibly nested value am I looking at. Is my configuration file twice as large as an equivalent YAML? Great, I'll pay 0.000001 more on R2 but I'll retain the ability to easily understand what nightmare configuration I am looking at I agree with you the "3.14" vs 3.14 difference is not easy on some users, albeit that could be fixed in the business logic. Casting to int is not the end of the world. I also hate indentation based anything (I hate Python with a passion - especially now when I'm forced to use it because AI people are fond of it)
- Decabytes 3y agoIf Json had a way from the beginning to add comments and types, everything would have been perfect
- lr1970 3y agoMy biggest problem with TOML is absence of `null` values. Null is fundamental in describing missing or undefined values. Especially problematic are (numeric) arrays with missing values. JSON has null as the first-class object. In TOML world the recommended work-around is to use empty key-value object `{}` in place of a null. Kind of a hack. Makes replacing JSON with TOML tricky. EDIT: typos
- NathanFlurry 3y agoTOML config files are used heavily in my product for developer-facing features. I personally think it's the cleaner compared to YAML + JSON. It also benefits from being easier to copy + paste configs from docs since the entire path is included in each block; users don't have to worry about setting up the correct hierarchy. Regardless, I strongly agree with points #1-#3. I also wish there was a way to support something similar to inline YAML schemas to catch users' typos in their IDE. I would love to use HCL or potentially Ion, but IDE support and widespread acceptance isn't strong enough yet.
- pelletier 3y agoI wrote one of the Go implementations [0] when TOML was announced and have maintained it since. As a library implementor, I wish arrays would hold only one type at a time, but I get that could be useful for users. But as a user, I wish tables were fully defined once (more can't be added up later in the file), especially when using larger files. [0]: https://github.com/pelletier/go-toml https://github.com/pelletier/go-toml
- friend_and_foe 3y agoI absolutely loathe YAML and avoid using it. I use TOML for user edited configurations and I use JSON for things that a user doesn't need to touch, because it's human readable in name only. I like the type enforcement.
- fiddlerwoaroof 3y ago> Doctypes and namespaces are horrendous additions to the language, for instance. I basically completely disagree with this, especially namespaces: the ability to combine elements and attributes from multiple document types without ambiguity is extremely powerful and pays dividends when designing a query language like XPath
- Arch-TK 3y agoThe suggested replacement: StrictYAML is actually pretty excellent. I like it a lot. Seems to solve a lot of the issues I have with regular JSON, YAML and TOML. The only thing I wish there was a way to go directly between StrictYAML and dataclasses (or have an easy way to generate a corresponding dataclass from the schema).
- Hendrikto 3y ago> Having symbols delimiting blocks and indentation violates the DRY principle. This has to be one of the worst takes I ever heard.