6 ms·

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am

> In conclusion, JSON is not a data format you can rely on blindly. What does HN suggest for configuration files (to be written by a human essentially)? I am looking at YAML and TOML. My experience with JSON based config files was horrible.

FreeFull 10y ago
In my personal experience TOML works really well. It's a little reminiscent of .ini files, but definitely is better.
jzwinck 10y ago
YAML or TSV depending on whether your configuration looks like a rectangular table. If you want extreme flexibility using C++ as the main language, take a look at my project: https://github.com/jzwinck/pccl https://github.com/jzwinck/pccl It lets you configure your C++ apps using Python. Config items can even be Python functions.
kozhevnikov 10y ago
https://github.com/typesafehub/config/blob/master/HOCON.md https://github.com/typesafehub/config/blob/master/HOCON.md
- ronnier 10y ago
  This is what I use, it's very good. https://github.com/typesafehub/config https://github.com/typesafehub/config
  - kodablah 10y ago
    I have found it is not very cross-language (and a bit of a pain w/ the replacement hierarchies to easily implement everywhere).
marios 10y ago
I don't have a specific recommendation, but when I see a project uses a JSON file as configuration, I wonder: "hasn't the author ever needed to include a comment in the configuration ?".
- cestith 10y ago
  I can't speak for others. When I write something that uses JSON for configuration files those files are pretty much always written from configuration management. The comments are then in the manifest/recipe/playbook that generates the config file, which is where people are actually working with the values.
robert_tweed 10y ago
Funnily enough, as I've been experimenting with Chef and trying to stick to JSON config files where allowed, I was again struck that (a) it's not a good choice for config files (b) it's an OK choice though (c) lots of people are using it anyway (d) nearly everyone that does so (including Chef) allows comments, so in reality are not actually using JSON at all. Point (d) is the important one. I really think we need a standard for json-with-comments. JSONC or whatever, but it should have a different standard filename and it should have an RFC dictating what is and isn't allowed. Personally I would allow only // comments because there are too many subtle issues with C-style comments, but it may be too late to agree on that. Half the point of JSON is that if application A stores its data as JSON then application B can parse that without any nasty surprises. Except, there are now probably thousands of noncompliant implementations in the wild that only exist because the standard doesn't allow comments. Each one of those standards adds subtle differences (in addition to the comments themselves) depending largely on how they remove the comments before passing to the standards-compliant JSON parser (assuming they do that, which being DC's recommended approach, is as close to a standard as currently exists).
- pilif 10y ago
  > (b) it's an OK choice though I really think it's not an OK choice. A config file format that doesn't allow comments provides some of the worst possible UX. One of the nice things about config files is that normally they are self-documenting, explaining the meaning of the various directives and providing possible values. Without comments, you have to constantly switch between the documentation and the config file. Also, the restriction on trailing commas is another really bad issue for a config file language as it pollutes diffs, makes moving lines around needlessly difficult and is one more landmine waiting to happen for the sysadmin editing a file. No. JSON not at all OK as a config file language.
  - mSparks 10y ago
    How does json "not support comments"? {"comment":"default values for this object"}
    - 10y ago
Ketak 10y ago
I have actually used Lua before with good success. It was on a smaller scale, so I can't speak to edge cases, but I would certainly recommend considering it at the least.
empath75 10y ago
i actually like hcl: https://www.terraform.io/docs/configuration/syntax.html https://www.terraform.io/docs/configuration/syntax.html
Ono-Sendai 10y ago
XML
isidor3 10y ago
Lua was written for exactly this purpose, and I personally enjoy writing with it, so it would be my first choice in most cases.
- petre 10y ago
  What purpose? Writing config files? Lua is a programming language (a nice one too). It's code. Code should not be used for config files, nor for data serialization because once you eval it you are executing it.
  - isidor3 10y ago
    Yes, Lua was originally written as a language for rich configuration files and has grown out of that into a more fully featured language. It's also one of the easier languages to sandbox, since you can evaluate user provided code in a custom environment that only contains the functions you deem safe. You can even use the standard debug hooks to set an upper limit on the number of instructions a script can execute to prevent someone from creating an infinite loop in a config file and locking whatever thread is reading the config. It's not very appropriate as a data serialization format, or as machine written config, but the parent post specifically asked about human written configuration files.
lucb1e 10y ago
> What does HN suggest for configuration files (to be written by a human essentially)? JSON. YAML confuses many people by being whitespace-sensitive; ini files I find too limited.
- DonHopkins 10y ago
  A general rule of thumb: Never use yet another non-markup language designed by people who claimed to be designing yet another markup language from the very outset, then after somebody awkwardly pointed out that what they'd designed wasn't actually a markup language, they invent a backronym to contradict that embarrassing historical fact. It just makes me wonder what the hell they thought they were doing all that time... It's like designing something called YACC, and ending up with an interpreter interpreter! https://en.wikipedia.org/wiki/YAML https://en.wikipedia.org/wiki/YAML >Originally YAML was said to mean Yet Another Markup Language, referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain't Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.
- inimino 10y ago
  JSON is great for data exchange but config files should be human readable and amply commented. Even rolling your own simple format is probably better than using JSON.
- sankyo 10y ago
  in Clojure we use EDN (Extensible Data Notation) for config files and a data transfer format. https://github.com/edn-format/edn https://github.com/edn-format/edn It is really a pleasure to use compared to JSON and XML. While it may not be as compact as ProtoBuffers, Thrift, or Avro, it is human readable and also valid Clojure code. Libraries are ready available to convert it to JSON.
tantalor 10y ago
Protocol Buffers text format: https://developers.google.com/protocol-buffers/docs/overview#whynotxml https://developers.google.com/protocol-buffers/docs/overview...
petre 10y ago
YAML is equally horrible and the spec is an order of magnitude more complex. I wasted half an hour trying to spot an error in the ejabberd yaml config, only to find out something trivial was missing. At least JSON has braces even though it's not suitable for configuration files. By all means choose TOML or something else (even ini or java properties files) instead.
- crdoconnor 10y ago
  What was the trivial missing thing?
  - deleted 10y ago
    [deleted]
- yes_or_gnome 10y ago
  YAML has braces. In fact, it's a super set of JSON. Any YAML parser should be able to parse JSON encoded data with one exception. Block comments. Which is a bastardization of JSON (as mentioned in the article) so block comments shouldn't be a problem in most cases.
  - deathanatos 10y ago
    > [YAML is] a super set of JSON. The specification says this, but I was never convinced it was true. Specifically, the spec around escaped unicode characters lacks any mention of surrogates being encoded in two \u sequences, but rather specifies \u as: > Escaped 16-bit Unicode character. Which is an unhelpfully not-even-wrong statement. In practice, trying to treat JSON as a subset of YAML results in things not round-tripping, like the following: In [9]: yaml.load(json.dumps('\N{PILE OF POO}')) Out[9]: '\ud83d\udca9' (in case it isn't clear, that's not a valid repr of pile-of-poo in Python: In [14]: _9 == '\N{PILE OF POO}' Out[14]: False ) I've now got surrogates in a decoded Unicode string (why is this even allowed, I don't know); this results in fun behavior like `.encode('utf-8')` raising. Edit: weird: HN appears to strip pile of poo from comments…
steveklabnik 10y ago
I've used JSON and YAML for a very long time, but whenever I have the option, I'll be using TOML every time. YAML is _huge_ (did you know all JSON is valid YAML?) and those features can come back to bite you... http://blog.codeclimate.com/blog/2013/01/10/rails-remote-code-execution-vulnerability-explained/ http://blog.codeclimate.com/blog/2013/01/10/rails-remote-cod... JSON has a number of annoyances, mostly no comments, no trailing commas, all the stuff this article gets into. Formats like INI or CSV don't really have a spec, or if they do, most implementations don't seem to follow them. TOML is a bit weird at first, but it's grown on me quite a bit.
slasaus 10y ago
After trying json, yaml, json5, java properties, ini and toml, I finally choose hjson* as the configuration file format for the software I'm building. It's the easiest format to read and write IMHO, a bit like nginx config files. * http://hjson.org http://hjson.org
mamcx 10y ago
INI files. Of course, some lunatics try to embed big amounts of text and that is where INI files not look ok.
sirn 10y ago
If you decide to use YAML, make sure to check out Strict YAML[1] and its FAQ[2]. [1]: https://github.com/crdoconnor/strictyaml https://github.com/crdoconnor/strictyaml [2]: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst...
uiri 10y ago
I would recommend TOML, but I am a bit biased as the author of the toml python package.

FreeFull 10y ago

In my personal experience TOML works really well. It's a little reminiscent of .ini files, but definitely is better.

jzwinck 10y ago

YAML or TSV depending on whether your configuration looks like a rectangular table. If you want extreme flexibility using C++ as the main language, take a look at my project: https://github.com/jzwinck/pccl https://github.com/jzwinck/pccl It lets you configure your C++ apps using Python. Config items can even be Python functions.

kozhevnikov 10y ago

https://github.com/typesafehub/config/blob/master/HOCON.md https://github.com/typesafehub/config/blob/master/HOCON.md

ronnier 10y ago
This is what I use, it's very good. https://github.com/typesafehub/config https://github.com/typesafehub/config
- kodablah 10y ago
  I have found it is not very cross-language (and a bit of a pain w/ the replacement hierarchies to easily implement everywhere).

marios 10y ago

I don't have a specific recommendation, but when I see a project uses a JSON file as configuration, I wonder: "hasn't the author ever needed to include a comment in the configuration ?".

cestith 10y ago
I can't speak for others. When I write something that uses JSON for configuration files those files are pretty much always written from configuration management. The comments are then in the manifest/recipe/playbook that generates the config file, which is where people are actually working with the values.

robert_tweed 10y ago

Funnily enough, as I've been experimenting with Chef and trying to stick to JSON config files where allowed, I was again struck that (a) it's not a good choice for config files (b) it's an OK choice though (c) lots of people are using it anyway (d) nearly everyone that does so (including Chef) allows comments, so in reality are not actually using JSON at all. Point (d) is the important one. I really think we need a standard for json-with-comments. JSONC or whatever, but it should have a different standard filename and it should have an RFC dictating what is and isn't allowed. Personally I would allow only // comments because there are too many subtle issues with C-style comments, but it may be too late to agree on that. Half the point of JSON is that if application A stores its data as JSON then application B can parse that without any nasty surprises. Except, there are now probably thousands of noncompliant implementations in the wild that only exist because the standard doesn't allow comments. Each one of those standards adds subtle differences (in addition to the comments themselves) depending largely on how they remove the comments before passing to the standards-compliant JSON parser (assuming they do that, which being DC's recommended approach, is as close to a standard as currently exists).

pilif 10y ago
> (b) it's an OK choice though I really think it's not an OK choice. A config file format that doesn't allow comments provides some of the worst possible UX. One of the nice things about config files is that normally they are self-documenting, explaining the meaning of the various directives and providing possible values. Without comments, you have to constantly switch between the documentation and the config file. Also, the restriction on trailing commas is another really bad issue for a config file language as it pollutes diffs, makes moving lines around needlessly difficult and is one more landmine waiting to happen for the sysadmin editing a file. No. JSON not at all OK as a config file language.
- mSparks 10y ago
  How does json "not support comments"? {"comment":"default values for this object"}
  - 10y ago

Ketak 10y ago

I have actually used Lua before with good success. It was on a smaller scale, so I can't speak to edge cases, but I would certainly recommend considering it at the least.

empath75 10y ago

i actually like hcl: https://www.terraform.io/docs/configuration/syntax.html https://www.terraform.io/docs/configuration/syntax.html

Ono-Sendai 10y ago

XML

isidor3 10y ago

Lua was written for exactly this purpose, and I personally enjoy writing with it, so it would be my first choice in most cases.

petre 10y ago
What purpose? Writing config files? Lua is a programming language (a nice one too). It's code. Code should not be used for config files, nor for data serialization because once you eval it you are executing it.
- isidor3 10y ago
  Yes, Lua was originally written as a language for rich configuration files and has grown out of that into a more fully featured language. It's also one of the easier languages to sandbox, since you can evaluate user provided code in a custom environment that only contains the functions you deem safe. You can even use the standard debug hooks to set an upper limit on the number of instructions a script can execute to prevent someone from creating an infinite loop in a config file and locking whatever thread is reading the config. It's not very appropriate as a data serialization format, or as machine written config, but the parent post specifically asked about human written configuration files.

lucb1e 10y ago

> What does HN suggest for configuration files (to be written by a human essentially)? JSON. YAML confuses many people by being whitespace-sensitive; ini files I find too limited.

DonHopkins 10y ago
A general rule of thumb: Never use yet another non-markup language designed by people who claimed to be designing yet another markup language from the very outset, then after somebody awkwardly pointed out that what they'd designed wasn't actually a markup language, they invent a backronym to contradict that embarrassing historical fact. It just makes me wonder what the hell they thought they were doing all that time... It's like designing something called YACC, and ending up with an interpreter interpreter! https://en.wikipedia.org/wiki/YAML https://en.wikipedia.org/wiki/YAML >Originally YAML was said to mean Yet Another Markup Language, referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain't Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.
inimino 10y ago
JSON is great for data exchange but config files should be human readable and amply commented. Even rolling your own simple format is probably better than using JSON.
sankyo 10y ago
in Clojure we use EDN (Extensible Data Notation) for config files and a data transfer format. https://github.com/edn-format/edn https://github.com/edn-format/edn It is really a pleasure to use compared to JSON and XML. While it may not be as compact as ProtoBuffers, Thrift, or Avro, it is human readable and also valid Clojure code. Libraries are ready available to convert it to JSON.

tantalor 10y ago

Protocol Buffers text format: https://developers.google.com/protocol-buffers/docs/overview#whynotxml https://developers.google.com/protocol-buffers/docs/overview...

petre 10y ago

YAML is equally horrible and the spec is an order of magnitude more complex. I wasted half an hour trying to spot an error in the ejabberd yaml config, only to find out something trivial was missing. At least JSON has braces even though it's not suitable for configuration files. By all means choose TOML or something else (even ini or java properties files) instead.

crdoconnor 10y ago
What was the trivial missing thing?
- deleted 10y ago
  [deleted]
yes_or_gnome 10y ago
YAML has braces. In fact, it's a super set of JSON. Any YAML parser should be able to parse JSON encoded data with one exception. Block comments. Which is a bastardization of JSON (as mentioned in the article) so block comments shouldn't be a problem in most cases.
- deathanatos 10y ago
  > [YAML is] a super set of JSON. The specification says this, but I was never convinced it was true. Specifically, the spec around escaped unicode characters lacks any mention of surrogates being encoded in two \u sequences, but rather specifies \u as: > Escaped 16-bit Unicode character. Which is an unhelpfully not-even-wrong statement. In practice, trying to treat JSON as a subset of YAML results in things not round-tripping, like the following: In [9]: yaml.load(json.dumps('\N{PILE OF POO}')) Out[9]: '\ud83d\udca9' (in case it isn't clear, that's not a valid repr of pile-of-poo in Python: In [14]: _9 == '\N{PILE OF POO}' Out[14]: False ) I've now got surrogates in a decoded Unicode string (why is this even allowed, I don't know); this results in fun behavior like `.encode('utf-8')` raising. Edit: weird: HN appears to strip pile of poo from comments…

steveklabnik 10y ago

I've used JSON and YAML for a very long time, but whenever I have the option, I'll be using TOML every time. YAML is _huge_ (did you know all JSON is valid YAML?) and those features can come back to bite you... http://blog.codeclimate.com/blog/2013/01/10/rails-remote-code-execution-vulnerability-explained/ http://blog.codeclimate.com/blog/2013/01/10/rails-remote-cod... JSON has a number of annoyances, mostly no comments, no trailing commas, all the stuff this article gets into. Formats like INI or CSV don't really have a spec, or if they do, most implementations don't seem to follow them. TOML is a bit weird at first, but it's grown on me quite a bit.

slasaus 10y ago

After trying json, yaml, json5, java properties, ini and toml, I finally choose hjson* as the configuration file format for the software I'm building. It's the easiest format to read and write IMHO, a bit like nginx config files. * http://hjson.org http://hjson.org

mamcx 10y ago

INI files. Of course, some lunatics try to embed big amounts of text and that is where INI files not look ok.

sirn 10y ago

If you decide to use YAML, make sure to check out Strict YAML[1] and its FAQ[2]. [1]: https://github.com/crdoconnor/strictyaml https://github.com/crdoconnor/strictyaml [2]: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst#what-is-wrong-with-implicit-typing https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst...

uiri 10y ago

I would recommend TOML, but I am a bit biased as the author of the toml python package.