15 ms·
TOML superficially looks clean, but it has many of the same problems other similar languages have. Such file formats are typically used for configuration files
by jiggawatts 6y ago
TOML superficially looks clean, but it has many of the same problems other similar languages have.
Such file formats are typically used for configuration files. Yet, a substantial amount of effort was expended on data types not commonly seen in configuration files, such as a date-time-offset. Meanwhile, much more common data types typically used in configuration files is missing, such as GUIDs, IP Addresses, and byte arrays.
Most disappointingly, TOML has a "table" type, but unfortunately, just like other "RPC as Config" formats, the column names need to be repeated for every row, which is just crazy. It also makes no sense to talk about "arrays of tables", which are actually just "rows" in standard terminology. An array of rows... is a table.
Here is a terse example of the type of thing that does come up very often in bulk provisioning of the type that needs config files full of data:
@"
Server,Subnet,IP
db01,databases,10.1.2.3
db02,databases,10.1.2.4
user01,filers,10.3.10.1
user02,filers,10.3.10.2
"@ | ConvertFrom-Csv | New-VMBuildScript.ps1
I'm yet to see a config file format that can even approach this in terms of its readability and terseness.
- jasonpeacock 6y agoIt all depends on context. Honestly, I've rarely (never?) needed to store GUIDs, IP addresses, or byte arrays in a configuration file. So that's not as common of a situation as you're thinking - maybe for your experience, but there are many different uses for configuration. The example you give would suck in any configuration file format, and honestly I don't believe it belongs in a configuration file - I'd put it into a data file (e.g. a CSV as you show) and then in the configuration file I'd include a pointer to that data file to be loaded separately.
- DC-3 6y agoEvery server config ever requires an IP address entry which in itself is a non trivial proportion of all the user edited config files in the world.
- joshvm 6y agoBut surely strings are valid? Unless you want some kind of static type checking/linting for eg an ipv4 vs ipv6 key? Strings are very portable as well (POD). Plenty of real world applications use static hostnames and rely on DCHP to assign IPs. When you have systems that can fail hard and you need to replace a NIC, it saves having to update either a ton of config files, router configs, or your hosts file(s). Most networking library api calls also take strings as arguments for this reason. Not saying its a better solution, but there times when either an IP or a string hostname should be acceptable. I've never had to use it, but Qt does have a hostname class, for an example of what such a dtype looks like. The class explicitly handles conversion for you: https://doc.qt.io/qt-5/qhostaddress.html https://doc.qt.io/qt-5/qhostaddress.html
- chillfox 6y agoIP is a configuration option for most server software, it’s just that it usually needs to be set to 0.0.0.0 and that is the default anyway so most admins just leave it out.
- tlamponi 6y agoLets use :: nowadays ;-) After all we got ~16 % IPv6 traffic hitting our repository CDN. (seems to match with an observation of LWN https://lwn.net/Articles/808896/ https://lwn.net/Articles/808896/ )
- tomjakubowski 6y agoI would love to see YAML or EDN-style tagging in TOML for use cases like marking strings as IP addresses.
- geofft 6y agoI think that's two different senses of the word "configuration". TOML is great for key/value data where you want a little bit of structure / nesting / categorization on the keys, but probably not more than one level or so. Think of, say, the preferences for your text editor, or (as widely used) a metadata file for a library that specifies its dependencies and build configuration. It looks like you have tabular data where you don't need any nesting, every row has the exact same structure, and there's no single column of a row that you'd call "the value." TOML is not well-suited for this, and I don't think it sets out to be. I see how you'd call it "configuration," but it seems like a pretty different category of thing. (I'd buy the argument that "table" is a poor choice of name for that TOML construct; I'd personally have called it a "multi-value" or something. It's an array of objects, and there's no guarantee that the objects have homogeneous structure, and it's usually useful for them to have different structures.)
- 3np 6y ago> GUIDs, IP Addresses, and byte arrays. In what situation is it not suitable to store these as strings, byte arrays as unbounded hex-encoded integers? 0xabcdef123400000000 As for your csv-example, I’d propose that if you have multi-value entries in a list like that large enough that it becomes unwieldy to repeat the property names every time (like in JSON or yaml), it really doesn’t belong in static configuration but in some kind of data store. (And in the odd situation where that’s not the case, just use csv like you seem to prefer?)
- jiggawatts 6y agoIt's about strong typing. I detest stringly-typed programming. If the format specifies some stronger types, the decoders can "bubble that up" to the programming language that decode the file. E.g.: a guid turns up as a "System.Guid" type in C# instead of "System.String". This matters, because all of these are the "same" GUID, but not all GUID parsers can handle all of these formats: {123e4567-e89b-12d3-a456-426652340000} (123e4567-e89b-12d3-a456-426652340000) 123e4567-e89b-12d3-a456-426652340000 123e4567e89b12d3a456426652340000 urn:uuid:123e4567-e89b-12d3-a456-426655440000 Similarly, with IPv6 all of the following are the "same" address, but if encoded as a string it's hit-and-miss if the far end can properly handle all of the variants: fe80:0000:0000:0000:01ff:fe23:4567:890a fe80:0000:0000:0000:1ff:fe23:4567:890a fe80:0:0:0:1ff:fe23:4567:890a [fe80::1ff:fe23:4567:890a] fe80::1ff:fe23:4567:890a fe80::1ff:fe23:4567:890a%3 Similarly, for your hex-encoded binary example, you'd be shocked at how rare it is for programming languages to provide a "hex-string-to-byte-array" decoding function. If you roll your own, it'll be about 10x less efficient than something done properly using SIMD instructions. Just recently I had to deal with Azure Resource Manager Templates, where even the distinction between numeric and text types has been eroded. There's several number fields that must be a string. There are also numeric fields that weirdly accept a string, but only if it is an ARM template expression that evaluates to a string containing a number at runtime. Ugh...
- 3np 6y agoSurely you shouldn’t be limited to types as interpreted and represented in the low-level config parser library but should have a strongly typed representation i of the config structure that you implement in whatever language you’re using. C# is a good example. JSON doesn’t have any of these types. It doesn’t even make a distinction between integers and other numbers. Yet it’s common practice to marshal these into the kind of types you mentioned at ingestion. At the end of the day, if you type it with your keyboard, it’s essentially a string. I think it’s completely fine that all this functionality is part of neither programming languages core libraries nor configuration file formats - there is no one-size-fits-all here. There are plenty of configuration frameworks that do what you ask for.
- pydry 6y agodatabases: db01: 10.1.2.3 db02: 10.1.2.4 filers: user01: 10.3.10.1 user02: 10.3.10.2
- jiggawatts 6y agoWhat if I needed to add about 5-10 additional columns to that. Would TOML still have a similarly terse way of encoding that? E.g.: Region, AZ, Network, Subnet, Description.
- echelon 6y agoThis is a table. Let's see it do a tree. (And not an adjacency matrix. That's cheating.)
- ktpsns 6y agoA tree is a graph without cycles. A graph is a relational structure E(a,b) which can easily be stored in tables. This is why relational databases like SQL store tables. So there's nothing exceptional about storing graphs, and adjacency lists or matrices are all fine. To connect with the other threads, SQLite would be probably the best candidate proposed here to store a tree.
- echelon 6y agoMy point was about ergonomics, or at least that's what I was trying to hit at. TOML does a much better job than CSV.
- mikepurvis 6y agoI've been using TOML recently and what displeases me most is that almost all string values need to be quoted.
- dnautics 6y agoArray of objects is terrible in TOML. I grudgingly use it only because it's categorically better than YAML IMO and INI isn't powerful enough.
- mikepurvis 6y agoAt least in YAML you only have to quote a string if it contains a colon-space and can therefore be inferred to be a dict. In TOML all strings are quoted: https://github.com/toml-lang/toml/issues/105#issuecomment-14054665 https://github.com/toml-lang/toml/issues/105#issuecomment-14...
- afiori 6y agostring quotation is a very interesting topic, one variation I would like in a config file is a single backtick that continues up to the first line break
- megous 6y agoOtoh, I got plenty of support requests from people not understanding that the space at the end of a line in an .ini file is part of the value. (app is using glib ini parser) Quoting at least makes it clear what's part of the value.
- alecbz 6y agoI think terseness and readability are often at odds past a certain point, and TOML seems to optimize for readability at the expense of terseness. E.g., having to specify column names again and again is tedious to write, but makes the file easier to read and modify safely.
- chillfox 6y agoYeah, I think toml is great short/simple config files, yaml is good for medium length/complexity, and I have yet to see anything that’s not painful for long config files.
- smitty1e 6y agoThere is always cutting to the chase and using SQLite. https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html
- chillfox 6y agoSQLite is lovely to work with, but I don't think it is entirely appropriate as a configuration format for server software considering that all of the existing tooling for configuration management (Ansible, Puppet, CFEngine) is based on template text files.
- smitty1e 6y agoI dunno. By the time one manages a full enterprise of configuration, one puts it in SQLite and treat your cookbooks/roles/stacks as report products. After normalizing, one could get to a tidy heterogeneous solution. (Only the finest vaporware spoken here.)
- alecbz 6y agoI actually think it's almost the opposite for me. In a tiny yaml file it's pretty easy to understand what everything is; it's huge files where toml's forcing you to full qualify nested tables becomes useful. Too many huge yaml files where I'm just scrolled somewhere in the middle and have no clue what part of the tree I'm actually looking at.
- alecbz 6y agoThough I'll agree that I don't love the name "table". I think it's referring to the kind of table you might see in an instruction booklet or something, with keys on the left and their values on the right, but the possible confusion with database tables seems not great.
- doonesbury 6y agoYah, you gotta point.
- GoblinSlayer 6y agoTerseness is a bad property though, it hurts readability and correctness.
- unionpivo 6y agoI get what you are saying, but everything has trade offs. Because everyone would agree for 80%, but then need different 20%.That's how you end up with 10k pages specs or XML and doc types. And if you need that just use XML, that is actually what it was designed for.
- baq 6y agohttps://dhall-lang.org/# https://dhall-lang.org/# perhaps?
- baby 6y agoYou need to use strings to store guids or ips, it should not be a configuration language’s type
- m463 6y agointeresting. good pattern. I frequently do something similar: opt=" # this is the foo option a=1 # define the bar #BAR=ON #BAR=OFF BAR=MAYBE " | egrep -v '^#|^$' | sed 's/^/-D/' or progs=" one two three #four five siz seven ate tree " | egrep -v '^#|^$' | while read o1 o2 o3 do program $o1 $o2 $o3 done sort of like on-the-fly readable configurability
- BiteCode_dev 6y agoThat's not config data that you have here, that's your program input (hint: you pipe it to stdin). TOML is for config, config that users edit by hand, not input.
- jiggawatts 6y agoConfig is data. Data is input.
- BiteCode_dev 6y agoAnything that is not hardcoded is technically input. But there is a reason we have a word for config. It's a specific kind of input: it feeds the boostrapping state of your program, it's not the main data your program process. The difference is important: structure, complexity and dynamism requirements are not the same.
- bronson 6y agoThat looks like hand-edited config to me. At least, I've maintained files similar to that in past gigs. It's a nice example of how CSV can work well if you don't need to worry about quoting.
- BiteCode_dev 6y ago1 - you don't pipe config to stdin. If you use standard input, that's your main input. 2 - config sets your program boostrapping state. It's not the main data your program processes. That's your main input. IP addresses are what the script processes. It's not config.
- bronson 6y ago1 - If you google 'config stdin' you'll see a number of good reasons a program might want to accept configuration on stdin. (also, not that it matters, but my program read it from a file) 2 - Those IP addresses are probably not the main data the program processes. In my case, they were configuration, describing where an auditing processor scrapes input and how to allocate compute.
- g8oz 6y agoI recently read that YAML has a way of specifying a "sequence of sequences" which could work for tabular data like that in your example. i.e - [Server,Subnet,IP] - [db01,databases,10.1.2.3] - [db02,databases,10.1.2.4] - [user01,filers,10.3.10.1] - [user02,filers,10.3.10.2] That being said, the CSV does seem more straightforward.