4 ms·

Having spent many years in the windows land I definitely get where you're coming from. However, I find the Unix way where everything is a file, much simpler to

Having spent many years in the windows land I definitely get where you're coming from. However, I find the Unix way where everything is a file, much simpler to work with. Just take a simple task of copying a subset of some system's configuration to another one. On Unix it's just a matter of copying some files. On Windows (and our imaginary is that provides object storage) we would need to find the relevant objects, then write a script to export/import it (or mess with export/import tools for a while). Personally I see "everything is a file" as strength not weakness. In addition to my personal preference having a centralised configuration system not only restricts in many ways, presents a potential performance bottleneck, but also is an extra point of failure. Additionally, I think most Unix software uses a fairly simple text format for config files. There really is little reason for using anything else. If you need complex data structures use YAML. I've found many more proprietary binary config file formats on Windows (despite registry) than on Unix. So if the os would make something like this available there would still be no guarantee all apps use it. It would just be an extra thing to deal with for little benefit IMO.

genman 3y ago
Actually, why not both? Why not have files and an database like interface for them? Even better, databases normally provide a query plan explanation, quite like what is demanded here.
- indymike 3y ago
  I once thought the same, but there is not a universal answer on how to store data that delivers the combination of performance, reliability for every use case. Files are the lcd and leave room for optimization - and end up being a nice abstraction to hide the inner workings of decades of different storage devices and apis.
- tqwhite 3y ago
  Though I use .ini, myself. I often think about changing to sqlite. It is insanely stable and provides all the goodness of a database. Since I speak SQL like a native, it would be easy for me to manage. But, I stick with .ini still because, 1) I have a great access module that I would have to rewrite and, 2) there's always a text editor and sometimes I'd have to add a sqlite UI. Sounds like work to me.
_dain_ 3y ago
>If you need complex data structures use YAML. Please god no. JSON or TOML at most. YAML is too complicated.
- Spivak 3y ago
  For who? Yaml gives you all the standard containers, tools to avoid repeating yourself, lord of ways to format strings so they stay readable and arbitrary object serialization. Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then.
  - _dain_ 3y ago
    >Yaml gives you all the standard containers So do JSON and TOML. If you think you need more than what they provide, something has gone very wrong. >tools to avoid repeating yourself, Bad. Config is not code. Settings should be explicit. DRY is not helpful. Because of the lack of anchors/aliasing/variables etc, JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory. >lord of ways to format strings so they stay readable Bad. Too many ways: https://stackoverflow.com/a/21699210/6107981 https://stackoverflow.com/a/21699210/6107981 No normal human being can remember all the rules for how a string gets parsed. Therefore it isn't actually human-readable or human-writeable, despite masquerading as such. >arbitrary object serialization You can write out the fields as a JSON object. If this isn't enough, then what you're trying to serialize doesn't belong in a human-readable text config file. >Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then. This assumes the guy who wrote the config schema is the same as the guy who has to later use and edit it, which is seldom the case. When people do needlessly complicated config like this, they're taking an enormous stinking shit on my lawn for me to clean up.
    - kortex 3y ago
      > Bad. Config is not code. Settings should be explicit. DRY is not helpful. I disagree. From experience, there are enough times that configuration requires repetition, loops, interpolation, etc, that you often end up in this gross space of either repeating chunks manually (error prone and brittle), or using code/templating to generate config (requires everyone on the codebase to use a specific toolchain). HCL2 (used by Terraform) has been my favorite configuration system. It's just the right amount of power that I don't need copypasta or feel the need to resort to configgen. TOML is great for a single human-defined configuration for a program, but HCL does just as well at that, while also working for systems. JSON definitely fails as human-editable. Again, you need some tooling to ensure syntax correctness. > JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory. I really don't care how much memory is allocated during parse, within reason, as long as execution is bounded, can't blow up, or otherwise misbehave.
crabbone 3y ago
> much simpler to work with. It's simple when the task is simple, and it becomes a lot more complex than necessary when problems become more complex. Consider the joy of sysfs / procfs and similar. These are filesystems that, through files, expose information that is numbers, strings, lists, even trees. There's no uniformity there, no way to predict where and what sort of files with what sort of data will appear... Sometimes you get extra files with "metadata" that describes the format found in other files, sometimes it's in the same file, and sometimes it just doesn't exist. Sometimes numbers are encoded as strings, and sometimes as bytes. Booleans can be encoded as numbers, (which are either strings or bytes). There's plenty of weird behavior that isn't described by files / their properties. For example, suppose you want to delete a RAID device, but if fails because it turns out that it's used by LVM volume, but you didn't even know you were supposed to look for that... It gets even worse with proprietary stuff that doesn't follow conventions. Take for instance NVidia GPUs. They do appear in sysfs as PCIe devices... but most interesting properties just aren't listed there. Instead, NVidia's driver creates "devices" in udevfs (i.e. /dev) with arbitrary properties that you cannot predict because the format is whatever the vendor wanted it to be at any version of their product. In other words, files are a very poor mechanism for describing data. In order to work with them you need conventions that aren't part of any file mechanism. Take network interface names, or block device names, stuff like eth0 or nvme0n1p1 -- you need to know the convention and how to interpret the name (i.e. the first suggests that the device is using Ethernet driver and is the first one connected, the second means that the device is using NVMe driver, it's the first device of its kind attached to PCIe bus, first namespace, first partition.) Specifically, in the case of block devices, you could discover the same information given in the file name by traversing the relevant filesystem subtree, but in some cases the information is just encoded as a file name or as it contents, but it doesn't utilize the filesystem primitives. It just relies on convention. ---- > I think most Unix software uses a fairly simple text format for config files How did you count? Do you mean software that is part of UNIX standard or software that runs on UNIX? What makes the majority an interesting metric? (I.e. suppose there are 10M of different programs that run on UNIX, and 6M of them are simple enough that they don't require complicated configuration, there are still 4M of those that do, and some of them are probably used daily by millions of users. I mean, who cares if simple programs don't need complicated configuration? The problem is the configuration of complex programs, and they are indispensible in our daily lives.
crabbone 3y ago
> If you need complex data structures use YAML YAML is a poorly engineered format. It doesn't think about storage at all, for example (i.e. in its definition, it's just a single text blob. What if you need to split it? The format is awful for random access: you need to read the whole thing to extract a single data item. If you need to query such a configuration multiple times you'll do a lot of unnecessary work. YAML poorly defines numerical types (how many digits can a float have? what about integers? Are floats and integers the same thing?) YAML lists... allow mixed types. Not sure it's such a good thing from performance and correctness perspective. YAML has unnecessary "hash-table" type. Hash-tables aren't a data format, they require function in the reader that makes asymptotic properties of hash-tables work. In fact, YAML has just a weird sub-type of list, where elements come in pairs. Since it inherited all this from JSON, there's also the ambiguity related to repeated keys in such hash-tables -- what should an application do with those nobody knows. YAML sucks for transmission due to the ambiguity caused by repeated keys in "hash-tables". You cannot stream such a format if you have the policy that the last key wins or if you have the policy that repetition is not allowed. Also, since YAML allows references but doesn't require that the reference lexically point to a previously defined element, you may have unresolved references which, again, will prevent you from streaming. YAML defines mappings to "native" elements of different other languages... but what should you do if you are parsing it in Ruby, but get a mapping to Python? YAML schema sucks. It would require a separate "expose" to describe why. YAML comes with no concept of users or namespaces which would be necessary for ownership / secure sharing of information. It also comes without triggers which would allow an application to respond to changes in data. YAML is very hard to parse for no reason. It's very easy to make a typo in YAML that will not make it invalid, but will be interpreted contrary to what was intended. YAML doesn't have a canoncial form which makes comparing two elements a non-trivial and in some cases undefined task. YAML doesn't have variables, nor does it have forall / exists functionality. This results in a lot of tools that work with YAML overlaying it with yet another layer of configuration, which includes variables (eg. Helm charts, Ansible playbooks). ---- I mean, honestly, people who created YAML probably saw it as an inconsequential, sketchy project that should take maybe a weekend or two. They didn't plan for this format to be the best... they just made something... that sort of did something... but not really. I have no idea why something like this received the acclaim that it did.

genman 3y ago

Actually, why not both? Why not have files and an database like interface for them? Even better, databases normally provide a query plan explanation, quite like what is demanded here.

indymike 3y ago
I once thought the same, but there is not a universal answer on how to store data that delivers the combination of performance, reliability for every use case. Files are the lcd and leave room for optimization - and end up being a nice abstraction to hide the inner workings of decades of different storage devices and apis.
tqwhite 3y ago
Though I use .ini, myself. I often think about changing to sqlite. It is insanely stable and provides all the goodness of a database. Since I speak SQL like a native, it would be easy for me to manage. But, I stick with .ini still because, 1) I have a great access module that I would have to rewrite and, 2) there's always a text editor and sometimes I'd have to add a sqlite UI. Sounds like work to me.

_dain_ 3y ago

>If you need complex data structures use YAML. Please god no. JSON or TOML at most. YAML is too complicated.

Spivak 3y ago
For who? Yaml gives you all the standard containers, tools to avoid repeating yourself, lord of ways to format strings so they stay readable and arbitrary object serialization. Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then.
- _dain_ 3y ago
  >Yaml gives you all the standard containers So do JSON and TOML. If you think you need more than what they provide, something has gone very wrong. >tools to avoid repeating yourself, Bad. Config is not code. Settings should be explicit. DRY is not helpful. Because of the lack of anchors/aliasing/variables etc, JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory. >lord of ways to format strings so they stay readable Bad. Too many ways: https://stackoverflow.com/a/21699210/6107981 https://stackoverflow.com/a/21699210/6107981 No normal human being can remember all the rules for how a string gets parsed. Therefore it isn't actually human-readable or human-writeable, despite masquerading as such. >arbitrary object serialization You can write out the fields as a JSON object. If this isn't enough, then what you're trying to serialize doesn't belong in a human-readable text config file. >Yaml only gets weird if you want to use complex objects as dict keys but that complexity is on you then. This assumes the guy who wrote the config schema is the same as the guy who has to later use and edit it, which is seldom the case. When people do needlessly complicated config like this, they're taking an enormous stinking shit on my lawn for me to clean up.
  - kortex 3y ago
    > Bad. Config is not code. Settings should be explicit. DRY is not helpful. I disagree. From experience, there are enough times that configuration requires repetition, loops, interpolation, etc, that you often end up in this gross space of either repeating chunks manually (error prone and brittle), or using code/templating to generate config (requires everyone on the codebase to use a specific toolchain). HCL2 (used by Terraform) has been my favorite configuration system. It's just the right amount of power that I don't need copypasta or feel the need to resort to configgen. TOML is great for a single human-defined configuration for a program, but HCL does just as well at that, while also working for systems. JSON definitely fails as human-editable. Again, you need some tooling to ensure syntax correctness. > JSON has the nice property that you can parse it with exactly one upfront memory allocation: N bytes of JSON requires at most N words of memory. I really don't care how much memory is allocated during parse, within reason, as long as execution is bounded, can't blow up, or otherwise misbehave.

crabbone 3y ago

> much simpler to work with. It's simple when the task is simple, and it becomes a lot more complex than necessary when problems become more complex. Consider the joy of sysfs / procfs and similar. These are filesystems that, through files, expose information that is numbers, strings, lists, even trees. There's no uniformity there, no way to predict where and what sort of files with what sort of data will appear... Sometimes you get extra files with "metadata" that describes the format found in other files, sometimes it's in the same file, and sometimes it just doesn't exist. Sometimes numbers are encoded as strings, and sometimes as bytes. Booleans can be encoded as numbers, (which are either strings or bytes). There's plenty of weird behavior that isn't described by files / their properties. For example, suppose you want to delete a RAID device, but if fails because it turns out that it's used by LVM volume, but you didn't even know you were supposed to look for that... It gets even worse with proprietary stuff that doesn't follow conventions. Take for instance NVidia GPUs. They do appear in sysfs as PCIe devices... but most interesting properties just aren't listed there. Instead, NVidia's driver creates "devices" in udevfs (i.e. /dev) with arbitrary properties that you cannot predict because the format is whatever the vendor wanted it to be at any version of their product. In other words, files are a very poor mechanism for describing data. In order to work with them you need conventions that aren't part of any file mechanism. Take network interface names, or block device names, stuff like eth0 or nvme0n1p1 -- you need to know the convention and how to interpret the name (i.e. the first suggests that the device is using Ethernet driver and is the first one connected, the second means that the device is using NVMe driver, it's the first device of its kind attached to PCIe bus, first namespace, first partition.) Specifically, in the case of block devices, you could discover the same information given in the file name by traversing the relevant filesystem subtree, but in some cases the information is just encoded as a file name or as it contents, but it doesn't utilize the filesystem primitives. It just relies on convention. ---- > I think most Unix software uses a fairly simple text format for config files How did you count? Do you mean software that is part of UNIX standard or software that runs on UNIX? What makes the majority an interesting metric? (I.e. suppose there are 10M of different programs that run on UNIX, and 6M of them are simple enough that they don't require complicated configuration, there are still 4M of those that do, and some of them are probably used daily by millions of users. I mean, who cares if simple programs don't need complicated configuration? The problem is the configuration of complex programs, and they are indispensible in our daily lives.

crabbone 3y ago

> If you need complex data structures use YAML YAML is a poorly engineered format. It doesn't think about storage at all, for example (i.e. in its definition, it's just a single text blob. What if you need to split it? The format is awful for random access: you need to read the whole thing to extract a single data item. If you need to query such a configuration multiple times you'll do a lot of unnecessary work. YAML poorly defines numerical types (how many digits can a float have? what about integers? Are floats and integers the same thing?) YAML lists... allow mixed types. Not sure it's such a good thing from performance and correctness perspective. YAML has unnecessary "hash-table" type. Hash-tables aren't a data format, they require function in the reader that makes asymptotic properties of hash-tables work. In fact, YAML has just a weird sub-type of list, where elements come in pairs. Since it inherited all this from JSON, there's also the ambiguity related to repeated keys in such hash-tables -- what should an application do with those nobody knows. YAML sucks for transmission due to the ambiguity caused by repeated keys in "hash-tables". You cannot stream such a format if you have the policy that the last key wins or if you have the policy that repetition is not allowed. Also, since YAML allows references but doesn't require that the reference lexically point to a previously defined element, you may have unresolved references which, again, will prevent you from streaming. YAML defines mappings to "native" elements of different other languages... but what should you do if you are parsing it in Ruby, but get a mapping to Python? YAML schema sucks. It would require a separate "expose" to describe why. YAML comes with no concept of users or namespaces which would be necessary for ownership / secure sharing of information. It also comes without triggers which would allow an application to respond to changes in data. YAML is very hard to parse for no reason. It's very easy to make a typo in YAML that will not make it invalid, but will be interpreted contrary to what was intended. YAML doesn't have a canoncial form which makes comparing two elements a non-trivial and in some cases undefined task. YAML doesn't have variables, nor does it have forall / exists functionality. This results in a lot of tools that work with YAML overlaying it with yet another layer of configuration, which includes variables (eg. Helm charts, Ansible playbooks). ---- I mean, honestly, people who created YAML probably saw it as an inconsequential, sketchy project that should take maybe a weekend or two. They didn't plan for this format to be the best... they just made something... that sort of did something... but not really. I have no idea why something like this received the acclaim that it did.