6 ms·

> GUIDs, IP Addresses, and byte arrays. In what situation is it not suitable to store these as strings, byte arrays as unbounded hex-encoded integers? 0xabcdef

by 3np 6y ago

> GUIDs, IP Addresses, and byte arrays. In what situation is it not suitable to store these as strings, byte arrays as unbounded hex-encoded integers? 0xabcdef123400000000 As for your csv-example, I’d propose that if you have multi-value entries in a list like that large enough that it becomes unwieldy to repeat the property names every time (like in JSON or yaml), it really doesn’t belong in static configuration but in some kind of data store. (And in the odd situation where that’s not the case, just use csv like you seem to prefer?)

jiggawatts 6y ago
It's about strong typing. I detest stringly-typed programming. If the format specifies some stronger types, the decoders can "bubble that up" to the programming language that decode the file. E.g.: a guid turns up as a "System.Guid" type in C# instead of "System.String". This matters, because all of these are the "same" GUID, but not all GUID parsers can handle all of these formats: {123e4567-e89b-12d3-a456-426652340000} (123e4567-e89b-12d3-a456-426652340000) 123e4567-e89b-12d3-a456-426652340000 123e4567e89b12d3a456426652340000 urn:uuid:123e4567-e89b-12d3-a456-426655440000 Similarly, with IPv6 all of the following are the "same" address, but if encoded as a string it's hit-and-miss if the far end can properly handle all of the variants: fe80:0000:0000:0000:01ff:fe23:4567:890a fe80:0000:0000:0000:1ff:fe23:4567:890a fe80:0:0:0:1ff:fe23:4567:890a [fe80::1ff:fe23:4567:890a] fe80::1ff:fe23:4567:890a fe80::1ff:fe23:4567:890a%3 Similarly, for your hex-encoded binary example, you'd be shocked at how rare it is for programming languages to provide a "hex-string-to-byte-array" decoding function. If you roll your own, it'll be about 10x less efficient than something done properly using SIMD instructions. Just recently I had to deal with Azure Resource Manager Templates, where even the distinction between numeric and text types has been eroded. There's several number fields that must be a string. There are also numeric fields that weirdly accept a string, but only if it is an ARM template expression that evaluates to a string containing a number at runtime. Ugh...
- 3np 6y ago
  Surely you shouldn’t be limited to types as interpreted and represented in the low-level config parser library but should have a strongly typed representation i of the config structure that you implement in whatever language you’re using. C# is a good example. JSON doesn’t have any of these types. It doesn’t even make a distinction between integers and other numbers. Yet it’s common practice to marshal these into the kind of types you mentioned at ingestion. At the end of the day, if you type it with your keyboard, it’s essentially a string. I think it’s completely fine that all this functionality is part of neither programming languages core libraries nor configuration file formats - there is no one-size-fits-all here. There are plenty of configuration frameworks that do what you ask for.
  - jiggawatts 6y ago
    > whatever language you’re using. This is the #1 mistake people make when designing, evaluating, or critiquing frameworks, file formats, protocols, and the like. I don't get to choose the language. I'm not writing 100% of the code that I use. In fact, I have control over approximately 0.00001% of the code involved in processing a typical JSON file, or TOML file, or YAML file. OTHER people control the language choice, and it certainly won't be ONE language. It'll be many languages. If submit an ARM template to Azure, it'll go through at least three languages in the process: C#, Python, and JavaScript. Possibly C++ and F#. Who knows? Even when the language is reasonably consistent, such as JavaScript, there is very little consistency in the specific parser used. After the parser-level inconsistencies, there's further inconsistencies in how the stringly-typed data is converted into some more strongly typed format. That's just up to whoever wrote the code that consumes the data. There is exactly zero standardisation of this. None whatsoever. It's almost never documented, and there's just no way to know without experimenting. NONE of this is in my control, or your control. I can't emphasise this enough. Stop thinking in terms of a developer sitting down in front of an IDE with a new project, where they type all of the code in, hit compile and ship the binary to some customer. Start thinking in terms of having to deal with inconsistencies between Terraform, Cloudformation, Amazon, AWS, GCP, and CloudFront. Starting thinking about having to publish something like a Rust module to three different crate repositories, and cache it locally, and have it work properly with the various IDEs people use. Start thinking in terms of security vulnerabilities of different parsers at different layers escaping data differently, or smuggling data using encodings accepted by the back-end past web application firewalls that don't understand that particular encoding. This stuff matters. JSON famously had a definition so short that it fit on a business card. Less is more, right? This "simplicity" lead to fun blog posts titled "Parsing JSON is a minefield": http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php The designer of TOML seems to have never read that article, or if he has, he hasn't learned any lessons from it...
- jasonpeacock 6y ago
  You're assuming that all languages have a "System.Guid" type that the configuration value can be parsed into. If you want a strongly type configuration that works across many languages, you'll need to also provide the per-language implementation of those types to fill the gaps. You'll notice that JSON, YAML, INI, and XML also do not have IP address nor GUID types. What configuration format does meet your approval?
  - jiggawatts 6y ago
    > What configuration format does meet your approval? None at the moment. My problem with all of them is that I can't scan them visually to spot typos or mistakes. They break the content up and move related data too far apart for the inconsistencies to be immediately apparent. Don't laugh, but in the absence of better options, I prefer to use Excel. With tabular data, things that belong together are visually adjacent. There are no repeated headers, or unnecessary syntax. Formulas are available, but hidden by default. In the past I liked to use XML, but only with Altova XmlSpy, which had a kind of "hierarchical table view" superficially similar to Excel, but for tree-like data. Both of the above are easy to use from PowerShell. I even wrote myself an Import-Xlsx module that works without Excel having to be installed. It uses the .NET framework libraries for opening XLSX files. (Under they hood they're essentially just a Zip file containing plain XML data.) Ideally? I'd like something like the SQL Server Analysis Services MDX query language, but with a front-end more like Excel for the editing of the data. It is purpose designed for multi-dimensional data, and in my field that's exactly what I have to do: provision every combination of a bunch of tables. For example: For each data centre, for each availability zone, for each of PRD/UAT/DEV, deploy each of the following roles, with 'n' instances each. This is a multi-dimensional cube, also known as a full outer product. Naively expanding everything is not good either, there needs to be some sort of language for making "exceptions". Such as: The DEV environment is only in this location. UAT has fewer sewers than PRD. Etc... That's where a nice language would work wonders. You'd want to be able to do things like: *.*.dev.sql.instances = 0 us-west.*.dev.sql.instances = 2 That would set the dev instances everywhere to zero, except US West. It's selecting a hyperplane through the cube and setting all of the cells to the constant on the right. This is the kind of thing MDX is designed to do, but no "simple" config language ever can. So instead you get monstrosities like the Azure ARM "copy" syntax: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/copy-resources https://docs.microsoft.com/en-us/azure/azure-resource-manage...
- toastal 6y ago
  Don't like stringly-typed programming or configuration? There's Dhall for you https://dhall-lang.org/ https://dhall-lang.org/ > Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports It's specifically not Turing-complete so it has a number of safeties built in because of this. I've been using it more and more wherever possible as of late.
- GoblinSlayer 6y ago
  See https://tools.ietf.org/html/rfc5952 https://tools.ietf.org/html/rfc5952 on how to write ipv6 addresses.
- secondcoming 6y ago
  > If you roll your own, it'll be about 10x less efficient than something done properly using SIMD instructions. Is this really an issue when loading a field from a config file? If it was something happening 1m times per second then maybe I could see your point.
  - jiggawatts 6y ago
    JSON is now at the point where yes, it does makes sense to optimise it with AVX2 instructions. https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
    - curryst 6y ago
      JSON is in a slightly different area since it's commonly used as a serialization format for RPC/REST so it's in a hot path. Optimizations there make a lot of sense; optimizations on reading your configuration file are almost certainly premature optimizations. The fact that your config will load some tiny amount faster with those optimizations is a side effect of the hot path optimizations, not a requirement for config formats.
- yellowapple 6y ago
  > It's about strong typing. I detest stringly-typed programming. I feel like that's something that's more an appropriate concern for the application, and not something that ought to be baked into the configuration format itself. For example: > E.g.: a guid turns up as a "System.Guid" type in C# instead of "System.String". Can you not just pass the System.String you get from the decoder into System.Guid's constructor and handle the resulting error should that string not actually be a GUID? Like, my problem with a lot of config file formats is that they're frequently too clever about assuming types of things when I would much rather they be strings by default for me to interpret later. > If you roll your own ["hex-string-to-byte-array" decoding function], it'll be about 10x less efficient than something done properly using SIMD instructions. Unless you roll your own that uses those SIMD instructions, whether in hand-written assembly or in a programming language with a compiler smart enough to use SIMD for this. This does, however, reek of a premature optimization.

by 3np 6y ago

jiggawatts 6y ago

It's about strong typing. I detest stringly-typed programming. If the format specifies some stronger types, the decoders can "bubble that up" to the programming language that decode the file. E.g.: a guid turns up as a "System.Guid" type in C# instead of "System.String". This matters, because all of these are the "same" GUID, but not all GUID parsers can handle all of these formats: {123e4567-e89b-12d3-a456-426652340000} (123e4567-e89b-12d3-a456-426652340000) 123e4567-e89b-12d3-a456-426652340000 123e4567e89b12d3a456426652340000 urn:uuid:123e4567-e89b-12d3-a456-426655440000 Similarly, with IPv6 all of the following are the "same" address, but if encoded as a string it's hit-and-miss if the far end can properly handle all of the variants: fe80:0000:0000:0000:01ff:fe23:4567:890a fe80:0000:0000:0000:1ff:fe23:4567:890a fe80:0:0:0:1ff:fe23:4567:890a [fe80::1ff:fe23:4567:890a] fe80::1ff:fe23:4567:890a fe80::1ff:fe23:4567:890a%3 Similarly, for your hex-encoded binary example, you'd be shocked at how rare it is for programming languages to provide a "hex-string-to-byte-array" decoding function. If you roll your own, it'll be about 10x less efficient than something done properly using SIMD instructions. Just recently I had to deal with Azure Resource Manager Templates, where even the distinction between numeric and text types has been eroded. There's several number fields that must be a string. There are also numeric fields that weirdly accept a string, but only if it is an ARM template expression that evaluates to a string containing a number at runtime. Ugh...

3np 6y ago
Surely you shouldn’t be limited to types as interpreted and represented in the low-level config parser library but should have a strongly typed representation i of the config structure that you implement in whatever language you’re using. C# is a good example. JSON doesn’t have any of these types. It doesn’t even make a distinction between integers and other numbers. Yet it’s common practice to marshal these into the kind of types you mentioned at ingestion. At the end of the day, if you type it with your keyboard, it’s essentially a string. I think it’s completely fine that all this functionality is part of neither programming languages core libraries nor configuration file formats - there is no one-size-fits-all here. There are plenty of configuration frameworks that do what you ask for.
- jiggawatts 6y ago
  > whatever language you’re using. This is the #1 mistake people make when designing, evaluating, or critiquing frameworks, file formats, protocols, and the like. I don't get to choose the language. I'm not writing 100% of the code that I use. In fact, I have control over approximately 0.00001% of the code involved in processing a typical JSON file, or TOML file, or YAML file. OTHER people control the language choice, and it certainly won't be ONE language. It'll be many languages. If submit an ARM template to Azure, it'll go through at least three languages in the process: C#, Python, and JavaScript. Possibly C++ and F#. Who knows? Even when the language is reasonably consistent, such as JavaScript, there is very little consistency in the specific parser used. After the parser-level inconsistencies, there's further inconsistencies in how the stringly-typed data is converted into some more strongly typed format. That's just up to whoever wrote the code that consumes the data. There is exactly zero standardisation of this. None whatsoever. It's almost never documented, and there's just no way to know without experimenting. NONE of this is in my control, or your control. I can't emphasise this enough. Stop thinking in terms of a developer sitting down in front of an IDE with a new project, where they type all of the code in, hit compile and ship the binary to some customer. Start thinking in terms of having to deal with inconsistencies between Terraform, Cloudformation, Amazon, AWS, GCP, and CloudFront. Starting thinking about having to publish something like a Rust module to three different crate repositories, and cache it locally, and have it work properly with the various IDEs people use. Start thinking in terms of security vulnerabilities of different parsers at different layers escaping data differently, or smuggling data using encodings accepted by the back-end past web application firewalls that don't understand that particular encoding. This stuff matters. JSON famously had a definition so short that it fit on a business card. Less is more, right? This "simplicity" lead to fun blog posts titled "Parsing JSON is a minefield": http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php The designer of TOML seems to have never read that article, or if he has, he hasn't learned any lessons from it...
jasonpeacock 6y ago
You're assuming that all languages have a "System.Guid" type that the configuration value can be parsed into. If you want a strongly type configuration that works across many languages, you'll need to also provide the per-language implementation of those types to fill the gaps. You'll notice that JSON, YAML, INI, and XML also do not have IP address nor GUID types. What configuration format does meet your approval?
- jiggawatts 6y ago
  > What configuration format does meet your approval? None at the moment. My problem with all of them is that I can't scan them visually to spot typos or mistakes. They break the content up and move related data too far apart for the inconsistencies to be immediately apparent. Don't laugh, but in the absence of better options, I prefer to use Excel. With tabular data, things that belong together are visually adjacent. There are no repeated headers, or unnecessary syntax. Formulas are available, but hidden by default. In the past I liked to use XML, but only with Altova XmlSpy, which had a kind of "hierarchical table view" superficially similar to Excel, but for tree-like data. Both of the above are easy to use from PowerShell. I even wrote myself an Import-Xlsx module that works without Excel having to be installed. It uses the .NET framework libraries for opening XLSX files. (Under they hood they're essentially just a Zip file containing plain XML data.) Ideally? I'd like something like the SQL Server Analysis Services MDX query language, but with a front-end more like Excel for the editing of the data. It is purpose designed for multi-dimensional data, and in my field that's exactly what I have to do: provision every combination of a bunch of tables. For example: For each data centre, for each availability zone, for each of PRD/UAT/DEV, deploy each of the following roles, with 'n' instances each. This is a multi-dimensional cube, also known as a full outer product. Naively expanding everything is not good either, there needs to be some sort of language for making "exceptions". Such as: The DEV environment is only in this location. UAT has fewer sewers than PRD. Etc... That's where a nice language would work wonders. You'd want to be able to do things like: *.*.dev.sql.instances = 0 us-west.*.dev.sql.instances = 2 That would set the dev instances everywhere to zero, except US West. It's selecting a hyperplane through the cube and setting all of the cells to the constant on the right. This is the kind of thing MDX is designed to do, but no "simple" config language ever can. So instead you get monstrosities like the Azure ARM "copy" syntax: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/copy-resources https://docs.microsoft.com/en-us/azure/azure-resource-manage...
toastal 6y ago
Don't like stringly-typed programming or configuration? There's Dhall for you https://dhall-lang.org/ https://dhall-lang.org/ > Dhall is a programmable configuration language that you can think of as: JSON + functions + types + imports It's specifically not Turing-complete so it has a number of safeties built in because of this. I've been using it more and more wherever possible as of late.
GoblinSlayer 6y ago
See https://tools.ietf.org/html/rfc5952 https://tools.ietf.org/html/rfc5952 on how to write ipv6 addresses.
secondcoming 6y ago
> If you roll your own, it'll be about 10x less efficient than something done properly using SIMD instructions. Is this really an issue when loading a field from a config file? If it was something happening 1m times per second then maybe I could see your point.
- jiggawatts 6y ago
  JSON is now at the point where yes, it does makes sense to optimise it with AVX2 instructions. https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
  - curryst 6y ago
    JSON is in a slightly different area since it's commonly used as a serialization format for RPC/REST so it's in a hot path. Optimizations there make a lot of sense; optimizations on reading your configuration file are almost certainly premature optimizations. The fact that your config will load some tiny amount faster with those optimizations is a side effect of the hot path optimizations, not a requirement for config formats.
yellowapple 6y ago
> It's about strong typing. I detest stringly-typed programming. I feel like that's something that's more an appropriate concern for the application, and not something that ought to be baked into the configuration format itself. For example: > E.g.: a guid turns up as a "System.Guid" type in C# instead of "System.String". Can you not just pass the System.String you get from the decoder into System.Guid's constructor and handle the resulting error should that string not actually be a GUID? Like, my problem with a lot of config file formats is that they're frequently too clever about assuming types of things when I would much rather they be strings by default for me to interpret later. > If you roll your own ["hex-string-to-byte-array" decoding function], it'll be about 10x less efficient than something done properly using SIMD instructions. Unless you roll your own that uses those SIMD instructions, whether in hand-written assembly or in a programming language with a compiler smart enough to use SIMD for this. This does, however, reek of a premature optimization.