8 ms·

I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation converts numbers to doubles, another to either

I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation converts numbers to doubles, another to either doubles or longs depending on whether they have a . or e, and another which converts them to bignums (or passes them as strings to the caller). One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. There are also issues when something looks numeric but doesn’t parse (eg 1.2.3, 3/2, 12in, 4h30m2s, 2:30, 2020-02-29, etc). One way to deal with these is a tokenisation rule like in Common Lisp: if it is a valid number syntax then treat it as a number, otherwise it’s a symbol, but this can lead to issues (eg you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes. It seems crazy to pass that detail on to the poor sod who has to write the config file).

kanox 6y ago
> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false". The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.
- pdonis 6y ago
  > The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. I don't think that's what this tool is for. This tool is for humans to read and edit data. That's a different use case from automated programs exchanging data, for which I agree you should be using standardized numeric and boolean data types and not making everything a string. But how that standardized data gets determined from data that humans enter should be up to the individual application.
  - miki123211 6y ago
    Agreed. Serialization is not really possible with this tool, due to the following: > A key that requires quoting must not contain both single and double quote characters. You can't really serialize user data with that restriction.
  - yoz-y 6y ago
    What is the use of this then? Humans don’t need structured data to exchange information and if this is insufficient to communicate it to the machine then it won’t be a great human to machine format either.
    - pdonis 6y ago
      > Humans don’t need structured data to exchange information Maybe we don't need it, but it often helps. Also, this data format isn't necessarily for humans to exchange data with other humans, but for humans to give data to applications in a format that's much easier for humans to use. > if this is insufficient to communicate it to the machine Not at all. Each specific application can easily parse this data format according to its own needs. What this data format doesn't specify is a single translation into application data that is the same for all applications. But applications don't need or want that, because they have different use cases.
      - yoz-y 6y ago
        I see. That makes sense. Now that I think about it maybe if its in front of some database or storage that only handles strings it would work well as well.
  - kanox 6y ago
    It is still useful for humans to have a well-understood way to input typed data. If you make everything a string then the interpretation of "no" as boolean true or false is left to each tool, and there are even tools which have different interpretations of "yes/no" for each field.
- kortex 6y ago
  Except YAML is straight-up wrong at times, unless you know all the edge cases. I just learned unquoted NO is coerced to False. A classic case of leaky abstraction/"bad magic". Unless you have actual type annotations/tags (eg xml, jsonld, graphql), everything IS a string. There's no assumptions otherwise.
  - nine_k 6y ago
    I would hazard to say that any magic is bad in engineering. (Sorry Perl fans!)
    - laumars 6y ago
      There isn’t really any magic in Perl, just lots of unfamiliar lexicon if you come from more traditional programming languages. Perls problem is it’s history is rooted in command line usage so there’s a tonne of inherited reserved single character variable names and such like that are optimised for keystrokes rather than readability. However you can certainly write Perl programs that don’t follow those older conventions and look more like a modern language.
      - bmn__ 6y ago
        > There isn’t really any magic in Perl http://p3rl.org/guts#Magic-Variables http://p3rl.org/guts#Magic-Variables perl -MDevel::Peek=Dump -mTie::Scalar -e' //g; Dump $_; tie $c => "Tie::StdScalar"; Dump $c; Dump \%ENV ' 2>&1 | grep MAGIC
        jlokier 6y ago
        That kind of magic in Perl has a different kind of meaning than "magic number/bool/string parsing" in YAML. Perl tied-variable magic just means there are (effectively) getter and setter properties attached to the variable. "Magic" is just the name that was chosen in the implementation, and it stuck. It's used to implement variables with special, automatic meanings, like $$ for "current pid" and $! for "last error". It's also used to implement variables with user-defined behaviours on access, which is quite handy for a lot of abstractions. A lot of modern languages support both of these things, because they are useful, but it's not called magic in those languages, it's called something like "watchers", "proxies", "getters and setters" or "hooks". No, the criticism of YAML-style "magic" is that it leads to entirely surprising behaviour from innocuous input. Perl magic is not that kind. If you're using a special variable, you already know why.
- pfraze 6y ago
  I think this is an “input format” vs “serialization format” issue. NestedText is meant for user input so it’s focusing more on the UX of the syntax. In that case, having to parse strings is a reasonable task for developers to take on
- geofft 6y ago
  > The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. In Python: >>> import json >>> json.loads('{"x": 9007199254740993}') {'x': 9007199254740993} In my browser's JavaScript console: > JSON.parse('{"x": 9007199254740993}') {x: 9007199254740992} (Consider what happens if you try to send a tweet's ID, a perfectly normal number like 205052027259195393, through JSON. Or if you try to serialize a stack trace on a 64-bit system, where addresses are also perfectly normal numbers.)
  - tmd83 6y ago
    I have very little understanding of JSON, so why would this happen?
    - sergeykish 6y ago
      Python promote to Bignum automatically. Javascript number is a floating point Number.MAX_SAFE_INTEGER // 9007199254740991 Number.MAX_SAFE_INTEGER + 1 // 9007199254740992 Number.MAX_SAFE_INTEGER + 2 // 9007199254740992 no automatic promotion to BigInt 9007199254740991n + 2n // 9007199254740993n
    - auscompgeek 6y ago
      JavaScript numbers are double-precision floating point numbers.
    - DougBTX 6y ago
      The example uses numbers outside the range of JS numbers (too many significant digits), so when the JSON is evaluated as a JS number some of the least significant bits are rounded. Personally, I think it is a mistake to use values in JSON that can’t be represented in JS, but the standard doesn’t explicitly forbid it.
    - zerocrates 6y ago
      It's more a JS thing: The lone JS number type is double-precision floating point, meaning beyond a certain range there are integers that cannot be accurately represented as a JS number. The JSON standard doesn't place restrictions on size or precision of numbers, instead just noting that implementations can vary their treatment of and limits on numbers. While JS uses doubles for all numbers, many other languages emit an integer type for a JSON integer. So, once you go beyond the range where a double can accurately represent all integers, you run the risk of a mismatch in how the number is interpreted by different languages parsing the same JSON. Of course the spec also allows you to create way too big or too precise numbers that would be problematic in most languages as well; it's just that this is a somewhat common bugbear. I wouldn't necessarily call it a flaw in JSON though, more an issue with JSON.parse or really just a fact of life when dealing with numbers in JS. Alternatives to the built in JSON.parse exist to read large integers as strings or bigints.
- epicureanideal 6y ago
  If NestedText is meant to be written by non-programmer humans, the details of null, undefined, 0, false, "NO", nil, or whatever else, would likely also be lost on them. Most likely if their input is meant to be machine interpreted they would need to be trained to provide specific inputs anyway. I like that NestedText doesn't hide that problem. It lets the user organization decide how it wants to manage that problem, and what symbols or words are understood by the people authoring the files.
- deleted 6y ago
  [deleted]
- GoblinSlayer 6y ago
  >you now have 30 tool-specific ways to write "true" and "false" ISO8601 joins the chat.
zamadatix 6y ago
That's not the only way to deal with them, you can specify type like bson. The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined. The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.
thayne 6y ago
My ideal config language would have unambigous syntax for at least 64-bit signed integers and doubles (so, for example the spec for the language requires 56 to be parsed as an integer and 56.0 to be a double. Additional types would be ok too, as long as the syntax is unambiguous and obvious.
- samatman 6y ago
  Both of those are true for TOML. The spec requires that a full 64 bits of signed integer be parsed and understood, that hexadecimal, octal, and binary values be integers, and that floating point values be parsed as doubles. It doesn't support hexadecimal float, however, which is a pity: having a guaranteed bit-identical format is a nice affordance.
Phrodo_00 6y ago
I am of the opposite view: As long as there are no edge cases, supporting more types in data languages is good, it leads to failing faster. Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot. > you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that) (disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda) [1] http://amzn.github.io/ion-docs/ http://amzn.github.io/ion-docs/
jiggawatts 6y ago
> I think avoiding numeric types is a good decision. Only if this format is intended for use-cases that never need to deal with numbers. > One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. That statement couldn't possibly be more wrong. Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following: - Unsigned 64-bit numbers. - A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error? - NaN - Infinity - Negative zero - Denormal numbers. - Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats! - Efficiently encoding floating point to use the minimum digits without losing precision. - Parsing those minimal numbers with perfect "round-tripping". - Doing the above efficiently. - Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser... The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient. Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that. Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit! The difficult bit is the parsing.
- lsorber 6y ago
  In my opinion, the best solution to these issues is to: 1. Declare numbers as numbers in the configuration language. E.g. "decimal(1e1000)". 2. Parse declared numbers with a lossless format like Python's decimal.Decimal. 3. Let users decide at their own risk if they want to convert to a lossy format like float.
- cycomanic 6y ago
  But which of these are problems in configuration files written by a human? That is the aim of the format. Moreover in applications were there could be issues, it would most certainly be tied to very specific fields and you would want specific application logic to handle that field. Now if people misuse it as a data exchange format or so, yes I agree with you, but at that point just use a binary format instead.
barumi 6y ago
> I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation (...) It seems you're mixing up the language definition with implementations that try to follow the language definition. If different implementations have different results then either they are buggy or the language has some important holes in the specification. Either way,the solution to this problem is not less validation. > One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. That's the whole point, isn't it? I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions? Wouldn't it simply easier to let the language and the parser do that already? I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.
7952 6y ago
I wonder if you could just include explicit type information? name(str): Dave age(int4): 22 dob(date): 2020-02-01 photo(base64):TWFuIGl....

kanox 6y ago

> One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. Getting rid of yaml's 30 ways to write "true" and "false" by making everything is a string just means that you now have 30 tool-specific ways to write "true" and "false". The "everything is a string" approach already exists in shell scripts and TCL and it's not really that great.

pdonis 6y ago
> The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. I don't think that's what this tool is for. This tool is for humans to read and edit data. That's a different use case from automated programs exchanging data, for which I agree you should be using standardized numeric and boolean data types and not making everything a string. But how that standardized data gets determined from data that humans enter should be up to the individual application.
- miki123211 6y ago
  Agreed. Serialization is not really possible with this tool, due to the following: > A key that requires quoting must not contain both single and double quote characters. You can't really serialize user data with that restriction.
- yoz-y 6y ago
  What is the use of this then? Humans don’t need structured data to exchange information and if this is insufficient to communicate it to the machine then it won’t be a great human to machine format either.
  - pdonis 6y ago
    > Humans don’t need structured data to exchange information Maybe we don't need it, but it often helps. Also, this data format isn't necessarily for humans to exchange data with other humans, but for humans to give data to applications in a format that's much easier for humans to use. > if this is insufficient to communicate it to the machine Not at all. Each specific application can easily parse this data format according to its own needs. What this data format doesn't specify is a single translation into application data that is the same for all applications. But applications don't need or want that, because they have different use cases.
    - yoz-y 6y ago
      I see. That makes sense. Now that I think about it maybe if its in front of some database or storage that only handles strings it would work well as well.
- kanox 6y ago
  It is still useful for humans to have a well-understood way to input typed data. If you make everything a string then the interpretation of "no" as boolean true or false is left to each tool, and there are even tools which have different interpretations of "yes/no" for each field.
kortex 6y ago
Except YAML is straight-up wrong at times, unless you know all the edge cases. I just learned unquoted NO is coerced to False. A classic case of leaky abstraction/"bad magic". Unless you have actual type annotations/tags (eg xml, jsonld, graphql), everything IS a string. There's no assumptions otherwise.
- nine_k 6y ago
  I would hazard to say that any magic is bad in engineering. (Sorry Perl fans!)
  - laumars 6y ago
    There isn’t really any magic in Perl, just lots of unfamiliar lexicon if you come from more traditional programming languages. Perls problem is it’s history is rooted in command line usage so there’s a tonne of inherited reserved single character variable names and such like that are optimised for keystrokes rather than readability. However you can certainly write Perl programs that don’t follow those older conventions and look more like a modern language.
    - bmn__ 6y ago
      > There isn’t really any magic in Perl http://p3rl.org/guts#Magic-Variables http://p3rl.org/guts#Magic-Variables perl -MDevel::Peek=Dump -mTie::Scalar -e' //g; Dump $_; tie $c => "Tie::StdScalar"; Dump $c; Dump \%ENV ' 2>&1 | grep MAGIC
      - jlokier 6y ago
        That kind of magic in Perl has a different kind of meaning than "magic number/bool/string parsing" in YAML. Perl tied-variable magic just means there are (effectively) getter and setter properties attached to the variable. "Magic" is just the name that was chosen in the implementation, and it stuck. It's used to implement variables with special, automatic meanings, like $$ for "current pid" and $! for "last error". It's also used to implement variables with user-defined behaviours on access, which is quite handy for a lot of abstractions. A lot of modern languages support both of these things, because they are useful, but it's not called magic in those languages, it's called something like "watchers", "proxies", "getters and setters" or "hooks". No, the criticism of YAML-style "magic" is that it leads to entirely surprising behaviour from innocuous input. Perl magic is not that kind. If you're using a special variable, you already know why.
pfraze 6y ago
I think this is an “input format” vs “serialization format” issue. NestedText is meant for user input so it’s focusing more on the UX of the syntax. In that case, having to parse strings is a reasonable task for developers to take on
geofft 6y ago
> The benefit of standard numeric and boolean types is that different tools can exchange data in a well-understood way. In Python: >>> import json >>> json.loads('{"x": 9007199254740993}') {'x': 9007199254740993} In my browser's JavaScript console: > JSON.parse('{"x": 9007199254740993}') {x: 9007199254740992} (Consider what happens if you try to send a tweet's ID, a perfectly normal number like 205052027259195393, through JSON. Or if you try to serialize a stack trace on a 64-bit system, where addresses are also perfectly normal numbers.)
- tmd83 6y ago
  I have very little understanding of JSON, so why would this happen?
  - sergeykish 6y ago
    Python promote to Bignum automatically. Javascript number is a floating point Number.MAX_SAFE_INTEGER // 9007199254740991 Number.MAX_SAFE_INTEGER + 1 // 9007199254740992 Number.MAX_SAFE_INTEGER + 2 // 9007199254740992 no automatic promotion to BigInt 9007199254740991n + 2n // 9007199254740993n
  - auscompgeek 6y ago
    JavaScript numbers are double-precision floating point numbers.
  - DougBTX 6y ago
    The example uses numbers outside the range of JS numbers (too many significant digits), so when the JSON is evaluated as a JS number some of the least significant bits are rounded. Personally, I think it is a mistake to use values in JSON that can’t be represented in JS, but the standard doesn’t explicitly forbid it.
  - zerocrates 6y ago
    It's more a JS thing: The lone JS number type is double-precision floating point, meaning beyond a certain range there are integers that cannot be accurately represented as a JS number. The JSON standard doesn't place restrictions on size or precision of numbers, instead just noting that implementations can vary their treatment of and limits on numbers. While JS uses doubles for all numbers, many other languages emit an integer type for a JSON integer. So, once you go beyond the range where a double can accurately represent all integers, you run the risk of a mismatch in how the number is interpreted by different languages parsing the same JSON. Of course the spec also allows you to create way too big or too precise numbers that would be problematic in most languages as well; it's just that this is a somewhat common bugbear. I wouldn't necessarily call it a flaw in JSON though, more an issue with JSON.parse or really just a fact of life when dealing with numbers in JS. Alternatives to the built in JSON.parse exist to read large integers as strings or bigints.
epicureanideal 6y ago
If NestedText is meant to be written by non-programmer humans, the details of null, undefined, 0, false, "NO", nil, or whatever else, would likely also be lost on them. Most likely if their input is meant to be machine interpreted they would need to be trained to provide specific inputs anyway. I like that NestedText doesn't hide that problem. It lets the user organization decide how it wants to manage that problem, and what symbols or words are understood by the people authoring the files.
deleted 6y ago
[deleted]
GoblinSlayer 6y ago
>you now have 30 tool-specific ways to write "true" and "false" ISO8601 joins the chat.

zamadatix 6y ago

That's not the only way to deal with them, you can specify type like bson. The nice thing about that is it solves the problem rather than hoping it doesn't matter or assuming each program's validator will think to note all of the possible data types not compatible with the program natively. If a u32 is defined in the file and you've only got doubles to work with it's a given you'll have to deal with it in your tool specific validation. For everyone else it's well defined. The downside is it's a bit more verbose and if you have all of that info already it's pretty easy to jump to just using a binary format which will be more efficient anyways.

thayne 6y ago

My ideal config language would have unambigous syntax for at least 64-bit signed integers and doubles (so, for example the spec for the language requires 56 to be parsed as an integer and 56.0 to be a double. Additional types would be ok too, as long as the syntax is unambiguous and obvious.

samatman 6y ago
Both of those are true for TOML. The spec requires that a full 64 bits of signed integer be parsed and understood, that hexadecimal, octal, and binary values be integers, and that floating point values be parsed as doubles. It doesn't support hexadecimal float, however, which is a pity: having a guaranteed bit-identical format is a nice affordance.

Phrodo_00 6y ago

I am of the opposite view: As long as there are no edge cases, supporting more types in data languages is good, it leads to failing faster. Too many types could get overwhelming, but I like where amazon's Ion is [1]. It actually supports multiple number types, with decimal being the default for values with a dot. > you would need to know that when your number needs more than float precision or otherwise doesn’t follow the rules, it should be in quotes Not really. The configuration value should either be a number or not, which is determined by the application reading the config. As a config writer you only care to make the type match (so, in json, if the application uses number you make sure you use number, and if it expects a string you use that) (disclaimer: I work for amazon, but have nothing to do with Ion other than having used it. Opinion is my own, not my employer's, yadda yadda) [1] http://amzn.github.io/ion-docs/ http://amzn.github.io/ion-docs/

jiggawatts 6y ago

> I think avoiding numeric types is a good decision. Only if this format is intended for use-cases that never need to deal with numbers. > One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. That statement couldn't possibly be more wrong. Number parsing (and encoding!) is a decidedly non-trivial problem. You need to concern yourself with -- at a minimum -- all of the following: - Unsigned 64-bit numbers. - A series of digits that would be bigger than a 64 bit whole number. Convert to float? Truncate in some way? Error? - NaN - Infinity - Negative zero - Denormal numbers. - Differentiating between decimal/currency types and floating point numbers. Not all decimal values can be exactly represented as floats! - Efficiently encoding floating point to use the minimum digits without losing precision. - Parsing those minimal numbers with perfect "round-tripping". - Doing the above efficiently. - Securely too! Efficient parsers cut corners on sanity checks. I hoped you fuzzed your parser... The above can easily amount to many kilobytes of extremely complex code. Look up "ryu" as an example of what Google came up with to make JSON number parsing reasonably efficient. Meanwhile, reading a fixed-length number from a binary format can be done in a single machine instruction. One. It might not even take an entire CPU clock cycle! Okay, two, if you need to bounds-check your buffer, but there's ways to avoid that. Afterwards, the bounds check is again literally just two machine instructions in complexity. That's not the difficult bit! The difficult bit is the parsing.

lsorber 6y ago
In my opinion, the best solution to these issues is to: 1. Declare numbers as numbers in the configuration language. E.g. "decimal(1e1000)". 2. Parse declared numbers with a lossless format like Python's decimal.Decimal. 3. Let users decide at their own risk if they want to convert to a lossy format like float.
cycomanic 6y ago
But which of these are problems in configuration files written by a human? That is the aim of the format. Moreover in applications were there could be issues, it would most certainly be tied to very specific fields and you would want specific application logic to handle that field. Now if people misuse it as a data exchange format or so, yes I agree with you, but at that point just use a binary format instead.

barumi 6y ago

> I think avoiding numeric types is a good decision. It tends to eventually cause problems when one implementation (...) It seems you're mixing up the language definition with implementations that try to follow the language definition. If different implementations have different results then either they are buggy or the language has some important holes in the specification. Either way,the solution to this problem is not less validation. > One should remember that any sane application will be parsing the config file into internal data structures and validating it anyway so it gets little benefit from the numbers being already “parsed”. That's the whole point, isn't it? I mean, if you already acknowledge the fact that this parsing and validation is a basic requirement, why handle it as an afterthought and force developers to add their own hand-rollef absurd and unnecessary type checks and type coversions? Wouldn't it simply easier to let the language and the parser do that already? I mean, no one ever complained that JSON had string types. In fact, one of JSON's main complaints is that it doesn't support enough types, such as timestamps.

7952 6y ago

I wonder if you could just include explicit type information? name(str): Dave age(int4): 22 dob(date): 2020-02-01 photo(base64):TWFuIGl....