7 ms·

Things I liked: - Versioning. - Time zone identifier instead of just a fixed offset (which are ambiguous for future events). - Native encoding of binary valu

Things I liked: - Versioning. - Time zone identifier instead of just a fixed offset (which are ambiguous for future events). - Native encoding of binary values. - Graph notation with support for labels. - Comments! - Trying to escape lookalike characters, even though I think that's a lost cause. Things I'm not so keen about: - NUL character in strings being platform and settings dependent. - Line break not being forced to a consistent value. - The most complicated number encoding scheme I've ever seen (e.g. 0xa,3fb8p+42). - Entity references are a footgun for anyone writing a depth-first or breadth-first algorithm. - Arrays-vs-list feels like it doesn't belong in encoding formats.

profquail 4y ago
The complicated number encoding scheme you mentioned is a hexfloat: C has them too. Hexfloat can be really useful when you need precise/exact floating-point constants for numerical methods. Without them, you end up having to do more-complicated hacks to preserve exact constant values when code gets compiled, or you have to live with compilers (sometimes) subtly altering constants. I wish more languages supported hexfloats.
- BoppreH 4y ago
  Thanks, I had never seen that before. But Concise Encoding did complicate them further by accepting commas as decimal separator.
- unwind 4y ago
  Just to clarify a bit what "C has them" means; the standard library string-to-float parsing function strtof() [1] support this format. You can also use them as floating-point literals in your code. [1]: https://linux.die.net/man/3/strtof https://linux.die.net/man/3/strtof
- formerly_proven 4y ago
  Hexfloat is also vastly easier to parse from/to IEEE 752 format than decimal floats.
lifthrasiir 4y ago
> Time zone identifier instead of just a fixed offset (which are ambiguous for future events) Time zone identifiers are not future-proof. As a concrete example imagine a fictitious country X with the largest city (not necessarily the capital) being Foo and a single time zone throughout the country. A natural time zone identifier would be something like `Region/Foo`. If a part of X switches to another time zone and Bar is the largest city in that region for whatever reason, there would be another time zone identifier `Region/Bar` assigned. You can't automatically determine which `Region/Foo` should be converted to `Region/Bar`. It is up to you to pick a strategy in such cases, but time zone identifiers themselves do not solve the ambiguity problem. Besides from this false sense of security, time zone identifiers were not meant to be for general interchange and there are several understandable but problematic assignments. Baking them into a data structure format doesn't sound a good idea.
- BoppreH 4y ago
  That's a valid scenario where time zone identifiers don't work well, but what's the alternative? A fixed offset is guaranteed to be wrong on DST switch, and that happens a lot more often than time zones splintering. I guess GPS coordinates would be more accurate, but then you have privacy concerns. Fun fact: time zones are named after cities for robustness: Country names are not used in this scheme, primarily because they would not be robust, owing to frequent political and boundary changes. The names of large cities tend to be more permanent. Usually the most populous city in a region is chosen to represent the entire time zone, although another city may be selected if it is more widely known, and another location, including a location other than a city, may be used if it results in a less ambiguous name. https://en.wikipedia.org/wiki/Tz_database#Location https://en.wikipedia.org/wiki/Tz_database#Location
  - lifthrasiir 4y ago
    You do need time zone identifiers or something like that, but they are not something you can blindly buy in. I mean, of course it would be great to see a semi-automated solution for that (for example, such a solution can automatically determine Region/Foo is split into Region/Foo and Region/Bar but nothing else)! But the time zone database doesn't provide anything like that, and a built-in support for time zone identifiers can hugely mislead users.
samatman 4y ago
Re. time zones, I'm of the increasing conviction that they should be avoided whenever possible, and it usually is. I think of a time zone as drawing a large box on a map, for any given instant in time. Future events either happen 'only in time' or they happen at a time and place. A time zone isn't a great representation of a place! I'll be much happier storing UTC and coordinates, then turning that into a timezone for display when I have to. A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage.
- sparsely 4y ago
  > A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage. That approximate and weird system is how humans tend to think about future events though. You're right that often it is not necessary, but for many usecases (e.g. when a TV show will be first broadcast in the future) they are the most robust model we have.
  - XorNot 4y ago
    Storing a set of coordinates for the timezone though would be more useful and future proof. The general case is a range which is the timezone bound, but full accuracy means you can store much more specific data.
    - samatman 4y ago
      Exactly. There is one case where time zones are both absolutely necessary and work as expected with no exceptions, and that case is displaying local time. My assertion is that a very strong argument should be won before they are used for any other purpose.
      - josephg 4y ago
        I arrived 1 hour late for a zoom meeting a few weeks ago. At meeting at which I was presenting. I checked and double checked the calendar event to make sure I arrived at the right time. Daylight savings time hadn't changed for me recently, and nor had it changed for the organizers (who advertised the event in GMT). The problem was that the calendar advertising the recurring event was set to PST. And daylight savings had changed in California (PST -> PDT or the other way around). The result was that the calendar event shifted by 1 hour local time for me and every other attendee who subscribed to the calendar event. This situation sucks, and its really confusing to everyone. But I can't think of a way around this whole issue: - If recurring calendar events weren't set to a timezone (and thus, just worked off GMT), then all recurring calendar events would drift forwards or backwards when daylight savings changes happen - If recurring calendar events are set to some local time zone, then things like this happen - which really confuses everyone involved. I mean, sure - in an ideal world we'd get rid of time zones. But until then, it seems like we're stuck with this problem. (Though at least daylight savings time is slowly being phased out in some countries.)
kstenerud 4y ago
NUL is a tricky beast even under ideal conditions. In the first draft I'd simply forbidden them. Simple, decisive, and of course people didn't like that. then I did allow it, and suddenly the security gates were flung open. Some platforms simply cannot support nuls in strings, so now it becomes platform dependent (bad from a security perspective). So then my next attack vector was to make it configurable with a default of deny. Better from a security perspective, but now it's complicated. TBH there are nights where I think I should just go back to no NULs at all again. But I really need more discussion with stakeholders to figure out what problems this solves and how important it really is... Line break is another sticky issue. Non-technical Windows users will inevitably produce documents that use CRLF, so if it rejects the file, that's a bad user experience. What's the best trade-off here? I'm not really sure. The number encoding thing is being discussed in other comments so I'll leave it be. I had allowed comma because that's how a huge chunk of the world represents the decimal separator. Once again, this is about user experience with the non-technical users. I'm REEEEEEEALLY on the fence with this one. Entity references are dangerous, yes, but also powerful. The point of them in the format is to solve the recursive reference problem, because you just can't do that otherwise, and these structures do exist in the world. It's another case of an imperfect solution for an imperfect world. Bear in mind I absolutely do NOT want this format to become some Turing complete language. This is just the minimal feature set I could think of to represent real world data. Arrays-vs-list is another one of those compromises. Encoding an actual array of fixed types into a list would be slow and bulky, leading people to just encode them as a chunk of bytes like they currently do in JSON and other formats. As an imperfect solution to an imperfect world, I want to at least let people preserve the semantic meaning of what they're sending since they're going to use array encodings regardless of what the format supports.
- BoppreH 4y ago
  Thanks for your reply! It's a really good proposal, and I'll definitely consider it next time I'm serializing something. > NUL is a tricky beast One alternative I found interesting is Modified UTF-8 (https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8), where NUL is encoded as 0xC0 0x80. It's already in use in Java, apparently. Disclaimer: I've never heard about it before, and I don't know how well supported it is. > Line breaks and commas I'm from a country that uses comma as decimal separator, and I still prefer dots when programming. I dread ambiguous numbers like "1,001". I'm confident this is true for almost all technical people. And I really don't see non-technical users editing this kind of file. If they can be trusted to read it, and to modify it without introducing syntax errors, they can be trusted to use dots and an editor that shows LF as linebreaks (i.e. anything but `notepad`). My choice would be dots only for decimal separator, accept LF and CR LF on reading, but prefer LF when writing. > Entity references You can always define manual ids: c1 { "alice"={friends=["bob" "charlie"]} "bob"={friends=["alice"]} "charlie"={friends=["alice"]} } Of all data types that could have native encodings (colors, IP addresses, lat/lon coordinates, enums, tags, markdown, hashes, file permissions, OIDs, DOIs, ISBNs, etc), I think arbitrary object graphs bring too many downsides for a serialization format. You don't want reader CVE's because of Billion Laughs, and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.

profquail 4y ago

The complicated number encoding scheme you mentioned is a hexfloat: C has them too. Hexfloat can be really useful when you need precise/exact floating-point constants for numerical methods. Without them, you end up having to do more-complicated hacks to preserve exact constant values when code gets compiled, or you have to live with compilers (sometimes) subtly altering constants. I wish more languages supported hexfloats.

BoppreH 4y ago
Thanks, I had never seen that before. But Concise Encoding did complicate them further by accepting commas as decimal separator.
unwind 4y ago
Just to clarify a bit what "C has them" means; the standard library string-to-float parsing function strtof() [1] support this format. You can also use them as floating-point literals in your code. [1]: https://linux.die.net/man/3/strtof https://linux.die.net/man/3/strtof
formerly_proven 4y ago
Hexfloat is also vastly easier to parse from/to IEEE 752 format than decimal floats.

lifthrasiir 4y ago

> Time zone identifier instead of just a fixed offset (which are ambiguous for future events) Time zone identifiers are not future-proof. As a concrete example imagine a fictitious country X with the largest city (not necessarily the capital) being Foo and a single time zone throughout the country. A natural time zone identifier would be something like `Region/Foo`. If a part of X switches to another time zone and Bar is the largest city in that region for whatever reason, there would be another time zone identifier `Region/Bar` assigned. You can't automatically determine which `Region/Foo` should be converted to `Region/Bar`. It is up to you to pick a strategy in such cases, but time zone identifiers themselves do not solve the ambiguity problem. Besides from this false sense of security, time zone identifiers were not meant to be for general interchange and there are several understandable but problematic assignments. Baking them into a data structure format doesn't sound a good idea.

BoppreH 4y ago
That's a valid scenario where time zone identifiers don't work well, but what's the alternative? A fixed offset is guaranteed to be wrong on DST switch, and that happens a lot more often than time zones splintering. I guess GPS coordinates would be more accurate, but then you have privacy concerns. Fun fact: time zones are named after cities for robustness: Country names are not used in this scheme, primarily because they would not be robust, owing to frequent political and boundary changes. The names of large cities tend to be more permanent. Usually the most populous city in a region is chosen to represent the entire time zone, although another city may be selected if it is more widely known, and another location, including a location other than a city, may be used if it results in a less ambiguous name. https://en.wikipedia.org/wiki/Tz_database#Location https://en.wikipedia.org/wiki/Tz_database#Location
- lifthrasiir 4y ago
  You do need time zone identifiers or something like that, but they are not something you can blindly buy in. I mean, of course it would be great to see a semi-automated solution for that (for example, such a solution can automatically determine Region/Foo is split into Region/Foo and Region/Bar but nothing else)! But the time zone database doesn't provide anything like that, and a built-in support for time zone identifiers can hugely mislead users.

samatman 4y ago

Re. time zones, I'm of the increasing conviction that they should be avoided whenever possible, and it usually is. I think of a time zone as drawing a large box on a map, for any given instant in time. Future events either happen 'only in time' or they happen at a time and place. A time zone isn't a great representation of a place! I'll be much happier storing UTC and coordinates, then turning that into a timezone for display when I have to. A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage.

sparsely 4y ago
> A time zone is an approximate (and weird) spacial coordinate system which causes nonlinearities in the representation of time, and unless you want both of those properties it comes with baggage. That approximate and weird system is how humans tend to think about future events though. You're right that often it is not necessary, but for many usecases (e.g. when a TV show will be first broadcast in the future) they are the most robust model we have.
- XorNot 4y ago
  Storing a set of coordinates for the timezone though would be more useful and future proof. The general case is a range which is the timezone bound, but full accuracy means you can store much more specific data.
  - samatman 4y ago
    Exactly. There is one case where time zones are both absolutely necessary and work as expected with no exceptions, and that case is displaying local time. My assertion is that a very strong argument should be won before they are used for any other purpose.
    - josephg 4y ago
      I arrived 1 hour late for a zoom meeting a few weeks ago. At meeting at which I was presenting. I checked and double checked the calendar event to make sure I arrived at the right time. Daylight savings time hadn't changed for me recently, and nor had it changed for the organizers (who advertised the event in GMT). The problem was that the calendar advertising the recurring event was set to PST. And daylight savings had changed in California (PST -> PDT or the other way around). The result was that the calendar event shifted by 1 hour local time for me and every other attendee who subscribed to the calendar event. This situation sucks, and its really confusing to everyone. But I can't think of a way around this whole issue: - If recurring calendar events weren't set to a timezone (and thus, just worked off GMT), then all recurring calendar events would drift forwards or backwards when daylight savings changes happen - If recurring calendar events are set to some local time zone, then things like this happen - which really confuses everyone involved. I mean, sure - in an ideal world we'd get rid of time zones. But until then, it seems like we're stuck with this problem. (Though at least daylight savings time is slowly being phased out in some countries.)

kstenerud 4y ago

NUL is a tricky beast even under ideal conditions. In the first draft I'd simply forbidden them. Simple, decisive, and of course people didn't like that. then I did allow it, and suddenly the security gates were flung open. Some platforms simply cannot support nuls in strings, so now it becomes platform dependent (bad from a security perspective). So then my next attack vector was to make it configurable with a default of deny. Better from a security perspective, but now it's complicated. TBH there are nights where I think I should just go back to no NULs at all again. But I really need more discussion with stakeholders to figure out what problems this solves and how important it really is... Line break is another sticky issue. Non-technical Windows users will inevitably produce documents that use CRLF, so if it rejects the file, that's a bad user experience. What's the best trade-off here? I'm not really sure. The number encoding thing is being discussed in other comments so I'll leave it be. I had allowed comma because that's how a huge chunk of the world represents the decimal separator. Once again, this is about user experience with the non-technical users. I'm REEEEEEEALLY on the fence with this one. Entity references are dangerous, yes, but also powerful. The point of them in the format is to solve the recursive reference problem, because you just can't do that otherwise, and these structures do exist in the world. It's another case of an imperfect solution for an imperfect world. Bear in mind I absolutely do NOT want this format to become some Turing complete language. This is just the minimal feature set I could think of to represent real world data. Arrays-vs-list is another one of those compromises. Encoding an actual array of fixed types into a list would be slow and bulky, leading people to just encode them as a chunk of bytes like they currently do in JSON and other formats. As an imperfect solution to an imperfect world, I want to at least let people preserve the semantic meaning of what they're sending since they're going to use array encodings regardless of what the format supports.

BoppreH 4y ago
Thanks for your reply! It's a really good proposal, and I'll definitely consider it next time I'm serializing something. > NUL is a tricky beast One alternative I found interesting is Modified UTF-8 (https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8), where NUL is encoded as 0xC0 0x80. It's already in use in Java, apparently. Disclaimer: I've never heard about it before, and I don't know how well supported it is. > Line breaks and commas I'm from a country that uses comma as decimal separator, and I still prefer dots when programming. I dread ambiguous numbers like "1,001". I'm confident this is true for almost all technical people. And I really don't see non-technical users editing this kind of file. If they can be trusted to read it, and to modify it without introducing syntax errors, they can be trusted to use dots and an editor that shows LF as linebreaks (i.e. anything but `notepad`). My choice would be dots only for decimal separator, accept LF and CR LF on reading, but prefer LF when writing. > Entity references You can always define manual ids: c1 { "alice"={friends=["bob" "charlie"]} "bob"={friends=["alice"]} "charlie"={friends=["alice"]} } Of all data types that could have native encodings (colors, IP addresses, lat/lon coordinates, enums, tags, markdown, hashes, file permissions, OIDs, DOIs, ISBNs, etc), I think arbitrary object graphs bring too many downsides for a serialization format. You don't want reader CVE's because of Billion Laughs, and you don't want to force every programmer writing a simple traversal algorithm to correctly handle cycles.