3 ms·

I'm still on the fence with NUL to be honest... I may remove it, but for now I want to garner discussion. I'm not sure how disallowing a reference for a map ke

I'm still on the fence with NUL to be honest... I may remove it, but for now I want to garner discussion. I'm not sure how disallowing a reference for a map key would cause repetition and space wasting. Keys are generally small, so a reference would in many cases end up taking a similar amount of space. Can you talk more about allowing references as keys? I need different points of view NOW, before I finalize v1 of the format! The negative is that it complicates processing, since you could reference a non-keyable value and attempt to use it in a key. Everybody PLEASE chime in with your thoughts on the format, because I want to make sure I get it as close to right as I can before I finalize V1!

pdimitar 6y ago
As a programmer who had to wrestle with like 30 formats during his career, here's one piece of non-negotiable requirement: ---------- NO. OPTIONAL. PARTS. OF. THE. STANDARD! (And no "but"-s.) ---------- Start there. This might sound flippant and snarky but I promise you that it isn't. Don't ever leave anything to the implementations. Oh, and build your own compliance tool. So people can just run the results of their own implementations against your tool and see if their code is compliant with your data format or not.
- tux3 6y ago
  Votes on comments are not public, so let me agree loudly: Many people will not want to use a format with implementation defined or optional behavior. That way lies pain.
  - kstenerud 6y ago
    Right. No optional stuff then!
grncdr 6y ago
Not the GP but regarding no references for map keys; I can imagine that this would be an issue when representing a list of maps with uniform keys. E.g. a list of 1000 points with the keys “latitude” and “longitude”. I can think of various more compact ways of structuring the same data, but JSON + gzip encodes the naive structure in a very compact way with very little programmer effort. In general, the design seems to miss a compact representation of an array of homogeneous records, which is a common use case IME. Edit: Also I agree with GP regarding NUL and other comments about no optional stuff.
- kstenerud 6y ago
  Hmm yes I see how that could be useful as a kind of "pointer-to-key" setup. A key that is a ref to a non-keyable type would be invalid anyway... I've added array types for arrays of common fixed-length types, but for records it gets complicated since it can only really work if all records are the same size. I could maybe expand array support to custom types for specialized applications that have a lot of record types (provided the custom record type is fixed size). But the primary purpose of the format is a way to allow disparate apps the ability to read each others data in a machine and human friendly way without requiring a bunch of extra pieces. A secondary concern is not trying to be everything for everyone, not complicating the format for too small a gain. But that's the trick, isn't it? (where to draw the line)
- viraptor 6y ago
  I was thinking of use cases like large slightly-heterogenous data dumps. Or structured logs. Let's say you have a few million entries, mostly with the same fields, but with enough differences that using a header + array of arrays is also not great. But! This is totally a specific use case and you don't need to try make everyone happy. There's GELF and others for structured logs already.

pdimitar 6y ago

As a programmer who had to wrestle with like 30 formats during his career, here's one piece of non-negotiable requirement: ---------- NO. OPTIONAL. PARTS. OF. THE. STANDARD! (And no "but"-s.) ---------- Start there. This might sound flippant and snarky but I promise you that it isn't. Don't ever leave anything to the implementations. Oh, and build your own compliance tool. So people can just run the results of their own implementations against your tool and see if their code is compliant with your data format or not.

tux3 6y ago
Votes on comments are not public, so let me agree loudly: Many people will not want to use a format with implementation defined or optional behavior. That way lies pain.
- kstenerud 6y ago
  Right. No optional stuff then!

grncdr 6y ago

Not the GP but regarding no references for map keys; I can imagine that this would be an issue when representing a list of maps with uniform keys. E.g. a list of 1000 points with the keys “latitude” and “longitude”. I can think of various more compact ways of structuring the same data, but JSON + gzip encodes the naive structure in a very compact way with very little programmer effort. In general, the design seems to miss a compact representation of an array of homogeneous records, which is a common use case IME. Edit: Also I agree with GP regarding NUL and other comments about no optional stuff.

kstenerud 6y ago
Hmm yes I see how that could be useful as a kind of "pointer-to-key" setup. A key that is a ref to a non-keyable type would be invalid anyway... I've added array types for arrays of common fixed-length types, but for records it gets complicated since it can only really work if all records are the same size. I could maybe expand array support to custom types for specialized applications that have a lot of record types (provided the custom record type is fixed size). But the primary purpose of the format is a way to allow disparate apps the ability to read each others data in a machine and human friendly way without requiring a bunch of extra pieces. A secondary concern is not trying to be everything for everyone, not complicating the format for too small a gain. But that's the trick, isn't it? (where to draw the line)
viraptor 6y ago
I was thinking of use cases like large slightly-heterogenous data dumps. Or structured logs. Let's say you have a few million entries, mostly with the same fields, but with enough differences that using a header + array of arrays is also not great. But! This is totally a specific use case and you don't need to try make everyone happy. There's GELF and others for structured logs already.