4 ms·

The spec doesn't specify what the first line contains. It only says "Semantically a CSJ file is an array of JSON objects which share a common set of keys." You

by dalke 10y ago

The spec doesn't specify what the first line contains. It only says "Semantically a CSJ file is an array of JSON objects which share a common set of keys." You used your knowledge of how these representations usually work to infer that the first line contains a list of keys for the remaining lines. Some obvious questions for a table spec include: 1) must those keys be strings? 2) Are duplicate keys allowed? 3) Can different rows have a different number of columns? Or must the writer generate null fields? FWIW, I've used JSON Lines for my own work. It's very easy to read a line and (in Python), json.loads(line). How should I parse this "JSON without [ and ]"? There's not a built-in function for it. Either I have to write my own JSON tokenizer, or I have to terminate the string with '[' and ']' and parse that. Both are much more complicated than using JSON Lines? You'll notice they only gave output generation timing numbers, not input parsing. If a JSON-based table format is important, why not layer those same table semantics on top of JSON Lines, rather than make a format which is harder to read?

simlevesque 10y ago
The format for the keys is the same as a javascript object. They are unique strings or numbers. It's already in the Ecmascript standard for objects.
- tlrobinson 10y ago
  That doesn't mean it's as easy to parse as plain JSON or JSON Lines. In JS, non-streaming: JSON.parse(blob) Or blob.split("\n").map(line => JSON.parse(line)) respectively.
- dalke 10y ago
  Could you help me understand your response? I find it hard to match your statement to the JSON spec(s). Neither RFC 7159 nor ECMA 404 use the term "key". The standard for objects requires a string for the 'name' of an object member, which is certainly what you refer to. However, JSON does not allow a number. I tried looking at the ECMAScript spec. In http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf http://www.ecma-international.org/publications/files/ECMA-ST... : > An Object is logically a collection of properties. ... Properties are identified using key values . A property key value is either an ECMAScript String value or a Symbol value . That 'Symbol' has nothing to do with numbers, so only strings make sense for an external representation. Perhaps you mean to include the integer from the Array object? The spec says "An Array object is an exotic object that gives special treatment to array index property key ... A String property name P is an array index if and only if ToString(ToUint32(P)) is equal to P and ToUint32(P) is not equal to 2^32-1." Is that limited range of integers what you mean when you saying that numbers are allowed as keys? Regarding uniqueness, the CSJ spec says nothing about uniqueness, so how do you know the keys must be unique? It doesn't inherit that from JSON. RFC 7159 says that in an object "The names within an object SHOULD be unique."; uniqueness is not required. It then adds "When the names within an object are not unique, the behavior of software that receives such an object is unpredictable." Therefore, I don't see how you drew the conclusion you did.
KayEss 10y ago
> rather than make a format which is harder to read Do you really believe that this is harder to read than JSON lines? We use both, and I've never heard a user complain that CSJ is hard to read, but the JSON lines has a lot of clutter. {"age": 45, "name": "Kirit Sælensminde", "job": "Minister Without Portfolio"} {"job": null, "age": 5, "name": "Freyja Sælensminde"} That seems fine for a low number of columns, but gets much harder as the column count goes up. Especially so as you lose control of the key order in the line.
- dalke 10y ago
  The difference between this format and JSON lines is 1) the leading and trailing '['/']' for each line, 2) the convention that each line encodes an array and all arrays have the same length, and 3) the convention that the first line contains string names for the keys for the values on the remaining lines. To clarify, JSON Lines says "Each Line is a Valid JSON Value", "The most common values will be objects or arrays, but any JSON value is permitted." It does not need to encode an object, so there needn't be "a lot of clutter". Here's a sketch of a table parser in Python built on top of JSON Lines using conventions 2) and 3): import json def read_table(infile): line = infile.readline() keys = json.loads(line) for line in infile: yield dict(zip(keys, json.loads(line))) The equivalent for a CSJ parser replaces the two "(line)" occurrences with "('[' + line + ']')". As such, no, it's not hard to write a CSJ parser, but the performance will be slower. I tested it with 3 columns (string, int, int) and 1456021 lines (including the header). Best of 4 for the JSON Lines + convention = 11.3 seconds. Best of 4 for CSJ = 11.8 seconds, which was slower than any of the JSON Lines parser times. Perhaps the 8% slowdown isn't important After all, the uncompressed file size is smaller. #bytes gzip size JSON Lines 34316877 6561939 CSJ 31404835 6576623 savings: ~10% ~0.2% So if the uncompressed savings is important, then use CSJ. But that's a small niche. Anyone concerned about space will use a custom format, while those who need to embed JSON objects or lists aren't going to be worried about 2 extra characters per line.
  - KayEss 10y ago
    Thanks, really interesting to see some numbers. When I parse this on the server I use a composable parser that doesn't need to do the string manipulation you have to in Python. I have no reason to believe that the parsing speed would be different at all with my code. I'd like to know though, so I'll probably do some tests -- that is if I ever get time :) There is one other benefit (which is actually quite important for what we're using it for, but YMMV) and that is that it's very nearly CSV and for many simple files it is the same as CSV. This makes it very familiar looking to users of the system and means that for the most part these files can be easily previewed using a spreadsheet.

by dalke 10y ago

simlevesque 10y ago

The format for the keys is the same as a javascript object. They are unique strings or numbers. It's already in the Ecmascript standard for objects.

tlrobinson 10y ago
That doesn't mean it's as easy to parse as plain JSON or JSON Lines. In JS, non-streaming: JSON.parse(blob) Or blob.split("\n").map(line => JSON.parse(line)) respectively.
dalke 10y ago
Could you help me understand your response? I find it hard to match your statement to the JSON spec(s). Neither RFC 7159 nor ECMA 404 use the term "key". The standard for objects requires a string for the 'name' of an object member, which is certainly what you refer to. However, JSON does not allow a number. I tried looking at the ECMAScript spec. In http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf http://www.ecma-international.org/publications/files/ECMA-ST... : > An Object is logically a collection of properties. ... Properties are identified using key values . A property key value is either an ECMAScript String value or a Symbol value . That 'Symbol' has nothing to do with numbers, so only strings make sense for an external representation. Perhaps you mean to include the integer from the Array object? The spec says "An Array object is an exotic object that gives special treatment to array index property key ... A String property name P is an array index if and only if ToString(ToUint32(P)) is equal to P and ToUint32(P) is not equal to 2^32-1." Is that limited range of integers what you mean when you saying that numbers are allowed as keys? Regarding uniqueness, the CSJ spec says nothing about uniqueness, so how do you know the keys must be unique? It doesn't inherit that from JSON. RFC 7159 says that in an object "The names within an object SHOULD be unique."; uniqueness is not required. It then adds "When the names within an object are not unique, the behavior of software that receives such an object is unpredictable." Therefore, I don't see how you drew the conclusion you did.

KayEss 10y ago

> rather than make a format which is harder to read Do you really believe that this is harder to read than JSON lines? We use both, and I've never heard a user complain that CSJ is hard to read, but the JSON lines has a lot of clutter. {"age": 45, "name": "Kirit Sælensminde", "job": "Minister Without Portfolio"} {"job": null, "age": 5, "name": "Freyja Sælensminde"} That seems fine for a low number of columns, but gets much harder as the column count goes up. Especially so as you lose control of the key order in the line.

dalke 10y ago
The difference between this format and JSON lines is 1) the leading and trailing '['/']' for each line, 2) the convention that each line encodes an array and all arrays have the same length, and 3) the convention that the first line contains string names for the keys for the values on the remaining lines. To clarify, JSON Lines says "Each Line is a Valid JSON Value", "The most common values will be objects or arrays, but any JSON value is permitted." It does not need to encode an object, so there needn't be "a lot of clutter". Here's a sketch of a table parser in Python built on top of JSON Lines using conventions 2) and 3): import json def read_table(infile): line = infile.readline() keys = json.loads(line) for line in infile: yield dict(zip(keys, json.loads(line))) The equivalent for a CSJ parser replaces the two "(line)" occurrences with "('[' + line + ']')". As such, no, it's not hard to write a CSJ parser, but the performance will be slower. I tested it with 3 columns (string, int, int) and 1456021 lines (including the header). Best of 4 for the JSON Lines + convention = 11.3 seconds. Best of 4 for CSJ = 11.8 seconds, which was slower than any of the JSON Lines parser times. Perhaps the 8% slowdown isn't important After all, the uncompressed file size is smaller. #bytes gzip size JSON Lines 34316877 6561939 CSJ 31404835 6576623 savings: ~10% ~0.2% So if the uncompressed savings is important, then use CSJ. But that's a small niche. Anyone concerned about space will use a custom format, while those who need to embed JSON objects or lists aren't going to be worried about 2 extra characters per line.
- KayEss 10y ago
  Thanks, really interesting to see some numbers. When I parse this on the server I use a composable parser that doesn't need to do the string manipulation you have to in Python. I have no reason to believe that the parsing speed would be different at all with my code. I'd like to know though, so I'll probably do some tests -- that is if I ever get time :) There is one other benefit (which is actually quite important for what we're using it for, but YMMV) and that is that it's very nearly CSV and for many simple files it is the same as CSV. This makes it very familiar looking to users of the system and means that for the most part these files can be easily previewed using a spreadsheet.