8 ms·
A Journey building a fast JSON parser and full JSONPath
- tomthe 3y agoI like the "Simple Encoding Notation" (SEN) of the underlying library: https://github.com/ohler55/ojg/blob/develop/sen.md https://github.com/ohler55/ojg/blob/develop/sen.md " A valid example of a SEN document is: { one: 1 two: 2 array: [a b c] yes: true } "
- koito17 3y agoAn interesting observation: if you move the colon on the opposite side then you get valid EDN data! {:one 1 :two 2 :array [a b c] :yes true} cf. https://github.com/edn-format/edn https://github.com/edn-format/edn Likewise, commas are considered whitespace. They are sometimes added to make lengthy maps easier to read.
- kubanczyk 3y ago> Which is the same as the following JSON: { "one": 1, "two": 2, "array": ["a", "b", "c"], "yes": true } That example also caught my attention, but in a bad way. It looks just like a comeback of one of the worst ideas of YAML. My immediate question would be what's the JSON for this SEN I've crafted: { array: [string1 string2 "true" true True TRUE yes y] } For more fun, there's a single problematic entry here, can you spot it?: 1.20.4 1.204.4 1.20 1.204 1.20.0 1.20.00 1.20-rc2 Or, level expert, there's exactly one problem here as well: 0a1f 0bfd 0c0c 0d01 0e02
- tomthe 3y agoThank you for thinking more deeply about this than I did! But I do not see a problem in your first example, only true is the true true (according to my browser and the linked definition on https://www.json.org https://www.json.org) I don't get your other examples, can you explain? I assumed that 1.20.4 is not a valid SEN entry, because it starts with a digit but is not a number.
- kubanczyk 3y agoMy bad, you are right about 1.20.4 (and also my 0e02 example wouldn't be a valid SEN value). The true/"true" leaves a bad taste in my mouth after yaml, but overall SEN is an improvement :)
- jarym 3y agoI'm not following: ` { array: [string1 string2 "true" true True TRUE yes y] } ` Doesn't look like a valid SEN or JSON. The `y` `yes`, `True`, TRUE` aren't valid keywords/variables/consts and `string1` and `string2` look like variable references which aren't something SEN or JSON support. The closest valid thing I can imagine is: ` { array: ["string1" "string2" "true" true "True" "TRUE" "yes" "y"] } `
- mjpa86 3y agoaren't they implied strings? If "[a b c]" is an array of 3 strings, "a", "b" and "c", then True is a string "True". That's the problem.
- jarym 3y agoI must be missing why you think they're implied strings - I don't see that in the spec. What I do see is: "Strings can also be delimited with a single quote character which allows for a string to be either "abc" or 'abc'." There's no mention of having a string without a delimeter.
- ReleaseCandidat 3y agoThe example below is this: > array: [a b c]
- jarym 3y agoohhh I see it now, that looks like a recipe for... issues.
- pjc50 3y agoLet me guess: 0e02 is interpreted as floating point?
- k_process 3y agoDitto 1.20, and when interpreted as floating point the trailing zero loses significance. So as a version this is indistinguishable from 1.2
- lazyasciiart 3y agoAm I missing something about the definition of “tokenStart”? It can be ‘letter’ or three other characters: but all those other characters (and more) are already in the definition of ‘letter’?
- pjc50 3y agoSee the comment upthread about S-expressions, but .. given that this doesn't have a marker for "atom" which it badly needs, isn't it strictly worse than S-expressions.
- ithkuil 3y agoreminder of recent efforts at standardizing JSONPath: https://datatracker.ietf.org/wg/jsonpath/about/ https://datatracker.ietf.org/wg/jsonpath/about/
- kubanczyk 3y agoTIL that JSONPath was not killed by JSONPointer: > Note that while JSON Pointer (RFC 6901) is already standardised, it is designed to provide a reference to a single, specific part of a JSON document, whereas JSONPath provides the ability to query a document and potentially return multiple values.
- baz00 3y agoIs JSON XML yet? Nearly! I’m going to invent Baz’s 11th law of computing here: any data format that isn’t XML will evolve into a badly specified version of XML over time.
- heresie-dabord 3y agoCorollary: The number (N) of ad hoc support tools needed to do any serious work with a given mark-up language is proportional to the naivety of the implementation (Y).
- baz00 3y agoI like this one a lot.
- kevingadd 3y agoWith respect for the pain everyone has suffered through due to XML... at this point I prefer XML with a good schema to JSON any day, even if it's more verbose and more awkward to hand-edit. It's just so much easier to validate it or generate code to handle it, and you get things like XSLT or XPath if you want them.
- shuiling 3y ago[dead]
- deepakarora3 3y agoNice work! I see that that this is for processing / parsing large data sets and where documents do not conform to a fixed structure and for Go language. I made something similar in Java - unify-jdocs - https://github.com/americanexpress/unify-jdocs https://github.com/americanexpress/unify-jdocs - though this is not for parsing - it is more for reading and writing when the structure of the document is known - read and write any JSONPath in one line of code and use model documents to define the structure of the data document (instead of using JSONSchema which I found very unwieldy to use) - no POJOs or model classes - along with many other features. Posting here as the topic is relevant and it may help people in the Java world. We have used it intensively within Amex for a very large complex project and it has worked great for us.
- latchkey 3y agoWe all know the builtin golang JSON parser is slow. How about doing comparisons against other implementations? Like this one: https://github.com/json-iterator/go https://github.com/json-iterator/go Update: found this outdated repo: https://github.com/ohler55/compare-go-json https://github.com/ohler55/compare-go-json
- pekim 3y agooj comes out very well in that comparison, in terms of both features and performance. Although as you mentioned, it's outdated, not having been updated for over 2 years.
- pstuart 3y agoSlightly tangential, but Go's JSON handling has long had room for improvement and it looks like there's going to be a serious overhaul of its capabilities and implementation: https://github.com/golang/go/discussions/63397 https://github.com/golang/go/discussions/63397 -- I'm looking forward to seeing this land.