4 ms·
Lots of comments here about XML vs. JSON... but there are areas where these two don't collide. I'm thinking about text/document encoding (real annotated text, t
by JodieBenitez 3y ago
Lots of comments here about XML vs. JSON... but there are areas where these two don't collide. I'm thinking about text/document encoding (real annotated text, things like books, etc).
Even though XML is still king here (see TEI and other norms), some of its limitations are a problem. Consider the following text:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Now say you want to qualify a part of it:
Lorem ipsum <sometag>dolor sit amet</sometag>, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Now say you want to qualify another part, but it's overlapping with previous part:
Lorem ipsum <sometag>dolor sit <someothertag>amet</sometag>, consectetur</someothertag> adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Of course, this is illegal XML... so we have to do dirty hacks like this:
Lorem ipsum <start someid="part1"/>dolor sit <start someid="part2"/>amet<end someid="part1"/>, consectetur<end someid="part2"> adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Which means rather inefficient queries afterwards :-/
- samwillis 3y agoYou are absolutely right that XML is better for document structures. My current theory is that Yjs [0] is the new JSON+XML. It gives you both JSON and XML types in one nested structure, all with conflict free merging via incremental updates. Also, you note the issue with XML and overlapping inline markup. Yjs has an answer for that with its text type, you can apply attributes (for styling or anything else) via arbatary ranges. They can overlap. Obviously I'm being a little hypabolic suggesting it will replace JSON, the beauty of JSON is is simplicity, but for many systems building on Yjs or similar CRDT based serialisation systems is the future. Maybe what we need is a YjsSchema... https://github.com/yjs/yjs/ https://github.com/yjs/yjs/
- thejohnconway 3y agoYjs isn’t a document structure is it? It seems to be a library for collaborative editing, but I’m not seeing something suitable for marking up a document, or am I missing something obvious?
- hahn-kev 3y agoYou can add metadata to ranges inside the text.
- deleted 3y ago[deleted]
- j-pb 3y agoI would argue that the inline way of annotating things in XML is actually ok-ish if one absolutely needs human edit-ability, but otherwise bad design. {text: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.", annotations: [{tag: "sometag", ranges: [{from: 12, to: 26}]}, {tag: "sometothertag", ranges: [{from: 21, to: 39}]} Note that this also removes the limitation that annotations have to be consecutive.
- mdaniel 3y agocrafty, but for your consideration: that places the burden upon every library author to be "accounting accurate" to any edits, and the only way anyone would know that it's not correct is to visually inspect the output text also, as I get older I have a deeper and deeper appreciation that "offset" and "text" are words that are fraught with peril
- j-pb 3y agoFair critique, but shouldn't strings be immutable anyways? Once you bring editing into play, you'll probably either want something like a rope or some CRDT and then you have more effective means of tracking positions, than manual offset computations, as part of the data-structure.
- feoren 3y agoWhat about using inline "floating" checkpoints for the ranges, instead of character indexes? {text: "Lorem ipsum {{1}}dolor sit {{2}}amet{{3}}, consectetur{{4}} adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.", annotations: [{tag: "sometag", ranges: [{from: 1, to: 3}]}, {tag: "sometothertag", ranges: [{from: 2, to: 4}]}
- meepmorp 3y agoA strategy I've seen for dealing the inability of XML to handle overlapping tags, is to treat the tagging as an annotation layer on top of the node with the data: <doc> <data type="text"> This is some sample text. </data> <annotations> <tag1 start="1" end="3" comment="foo"/> <tag2 start="2" end="4" type="bar" /> </annotations> </doc> The start and end are usually byte offsets from the start of the text content in the data node. It still sucks, but at least you could apply the same general stragegy to more than just text data - I've seen it used with audio/video where the offsets are treated as time offsets into the media.
- JodieBenitez 3y agoGood idea. You would have to edit your annotation layer in case the text data changes though.
- euroderf 3y agoIt seems klutzy and yet fully in the spirit of XML.
- yencabulator 3y agoNow you've lost human editability/readability and could just as well encode that in a non-XML format.
- meepmorp 3y agoYep. In my experience, the traditional solution for editing with these kinds of hacks is to write a buggy piece of shit custom GUI so people can edit documents. That way, the complaints shift away from your lousy data format to your lousy UI. Problem solved!
- dwaite 3y agoThis is actually one of the things processing instructions are useful for - but you would need to define the data within the PI, since they don't have attributes.