8 ms·

I keep seeing people make the same mistake as XML made over and over; without learning from it. I will clarify the problem thusly: > The more capabilities you

I keep seeing people make the same mistake as XML made over and over; without learning from it. I will clarify the problem thusly: > The more capabilities you add to a interchange format, the harder that format is to parse. There is a reason why JSON is so popular, it supports so little, that it is legitimately easy to import. Whereas XML supports attributes, namespaces, CDATA, DTDs, QNames, xml:base, xml:lang, XInclude, etc etc. They gave it everything, including the kitchen sink. There was a thread here the other day about using Sqlite as an interchange format to REDUCE complexity. Look, I love Sqlite, as an application specific data-store. But much like XML it has a ton of capabilities, which is good for a data-store, but awful for an interchange format with multiple producers/consumers with their own ideas. CSV may be under-specified, but it remains popular largely due to its simplicity to produce/consume. Unfortunately, we're seeing people slowly ruin JSON by adding e.g. commands to the format, with others than using those "comments" to hold data (e.g. type information), which must be parsed. Which is a bad version of an XML Attribute.

xienze 7mo ago
> Whereas XML supports attributes, namespaces, CDATA, DTDs, QNames, xml:base, xml:lang, XInclude, etc etc. They gave it everything, including the kitchen sink. But you don't have to use all those things. Configure your parser without namespace support, DTD support, etc. I'd much rather have a tool with tons of capabilities that can be selectively disabled rather than a "simple" one that requires _me_ to bolt on said extra capabilities.
- cbm-vic-20 7mo ago
  As a data interchange format, you can only depend on the lowest commonly implemented features, which for XML is the base XML spec. For example, Namespaces is a "recommendation", and a conformant XML parser doesn't need to support it.
  - smashed 7mo ago
    The problem comes when malicious actors start crafting documents with extra features that should not be parsed, but many software will wrongly parse them because they use the default, full featured parser. Or various combinations of this. It's a pretty well understood problem and best practices exist, not everyone implements them.
- catlifeonmars 7mo ago
  It has the same problem as YAML, there are many, many ways to misconfigure your parser and there lie interesting security vulnerabilities. complex dsls are difficult to implement parsers for. A simple dsl can be implemented in many programming languages very cheaply and can easily be verified against a specification. S-expressions are probably the most trivial language to write parsers for. JSON is also pretty simple, but the spec being underspecified leads to ambiguous parsing (another security issue). In particular: duplicate key handling, key order, and array item order are not specified and different parsers may treat them differently.
- necovek 7mo ago
  If you do not go with DTD or XSD, you are only doing XML lookalike language, as these are XML mechanisms to really define the XML schema: a compliant parser won't be able to validate it, or maybe even to parse it. Thus people go with custom parsers (how hard can it be, right?), and then have to keep fixing issues as someone or other submits an XML with CDATA in or similar.
  - zahlman 7mo ago
    What if we just formalize some reasonable minimal subset, and call it something else?
- pseudohadamard 7mo ago
  The problem with this is that it only works as long as everyone instinctively knows that you don't use all the kitchen-sink stuff. It's there but everyone knows you don't use it because that way insanity lies. And it works more or less OK until someone comes along who doesn't know that you don't use X, and it's in the standard so your implementation isn't standards-compliant and we'll go with your competitor over there instead because unlike you they do support it. And so, over time, all the crap that "everyone knows" you don't use, gets activated and used. Speaking from experience here, not an invented edge case.
conartist6 7mo ago
Just gonna drop this here : ) https://docs.bablr.org/guides/cstml https://docs.bablr.org/guides/cstml CSTML is my attempt to fix all these issues with XML and revive the idea of HTML as a specific subset of a general data language. As you mention one of the major learnings from the success of JSON was to keep the syntax stupid-simple -- easy to parse, easy to handle. Namespaces were probably the feature to get the most rework. In theory it could also revive the ability we had with XHTML/XSLT to describe a document in a minimal, fully-semantic DSL, only generating the HTML tag structure as needed for presentation.
- phlakaton 7mo ago
  I unfortunately disagree that your syntax is "stupid-simple." But it highlights an impedance mismatch between XML users and JSON users. JSON treats text as one of several equally-supported datatypes, and quotes all strings. Great if your data is heavily structured, and text is short and mixed with other types of data. Awful if your data is text. XML and other SGML apps put the text first and foremost. Anything that's not text needs to be tagged, maybe with an attribute to indicate the intended type. It's annoying to express lots of structured, short-valued data. But it's simple and easy for text markup where the text predominates. CSTML at first glance seems to fall into the JSON camp. Quoting every string literal makes plenty of sense in JSON, but not in the HTML/text-markup world you seem to want to play in.
  - conartist6 7mo ago
    Yeah "impedance mismatch" is a good way of putting it. I wouldn't say we fall into the JSON camp at all though, but quite squarely into the XML-ish camp! We just wrap the inner text in quotes to make sure there's no confusion between the formatting of the text stored IN the document and the formatting of the document itself. HTML is hiding a lot of complexity here: https://blog.dwac.dev/posts/html-whitespace/ https://blog.dwac.dev/posts/html-whitespace/. We're actually doing exactly what the author of that detailed investigation recommends. You can see how it plays out when CSTML is used to store an HTML document https://github.com/bablr-lang/bablr-docs/blob/1af99211b2e31f4c6d61b6b0e43469f7af61f1ee/src/content/docs/guides/cstml.cstml#L275-L289 https://github.com/bablr-lang/bablr-docs/blob/1af99211b2e31f.... Having the string wrappers makes it possible to precisely control spaces and newlines shown to the user while also having normal pretty-formatting. Compare this to a competing product SrcML which uses XML containers for parse trees and no wrapper strings. Take a look at the example document here: https://www.srcml.org/about.html https://www.srcml.org/about.html. A simple example is three screens wide because they can't put in line breaks and indentation without changing the inner text!
moron4hire 7mo ago
I consider CSV to be a signal of an unserious organization. The kind of place that uses thousand line Excel files with VBA macros instead of just buying a real CRM already. The kind of place that thinks junior developers are cheaper than senior developers. The kind of place where the managers brow beat you into working overtime by arguing from a single personal perspective that "this is just how business is done, son." People will blithely parrot, "it's a poor Workman who blames his tools." But I think the saying, as I've always heard it used to suggest that someone who is complaining is a just bad at their job, is a backwards sentiment. Experts in their respective fields do not complain about their tools not because they are internalizing failure as their own fault. They don't complain because they insist on only using the best tools and thus have nothing to complain about.
- groundzeros2015 7mo ago
  Boy. Wait until you see how much of the world runs on Unix tabular columns
- PunchyHamster 7mo ago
  Ah, such youthful ignorace. You just classified probably every single bank in existence as "unserious organization"
  - Someone1234 7mo ago
    Yep, healthcare, grocery, logistics, data science. Heck it would be easier to list industries that DON'T have any CSV. There aren't many. In terms of interchange formats these are quite popular/common: EDI (serialized as text or binary), CSV, XML, ASN.1, and JSON are extremely popular. I 100% assure everyone reading that their personal information was transmitted as CSV at least once in the last week; but once is a very low estimate.
  - clhodapp 7mo ago
    They kind of actually are, though. Not because they use CSV's but because, as an industry, they have not figured out how to reliably create, exchange, and parse well-formed CSV's.
GuB-42 7mo ago
I think JSON has the opposite problem, it is too simple, the lack of comments in particular is particularly bad for many common usages of the format today. I know some implementations of JSON support comments and other things, but is is not true JSON, in the same way that most simple XML implementations are not true XML. That's what I say "opposite problem", XML is too complex, and most practical uses of XML use incomplete implementations, while many practical uses of JSON use extended implementations. By the way, this is not a problem for what JSON was designed for: a text interchange format, with JS being the language of choice, but it has gone beyond its design: configuration files, data stores, etc...
- conartist6 7mo ago
  A lot of people dislike that decision not to include comments in JSON, but I think while shocking it was and is totally correct. In a programming language it's usually free to have comments because the comment is erased before the program runs; we usually render comments in grey text because they can't change the meaning of the program. In a data language you have no such luxury. In a data language there's no comment erasure happening between the producer and the consumer, so comments are just dangerous as they would without doubt evolve into a system of annotations -- an additional layer of communication which would then not be standardized at all and which then would grow into a wild west of nonstandard features and compatibility workarounds.
  - blackcatsec 7mo ago
    Could you imagine hitting a rest api and like 25% of the bytes are comments? lol
    - bmacho 7mo ago
      HTML and JS both have comments, I don't see the problem
      - Someone1234 7mo ago
        And both are poor interchange formats. When things stay in their lane, there is no "problem." When you try to make an interchange format using a language with too many features, or comments that people abuse to add parsable information (e.g. "type information") then there is a BIG problem.
PunchyHamster 7mo ago
Constant erosion of data formats into the shittiest DSLs in existence is annoying. "Oh, hey, instead of writing Python, how about you write in * YAML, with magical keywords that turn data into conditions/commands * template language for the YAML in places when that isn't enough * ....Python, because you need to eventually write stuff that ingests the above either way .... ansible is great isn't it?" ... and for some reason others decide "YES THIS IS AWESOME" and we now have a bunch of declarative YAML+template garbage. > There was a thread here the other day about using Sqlite as an interchange format to REDUCE complexity. Look, I love Sqlite, as an application specific data-store. But much like XML it has a ton of capabilities, which is good for a data-store, but awful for an interchange format with multiple producers/consumers with their own ideas. It's just a bunch of records put in tables with pretty simple data types. And it's trivial to convert into other formats while being compact and queryable on its own. So as far as formats go, you could do a whole lot worse.
- gaigalas 7mo ago
  Basic dicts, arrays and templates might be the killer feature set for declarative data languages. If everyone coalesces to those eventually, it means there's something to it.
- 01HNNWZ0MV43FF 7mo ago
  One issue with SQLite is that it's _not_ rewritten every time like JSON and XML, so if you forget to vacuum it or roundtrip it through SQL, you can easily leak deleted data in the binary file.
quotemstr 7mo ago
> XML supports attributes, namespaces, CDATA, DTDs, QNames, xml:base, xml:lang, XInclude, etc etc. They gave it everything, including the kitchen sink. Ah, the old "throw a bag of nouns at the reader and hope he's intimidated" rhetorical flutist. These things are either non-issues (like QName), things a parser does for you, or optional standards adjacent to XML but not essential to it, e.g. XInclude.
- maccard 7mo ago
  > Ah, the old "throw a bag of nouns at the reader and hope he's intimidated" rhetorical flutist. The accusation here is a defleciton. OP's point isn't a gish gallop, it's that xml is absolutely littered with edge cases and complexities that all need to be understood. > optional standards adjacent to XML but not essential This is exactly OP's point. The standard is everything and the kitchen sink, except for all the bits it doesn't include which are almost imperceptible from the actual standard because of how widely used they are.
  - quotemstr 7mo ago
    XInclude isn't part of the standard, and IME, a minority of systems support it anyway. The OP's comment is an obvious gish-gallop. You can assemble a similarly scary noun list for practically any technology. Probably the same kind of person who tries to praise JSON's lack of comments as a feature or something.
- deleted 7mo ago
  [deleted]
- thayne 7mo ago
  > things a parser does for you IME there are two kinds of xml implementations, ones that handle DTDs and entitie definitions for you and are insecure by default (XXE and SSRF vulnerabilities), and ones that don't and reject valid XML documents.
- pocksuppet 7mo ago
  The parser does everything for me. It helpfully loads the external URL in an inline entity definition for me. Oops! All /etc/passwd! There are two kinds of XML parsers: those which are secure and those which are correct.
necovek 7mo ago
Funnily enough, XML was an attempt to simplify SGML so it is easier to parse (as SGML only ever had one compliant parser, nsgml).
- tannhaeuser 7mo ago
  SGML has at least SP/OpenSP, sgmljs, and nsgml as full-featured, stand-alone parsers. There are also parsers integrated into older versions of products such as MarkLogic, ArborText, and other pre-XML authoring suites, renderers, and CMSs. Then there are language runtime libs such as SWI Prolog's with a fairly complete basic SGML parser. ISO 8879 (SGML) doesn't define an API or a set of required language features; it just describes SGML from an authoring perspective and leaves the rest to an application linked to a parser. It even uses that term for the original form of stylesheets ("link types", reusing other SGML concepts such as attributes to define rendering properties). SGML doesn't even require a parser implementation to be able to parse an SGML declaration which is a complex formal document describing features, character sets, etc. used by an SGML document, the idea being that the declaration could be read by a human operator to check and arrange for integration into a foreign document pipeline. Even SCRIPT/VS (part of IBM's DCF and the origin of GML) could thus technically be considered SGML. There are also a number of historical/academic parsers, and SGML-based HTML parsers used in old web browsers.
python-b5 7mo ago
I've been working on an XML parser of my own recently and, to be honest, as long as you're fine with a non-validating parser (which are still compliant), it's really not that bad. You have to parse DTDs, but you don't need to actually _do_ anything with them. Namespaces are annoying but they're not in the main spec. CDATA sections aren't all that useful, but they're easy to parse. As far as I'm aware, parsers don't actually need to handle xml:lang/xml:space/etc themselves - they're for use by applications using the parser. Really the only thing that's been particularly frustrating for me is entity expansion. If you want to support the wider XML ecosystem, with all the complex auxiliary standards, then yes, it's a lot of work, but the language itself isn't that awful to parse. It's a little messy, but I appreciate it at least being well-specified, which JSON is absolutely not.
0xbadcafebee 7mo ago
The problem is that engineers of data formats have ignored the concept of layers. With network protocols, you make one layer (Ethernet), you add another layer (IP), then another (TCP), then another (HTTP). Each one fits inside the last, but is independent, and you can deal with them separately or together. Each one has a specialty and is used for certain things. The benefits are 1) you don't need "a kitchen sink", 2) you can replace layers as needed for your use-case, 3) you can ship them together or individually. I don't think anyone designs formats this way, and I doubt any popular formats are designed for this. I'm not that familiar with enterprise/big-data formats so maybe one of them is? For example: CSV is great, but obviously limited, and not specified all that well. A replacement table data format could be binary (it's 2026, let's stop "escaping quotes", and make room for binary data). Each row can have header metadata to define which columns are contained, so you can skip empty columns. Each cell can be any data format you want (specifically so you can layer!). The header at the beginning of the data format could (optionally) include an index of all the rows, or it could come at the end of the file. And this whole table data format could be wrapped by another format. Due to this design, you can embed it in other formats, you can choose how to define cells (pick a cell-data-format of your choosing to fit your data/type/etc, replace it later without replacing the whole table), you can view it out-of-order, you can stream it, and you can use an index.
- gmueckl 7mo ago
  Some early binary formats followed similar concepts. Look up Interchange File Format, AIFF, RIFF, and their applications and all the file formats using this structure to this day.
  - necovek 7mo ago
    I would say that most of the video file formats today are a bit like that too: they allow different stream data encoding schemes with metadata being the definition of a particular format (mostly to bring up a more familiar example that is not as generic).
- mristin 7mo ago
  Have a look at Asset Administration Shells (AAS) -- it is a data exchange format built on top of JSON and XML (and RDF, and OPC UA and Protobuf, etc.). https://industrialdigitaltwin.org/ https://industrialdigitaltwin.org/ (Disclaimer: I work on AAS SDKs https://github.com/aas-core-works https://github.com/aas-core-works.)
neonstatic 7mo ago
What do you think about Apache Arrow binary formats in this context?

xienze 7mo ago

> Whereas XML supports attributes, namespaces, CDATA, DTDs, QNames, xml:base, xml:lang, XInclude, etc etc. They gave it everything, including the kitchen sink. But you don't have to use all those things. Configure your parser without namespace support, DTD support, etc. I'd much rather have a tool with tons of capabilities that can be selectively disabled rather than a "simple" one that requires _me_ to bolt on said extra capabilities.

cbm-vic-20 7mo ago
As a data interchange format, you can only depend on the lowest commonly implemented features, which for XML is the base XML spec. For example, Namespaces is a "recommendation", and a conformant XML parser doesn't need to support it.
- smashed 7mo ago
  The problem comes when malicious actors start crafting documents with extra features that should not be parsed, but many software will wrongly parse them because they use the default, full featured parser. Or various combinations of this. It's a pretty well understood problem and best practices exist, not everyone implements them.
catlifeonmars 7mo ago
It has the same problem as YAML, there are many, many ways to misconfigure your parser and there lie interesting security vulnerabilities. complex dsls are difficult to implement parsers for. A simple dsl can be implemented in many programming languages very cheaply and can easily be verified against a specification. S-expressions are probably the most trivial language to write parsers for. JSON is also pretty simple, but the spec being underspecified leads to ambiguous parsing (another security issue). In particular: duplicate key handling, key order, and array item order are not specified and different parsers may treat them differently.
necovek 7mo ago
If you do not go with DTD or XSD, you are only doing XML lookalike language, as these are XML mechanisms to really define the XML schema: a compliant parser won't be able to validate it, or maybe even to parse it. Thus people go with custom parsers (how hard can it be, right?), and then have to keep fixing issues as someone or other submits an XML with CDATA in or similar.
- zahlman 7mo ago
  What if we just formalize some reasonable minimal subset, and call it something else?
pseudohadamard 7mo ago
The problem with this is that it only works as long as everyone instinctively knows that you don't use all the kitchen-sink stuff. It's there but everyone knows you don't use it because that way insanity lies. And it works more or less OK until someone comes along who doesn't know that you don't use X, and it's in the standard so your implementation isn't standards-compliant and we'll go with your competitor over there instead because unlike you they do support it. And so, over time, all the crap that "everyone knows" you don't use, gets activated and used. Speaking from experience here, not an invented edge case.

conartist6 7mo ago

Just gonna drop this here : ) https://docs.bablr.org/guides/cstml https://docs.bablr.org/guides/cstml CSTML is my attempt to fix all these issues with XML and revive the idea of HTML as a specific subset of a general data language. As you mention one of the major learnings from the success of JSON was to keep the syntax stupid-simple -- easy to parse, easy to handle. Namespaces were probably the feature to get the most rework. In theory it could also revive the ability we had with XHTML/XSLT to describe a document in a minimal, fully-semantic DSL, only generating the HTML tag structure as needed for presentation.

phlakaton 7mo ago
I unfortunately disagree that your syntax is "stupid-simple." But it highlights an impedance mismatch between XML users and JSON users. JSON treats text as one of several equally-supported datatypes, and quotes all strings. Great if your data is heavily structured, and text is short and mixed with other types of data. Awful if your data is text. XML and other SGML apps put the text first and foremost. Anything that's not text needs to be tagged, maybe with an attribute to indicate the intended type. It's annoying to express lots of structured, short-valued data. But it's simple and easy for text markup where the text predominates. CSTML at first glance seems to fall into the JSON camp. Quoting every string literal makes plenty of sense in JSON, but not in the HTML/text-markup world you seem to want to play in.
- conartist6 7mo ago
  Yeah "impedance mismatch" is a good way of putting it. I wouldn't say we fall into the JSON camp at all though, but quite squarely into the XML-ish camp! We just wrap the inner text in quotes to make sure there's no confusion between the formatting of the text stored IN the document and the formatting of the document itself. HTML is hiding a lot of complexity here: https://blog.dwac.dev/posts/html-whitespace/ https://blog.dwac.dev/posts/html-whitespace/. We're actually doing exactly what the author of that detailed investigation recommends. You can see how it plays out when CSTML is used to store an HTML document https://github.com/bablr-lang/bablr-docs/blob/1af99211b2e31f4c6d61b6b0e43469f7af61f1ee/src/content/docs/guides/cstml.cstml#L275-L289 https://github.com/bablr-lang/bablr-docs/blob/1af99211b2e31f.... Having the string wrappers makes it possible to precisely control spaces and newlines shown to the user while also having normal pretty-formatting. Compare this to a competing product SrcML which uses XML containers for parse trees and no wrapper strings. Take a look at the example document here: https://www.srcml.org/about.html https://www.srcml.org/about.html. A simple example is three screens wide because they can't put in line breaks and indentation without changing the inner text!

moron4hire 7mo ago

I consider CSV to be a signal of an unserious organization. The kind of place that uses thousand line Excel files with VBA macros instead of just buying a real CRM already. The kind of place that thinks junior developers are cheaper than senior developers. The kind of place where the managers brow beat you into working overtime by arguing from a single personal perspective that "this is just how business is done, son." People will blithely parrot, "it's a poor Workman who blames his tools." But I think the saying, as I've always heard it used to suggest that someone who is complaining is a just bad at their job, is a backwards sentiment. Experts in their respective fields do not complain about their tools not because they are internalizing failure as their own fault. They don't complain because they insist on only using the best tools and thus have nothing to complain about.

groundzeros2015 7mo ago
Boy. Wait until you see how much of the world runs on Unix tabular columns
PunchyHamster 7mo ago
Ah, such youthful ignorace. You just classified probably every single bank in existence as "unserious organization"
- Someone1234 7mo ago
  Yep, healthcare, grocery, logistics, data science. Heck it would be easier to list industries that DON'T have any CSV. There aren't many. In terms of interchange formats these are quite popular/common: EDI (serialized as text or binary), CSV, XML, ASN.1, and JSON are extremely popular. I 100% assure everyone reading that their personal information was transmitted as CSV at least once in the last week; but once is a very low estimate.
- clhodapp 7mo ago
  They kind of actually are, though. Not because they use CSV's but because, as an industry, they have not figured out how to reliably create, exchange, and parse well-formed CSV's.

GuB-42 7mo ago

I think JSON has the opposite problem, it is too simple, the lack of comments in particular is particularly bad for many common usages of the format today. I know some implementations of JSON support comments and other things, but is is not true JSON, in the same way that most simple XML implementations are not true XML. That's what I say "opposite problem", XML is too complex, and most practical uses of XML use incomplete implementations, while many practical uses of JSON use extended implementations. By the way, this is not a problem for what JSON was designed for: a text interchange format, with JS being the language of choice, but it has gone beyond its design: configuration files, data stores, etc...

conartist6 7mo ago
A lot of people dislike that decision not to include comments in JSON, but I think while shocking it was and is totally correct. In a programming language it's usually free to have comments because the comment is erased before the program runs; we usually render comments in grey text because they can't change the meaning of the program. In a data language you have no such luxury. In a data language there's no comment erasure happening between the producer and the consumer, so comments are just dangerous as they would without doubt evolve into a system of annotations -- an additional layer of communication which would then not be standardized at all and which then would grow into a wild west of nonstandard features and compatibility workarounds.
- blackcatsec 7mo ago
  Could you imagine hitting a rest api and like 25% of the bytes are comments? lol
  - bmacho 7mo ago
    HTML and JS both have comments, I don't see the problem
    - Someone1234 7mo ago
      And both are poor interchange formats. When things stay in their lane, there is no "problem." When you try to make an interchange format using a language with too many features, or comments that people abuse to add parsable information (e.g. "type information") then there is a BIG problem.

PunchyHamster 7mo ago

Constant erosion of data formats into the shittiest DSLs in existence is annoying. "Oh, hey, instead of writing Python, how about you write in * YAML, with magical keywords that turn data into conditions/commands * template language for the YAML in places when that isn't enough * ....Python, because you need to eventually write stuff that ingests the above either way .... ansible is great isn't it?" ... and for some reason others decide "YES THIS IS AWESOME" and we now have a bunch of declarative YAML+template garbage. > There was a thread here the other day about using Sqlite as an interchange format to REDUCE complexity. Look, I love Sqlite, as an application specific data-store. But much like XML it has a ton of capabilities, which is good for a data-store, but awful for an interchange format with multiple producers/consumers with their own ideas. It's just a bunch of records put in tables with pretty simple data types. And it's trivial to convert into other formats while being compact and queryable on its own. So as far as formats go, you could do a whole lot worse.

gaigalas 7mo ago
Basic dicts, arrays and templates might be the killer feature set for declarative data languages. If everyone coalesces to those eventually, it means there's something to it.
01HNNWZ0MV43FF 7mo ago
One issue with SQLite is that it's _not_ rewritten every time like JSON and XML, so if you forget to vacuum it or roundtrip it through SQL, you can easily leak deleted data in the binary file.

quotemstr 7mo ago

> XML supports attributes, namespaces, CDATA, DTDs, QNames, xml:base, xml:lang, XInclude, etc etc. They gave it everything, including the kitchen sink. Ah, the old "throw a bag of nouns at the reader and hope he's intimidated" rhetorical flutist. These things are either non-issues (like QName), things a parser does for you, or optional standards adjacent to XML but not essential to it, e.g. XInclude.

maccard 7mo ago
> Ah, the old "throw a bag of nouns at the reader and hope he's intimidated" rhetorical flutist. The accusation here is a defleciton. OP's point isn't a gish gallop, it's that xml is absolutely littered with edge cases and complexities that all need to be understood. > optional standards adjacent to XML but not essential This is exactly OP's point. The standard is everything and the kitchen sink, except for all the bits it doesn't include which are almost imperceptible from the actual standard because of how widely used they are.
- quotemstr 7mo ago
  XInclude isn't part of the standard, and IME, a minority of systems support it anyway. The OP's comment is an obvious gish-gallop. You can assemble a similarly scary noun list for practically any technology. Probably the same kind of person who tries to praise JSON's lack of comments as a feature or something.
deleted 7mo ago
[deleted]
thayne 7mo ago
> things a parser does for you IME there are two kinds of xml implementations, ones that handle DTDs and entitie definitions for you and are insecure by default (XXE and SSRF vulnerabilities), and ones that don't and reject valid XML documents.
pocksuppet 7mo ago
The parser does everything for me. It helpfully loads the external URL in an inline entity definition for me. Oops! All /etc/passwd! There are two kinds of XML parsers: those which are secure and those which are correct.

necovek 7mo ago

Funnily enough, XML was an attempt to simplify SGML so it is easier to parse (as SGML only ever had one compliant parser, nsgml).

tannhaeuser 7mo ago
SGML has at least SP/OpenSP, sgmljs, and nsgml as full-featured, stand-alone parsers. There are also parsers integrated into older versions of products such as MarkLogic, ArborText, and other pre-XML authoring suites, renderers, and CMSs. Then there are language runtime libs such as SWI Prolog's with a fairly complete basic SGML parser. ISO 8879 (SGML) doesn't define an API or a set of required language features; it just describes SGML from an authoring perspective and leaves the rest to an application linked to a parser. It even uses that term for the original form of stylesheets ("link types", reusing other SGML concepts such as attributes to define rendering properties). SGML doesn't even require a parser implementation to be able to parse an SGML declaration which is a complex formal document describing features, character sets, etc. used by an SGML document, the idea being that the declaration could be read by a human operator to check and arrange for integration into a foreign document pipeline. Even SCRIPT/VS (part of IBM's DCF and the origin of GML) could thus technically be considered SGML. There are also a number of historical/academic parsers, and SGML-based HTML parsers used in old web browsers.

python-b5 7mo ago

I've been working on an XML parser of my own recently and, to be honest, as long as you're fine with a non-validating parser (which are still compliant), it's really not that bad. You have to parse DTDs, but you don't need to actually _do_ anything with them. Namespaces are annoying but they're not in the main spec. CDATA sections aren't all that useful, but they're easy to parse. As far as I'm aware, parsers don't actually need to handle xml:lang/xml:space/etc themselves - they're for use by applications using the parser. Really the only thing that's been particularly frustrating for me is entity expansion. If you want to support the wider XML ecosystem, with all the complex auxiliary standards, then yes, it's a lot of work, but the language itself isn't that awful to parse. It's a little messy, but I appreciate it at least being well-specified, which JSON is absolutely not.

0xbadcafebee 7mo ago

The problem is that engineers of data formats have ignored the concept of layers. With network protocols, you make one layer (Ethernet), you add another layer (IP), then another (TCP), then another (HTTP). Each one fits inside the last, but is independent, and you can deal with them separately or together. Each one has a specialty and is used for certain things. The benefits are 1) you don't need "a kitchen sink", 2) you can replace layers as needed for your use-case, 3) you can ship them together or individually. I don't think anyone designs formats this way, and I doubt any popular formats are designed for this. I'm not that familiar with enterprise/big-data formats so maybe one of them is? For example: CSV is great, but obviously limited, and not specified all that well. A replacement table data format could be binary (it's 2026, let's stop "escaping quotes", and make room for binary data). Each row can have header metadata to define which columns are contained, so you can skip empty columns. Each cell can be any data format you want (specifically so you can layer!). The header at the beginning of the data format could (optionally) include an index of all the rows, or it could come at the end of the file. And this whole table data format could be wrapped by another format. Due to this design, you can embed it in other formats, you can choose how to define cells (pick a cell-data-format of your choosing to fit your data/type/etc, replace it later without replacing the whole table), you can view it out-of-order, you can stream it, and you can use an index.

gmueckl 7mo ago
Some early binary formats followed similar concepts. Look up Interchange File Format, AIFF, RIFF, and their applications and all the file formats using this structure to this day.
- necovek 7mo ago
  I would say that most of the video file formats today are a bit like that too: they allow different stream data encoding schemes with metadata being the definition of a particular format (mostly to bring up a more familiar example that is not as generic).
mristin 7mo ago
Have a look at Asset Administration Shells (AAS) -- it is a data exchange format built on top of JSON and XML (and RDF, and OPC UA and Protobuf, etc.). https://industrialdigitaltwin.org/ https://industrialdigitaltwin.org/ (Disclaimer: I work on AAS SDKs https://github.com/aas-core-works https://github.com/aas-core-works.)

neonstatic 7mo ago

What do you think about Apache Arrow binary formats in this context?