4 ms·

I worked a decade in public sector digitalisation in Denmark, where for some reason they still use a lot of SOAP and thus XML. I’m not so against XML in theory,

by devjab 3y ago

I worked a decade in public sector digitalisation in Denmark, where for some reason they still use a lot of SOAP and thus XML. I’m not so against XML in theory, but I hate it in practice. You’d have these completely over engineered solutions where you’d basically need to call a separate micro services for every “field” of anything. So if you wanted a name, a ssn, and, an address for a citizen you’d need to make several calls. We ended up buying an API for the APIs, but that is besides the point I’m trying to make. The point is that it was very designed, but the XML. Well… you’d have things like: <ssn ssn=“123”/><id>123</id> and worse. And it’s like, why would they spend all that time designing micro-service and data architectures and then output really inconsistent and often even standards breaking XML? But I know why, because out of the maybe 300 different systems to output XML that I’ve worked with over the years, none of them did it “right”. If the future is XML then we better step up our game!

mythhabit 3y ago
> You’d have these completely over engineered solutions where you’d basically need to call a separate micro services for every “field” of anything. So if you wanted a name, a ssn, and, an address for a citizen you’d need to make several calls. The official CPR API you can get all of that information with a single call and I think it returns JSON. I'm not sure when that API was released, but I think it was quite a while ago. There is one for companies as well (CVR API, but technically it's just an Elastic Search query iirc).
olsgaarddk 3y ago
At the tax authority we could query that directly from a database. But yes, the public sector suffers from countless different systems that have accumulated over the years. It happens because everything has to go out to a public bid and throughout the times it has been considered anti competitive to built things in-house.
- TeMPOraL 3y ago
  More than that, the vendor winning the bid will happily use the opportunity to latch itself to the flow of taxpayer money. That's a big reason you end up with XML hell - the systems are designed to make future interop impossible without going through the vendor. In fact, wasn't Denmark a case study of that? I remember there was someone on HN a while ago, IIRC working in public service of some country in Europe, who posted stories about their constant fighting with a vendor who intentionally makes it very hard to have any kind of integration and data exchange between various administrative branches.
fodkodrasz 3y ago
I agree that there is nothing inherently wrong with XML, the problem is with the way it is used. I have worked with great XML schemas, and terrible ones also. Also the XML APIs in the programming languages are a bit dated, and oftentimes a bit cumbersome to use. Yet the standard API to support not only DOM style, but streaming parsing is also a great thing, should you have to process several gigabytes large XML files at one time on your career (as I had to).
- magicalhippo 3y ago
  > I agree that there is nothing inherently wrong with XML, the problem is with the way it is used. Right, but the spec allows for so many different ways of using it, that you end up with lots of variations in how it's done. I've mulled over these choices myself when trying to come up with a XSD spec, and I wish I could slap that idiot who wrote the last XSD spec around with a large trout (that idiot being past me). While you can do a lot of weird things with JSON as well, the syntax is more restrictive and naturally leads to fewer variations.
Traubenfuchs 3y ago
This is not the fault of SOAP/XML at all though? There'd be several ways to offer a soap service that dynamically returns the set of requested fields.
- xorcist 3y ago
  It's the XML equivalent of having SQL tables with two columns, "key" and "value". (I've seen variants of that far too many times for it to be a joke.)
alphager 3y ago
I think "XML" is way too broad to meaningful discuss anything. XML the data format is pretty great in its basic form. Not space-efficient, but unless you're dealing with bulk data, the inefficiencies don't matter unless you're googlescale or can't spare a millisecond. The format itself doesn't absolve you from the design work of your model; your ssn example is typical of the "CEO said we have to use XML now" rush-jobs at the peak of the SOA hypecycle. XML data validation through XSD is a dream. The insistence of the JSON-people to sabotage any kind of data validation is absurd to me. XML Webservices are easy and consuming them from a WSDL is easy. SOA on an enterprise level was a stupid idea creating a giant expensive communication middleware with the ESB without delivering on the promises. Every other thing in the XML space is brainfuck-level overcomplicated bullshit dreamt up by bored enterprise architects. WS-* is a cacophony of overcomplicated solutions to problems that only exist because enterprise architects dreamt of having a single data model that's inter-operable between companies.
- fragmede 3y ago
  > The insistence of the JSON-people to sabotage any kind of data validation is absurd to me. I don't understand what you mean by this. JSON supports schemas which can be used to validate input. It's not super advanced, but it's functional.
  - xnorswap 3y ago
    JSON schema aren't part of the JSON definition, and JSON itself suffers from lacking good handling of types such as dates. JSON itself succeeds as a way to represent JavaScript objects but it is far less useful than XML for representing custom types, which are easily enforced through XML Schema. JSON Schema adds some of this, but it isn't a standard (yet).
    - sacado2 3y ago
      XML schemas aren't part of the XML definition either, although, granted, they're an official recommendation too.
  - 3y ago
ExoticPearTree 3y ago
Not in Denmark only. In some projects across the EU I only saw XML/SOAP used for the so called API access. I really do not understand the XML fetish still going on in 2020+.
- jmhammond 3y ago
  I’m (and many others) are using XML via the PreTeXt[1] project. Because of the strict schema, the processors are able to transpile from xml source to LaTeX/pdf, HTML, epub, and more. The syntax is obnoxious, but I love the results! [1]: https://pretextbook.org/doc/guide/html/part-author.html https://pretextbook.org/doc/guide/html/part-author.html
yohannparis 3y ago
> If the future is XML then we’d better step up our game! I don't think you understood the meaning of the article. It's talking about how XML was, at the end, a red hearing and not best solutions for everything as it was tooted by the industry.
poizan42 3y ago
I also have experience with the digital services of the Danish public sector. Trafikstyrelsen's SOAP service validates its own responses against an xsd. The data in the motor vehicle registry does not always conform to the xsd. In which case it returns a SOAP fault with the original response embedded as an escaped string in the details. So to actually get the response we need to parse it out of the SOAP fault... Another piece of insanity is the e-TL (elektronisk tinglysning) SOAP service. Getting a non-java client generated is a PITA because of the use of the java Catalog feature. Even after hacking around it by setting up a local webserver and liberal abuse of the host file the generated .net WCF client needed manual fixes in a lot of places. After doing that your calls still fails. It turns out the service wants the client to tell it which xsd to validate the request against (?!?). Also did I mention that the service expects every request to be signed, even when only reading information - it doesn't do anything to validate the signature though.
Cthulhu_ 3y ago
But the issues you describe isn't XML's fault, it's poor oversight and lack of consistency from developers. Does anyone have an architect or API designer hat at your company? They should be responsible for designing the XML structure, setting rules on where values go, etc. This is an issue that you will have with every kind of data transfer / representation tech, be it XML, JSON, protobuf, etc. Someone needs to have ownership, someone needs to set a standard, and everyone working on it should stick to that.

by devjab 3y ago

mythhabit 3y ago

> You’d have these completely over engineered solutions where you’d basically need to call a separate micro services for every “field” of anything. So if you wanted a name, a ssn, and, an address for a citizen you’d need to make several calls. The official CPR API you can get all of that information with a single call and I think it returns JSON. I'm not sure when that API was released, but I think it was quite a while ago. There is one for companies as well (CVR API, but technically it's just an Elastic Search query iirc).

olsgaarddk 3y ago

At the tax authority we could query that directly from a database. But yes, the public sector suffers from countless different systems that have accumulated over the years. It happens because everything has to go out to a public bid and throughout the times it has been considered anti competitive to built things in-house.

TeMPOraL 3y ago
More than that, the vendor winning the bid will happily use the opportunity to latch itself to the flow of taxpayer money. That's a big reason you end up with XML hell - the systems are designed to make future interop impossible without going through the vendor. In fact, wasn't Denmark a case study of that? I remember there was someone on HN a while ago, IIRC working in public service of some country in Europe, who posted stories about their constant fighting with a vendor who intentionally makes it very hard to have any kind of integration and data exchange between various administrative branches.

fodkodrasz 3y ago

I agree that there is nothing inherently wrong with XML, the problem is with the way it is used. I have worked with great XML schemas, and terrible ones also. Also the XML APIs in the programming languages are a bit dated, and oftentimes a bit cumbersome to use. Yet the standard API to support not only DOM style, but streaming parsing is also a great thing, should you have to process several gigabytes large XML files at one time on your career (as I had to).

magicalhippo 3y ago
> I agree that there is nothing inherently wrong with XML, the problem is with the way it is used. Right, but the spec allows for so many different ways of using it, that you end up with lots of variations in how it's done. I've mulled over these choices myself when trying to come up with a XSD spec, and I wish I could slap that idiot who wrote the last XSD spec around with a large trout (that idiot being past me). While you can do a lot of weird things with JSON as well, the syntax is more restrictive and naturally leads to fewer variations.

Traubenfuchs 3y ago

This is not the fault of SOAP/XML at all though? There'd be several ways to offer a soap service that dynamically returns the set of requested fields.

xorcist 3y ago
It's the XML equivalent of having SQL tables with two columns, "key" and "value". (I've seen variants of that far too many times for it to be a joke.)

alphager 3y ago

I think "XML" is way too broad to meaningful discuss anything. XML the data format is pretty great in its basic form. Not space-efficient, but unless you're dealing with bulk data, the inefficiencies don't matter unless you're googlescale or can't spare a millisecond. The format itself doesn't absolve you from the design work of your model; your ssn example is typical of the "CEO said we have to use XML now" rush-jobs at the peak of the SOA hypecycle. XML data validation through XSD is a dream. The insistence of the JSON-people to sabotage any kind of data validation is absurd to me. XML Webservices are easy and consuming them from a WSDL is easy. SOA on an enterprise level was a stupid idea creating a giant expensive communication middleware with the ESB without delivering on the promises. Every other thing in the XML space is brainfuck-level overcomplicated bullshit dreamt up by bored enterprise architects. WS-* is a cacophony of overcomplicated solutions to problems that only exist because enterprise architects dreamt of having a single data model that's inter-operable between companies.

fragmede 3y ago
> The insistence of the JSON-people to sabotage any kind of data validation is absurd to me. I don't understand what you mean by this. JSON supports schemas which can be used to validate input. It's not super advanced, but it's functional.
- xnorswap 3y ago
  JSON schema aren't part of the JSON definition, and JSON itself suffers from lacking good handling of types such as dates. JSON itself succeeds as a way to represent JavaScript objects but it is far less useful than XML for representing custom types, which are easily enforced through XML Schema. JSON Schema adds some of this, but it isn't a standard (yet).
  - sacado2 3y ago
    XML schemas aren't part of the XML definition either, although, granted, they're an official recommendation too.
- 3y ago

ExoticPearTree 3y ago

Not in Denmark only. In some projects across the EU I only saw XML/SOAP used for the so called API access. I really do not understand the XML fetish still going on in 2020+.

jmhammond 3y ago
I’m (and many others) are using XML via the PreTeXt[1] project. Because of the strict schema, the processors are able to transpile from xml source to LaTeX/pdf, HTML, epub, and more. The syntax is obnoxious, but I love the results! [1]: https://pretextbook.org/doc/guide/html/part-author.html https://pretextbook.org/doc/guide/html/part-author.html

yohannparis 3y ago

> If the future is XML then we’d better step up our game! I don't think you understood the meaning of the article. It's talking about how XML was, at the end, a red hearing and not best solutions for everything as it was tooted by the industry.

poizan42 3y ago

I also have experience with the digital services of the Danish public sector. Trafikstyrelsen's SOAP service validates its own responses against an xsd. The data in the motor vehicle registry does not always conform to the xsd. In which case it returns a SOAP fault with the original response embedded as an escaped string in the details. So to actually get the response we need to parse it out of the SOAP fault... Another piece of insanity is the e-TL (elektronisk tinglysning) SOAP service. Getting a non-java client generated is a PITA because of the use of the java Catalog feature. Even after hacking around it by setting up a local webserver and liberal abuse of the host file the generated .net WCF client needed manual fixes in a lot of places. After doing that your calls still fails. It turns out the service wants the client to tell it which xsd to validate the request against (?!?). Also did I mention that the service expects every request to be signed, even when only reading information - it doesn't do anything to validate the signature though.

Cthulhu_ 3y ago

But the issues you describe isn't XML's fault, it's poor oversight and lack of consistency from developers. Does anyone have an architect or API designer hat at your company? They should be responsible for designing the XML structure, setting rules on where values go, etc. This is an issue that you will have with every kind of data transfer / representation tech, be it XML, JSON, protobuf, etc. Someone needs to have ownership, someone needs to set a standard, and everyone working on it should stick to that.