3 ms·
The author is obviously technically capable, but makes some ridiculous/ignorant claims about XML and Character Encoding. Anyone who works with texts today as a
by adamretter 6y ago
The author is obviously technically capable, but makes some ridiculous/ignorant claims about XML and Character Encoding.
Anyone who works with texts today as a programmer who doesn't understand the basics of character encoding is likely in trouble, I find it quite ignorant to claim "Only 90s Kids Will Remember".
Also, most modern XML parser will handle/detect the correct encoding for you, so whilst you should understand character encoding, you don't need to do the heavy lifting yourself.
There are also inaccuracies in this article, the author says:
"Instead, we’ll use a streaming approach (the XML folks call this “SAX”, for reasons I didn’t care enough to research)."
Erm... actually there are multiple streaming approaches available. SAX is just one of them (which was pioneered in Java).
The author also says:
"I decided to go with the quick-xml library because it had good documentation, good performance, and roughly the API I was looking for."
Erm... but... quick-xml provides a StAX like API, NOT a SAX API. The former is pull and the latter is push; quite different things, with different performance characteristics depending on your needs.
I guess when everything looks like a nail to the author they reach for the rusty hammer. The amount of time and code they put into solving their problem in a limited way is rather staggering.
I would have suggested using - the correct tool for the correct job! If you have a small amount of XML (38GB isn't big these days), then work with it as XML.
Alternative options would have been to:
1. Load the XML document into a Native XML Database, which takes all of about 10 minutes whilst you make a cup of Tea. There are several Open Source ones if that's a requirement for you. Write probably less than 10 lines of query, and voila you have your CSV.
2. XSLT 3.0 has a streaming mode. You could write a small amount of XSLT 3.0, execute it using an Open Source processor from the command line (e.g. Saxon), and voila you have your CSV.
3. If you wanted to go really fast, and can spare a few dollars, they could have rented a cloud instance with more than enough memory for 1 hour, stuck it all in RAM and processed it quickly using whatever temporary scripting language they liked...
If there was an article down-vote button, I would be using it!
- zczc 6y agoXSLT 3.0 streaming with Saxon requires Saxon-EE (closed-source Enterprise Edition), see https://www.saxonica.com/html/documentation/sourcedocs/streaming/xslt-streaming.html https://www.saxonica.com/html/documentation/sourcedocs/strea... Are there Open source xstl 3.0 processors with streaming support?
- adamretter 6y agoThere is a 30-day evaluation license for Saxon-EE which can be used for non-production work, https://www.saxonica.com/download/download.xml https://www.saxonica.com/download/download.xml There is also Exselt http://exselt.net http://exselt.net (appears to be down at the moment), I don't think this is Open Source, but it was freely available for download previously.
- michaelhkay 6y agoIf you've got a few hundred spare weekends, please write one!
- otabdeveloper4 6y agoJeez, the hoops people will jump through just because they can't program! Reminds me of the intricate machinery people build out of Excel spreadsheets and PowerBI. P.S. I'm only half joking.
- qz2 6y agoThe human race are excellent at using the wrong tools for the job. This one always makes me laugh because I walk into this scenario regularly: https://www.neopoleon.com/blog/excel-as-a-database/ https://www.neopoleon.com/blog/excel-as-a-database/
- magicalhippo 6y ago> Anyone who works with texts today as a programmer who doesn't understand the basics of character encoding is likely in trouble And they can create a lot of trouble! We have to deal with a system which generates what they call XML files using simple string concats, without regard for encoding. So while the header says encoding="utf-8", the data is a mix of different encodings depending on which database the data comes from, none of them being UTF-8.
- adamretter 6y agoMany years ago I worked for a BlueChip company in the UK where they had an XML processing application which provided a nation-wide service. When I started, I looked at the code and saw that all XML production was done through concat and substring calls on Java Strings. When I asked why? I was told it was for "performance", yet the system was slow and they frequently introduced bugs in their XML output and processing due to badly written string operations - I didn't stay there all that long!
- michaelhkay 6y agoI flew all the way from London to California once to help a household-name IT company with a perceived XML performance problem, which turned out to be simply that they were constructing the XML document using Java string concatenation rather than using a StringBuilder. In an hour I saved them about $1m worth of hardware spend.
- wodenokoto 6y agoThis is very much a lost art and it is surprisingly difficult to google your way to success. I’ve recently seen other people complain about receiving large xml and having to hack together a parser. Can you write a how-to? Something deeper than “you can do this with XSLT” - it would be very valuable to a lot of people.