9 ms·
XMHell: Handling 38GB of UTF-16 XML with Rust
- slaymaker1907 6y agoWhy not just change the encoding to UTF-8? That would probably also reduce the file size since most of it is ASCII, potentially even halving the size.
- zamadatix 6y agoThat's what is done: "Very fortunately for us, we can use a library to take care of all of this for us. The encoding_rs and encoding_rs_io libraries will let us translate from UTF-16 to UTF-8 on the fly, which will let us send ordinary Rust strings into our XML parser."
- xvilka 6y agoUTF-16 is a mistake and should be abolished. Long live UTF-8 [1]! [1] http://utf8everywhere.org/ http://utf8everywhere.org/
- simonw 6y agoIf anyone needs to handle something like this in Python I've had good results using the (sadly under-documented) XMLPullParser class in the Python standard library. I've used it to parse a multiple GB XML file using less than 60MB of RAM - details here: https://github.com/dogsheep/healthkit-to-sqlite/issues/7 https://github.com/dogsheep/healthkit-to-sqlite/issues/7
- guitarbill 6y agoI believe lxml also supports streaming/incremental parsing (https://lxml.de/parsing.html#incremental-event-parsing https://lxml.de/parsing.html#incremental-event-parsing). Nothing wrong with the standard library, but I seem to always turn to lxml for non-trivial XML work.
- masklinn 6y agoThese are part of the stdlib’s elementtree so also present there. And somewhat inconveniently you can’t hand a null builder to iterparse so for large documents you must remember to clear the elements it builds lest you explode your memory.
- raverbashing 6y agoAnything for streaming JSON in Python?
- simonw 6y agoYes, ijson. I figured out how to use that a few days ago: https://github.com/simonw/sqlite-utils/issues/173 https://github.com/simonw/sqlite-utils/issues/173
- StillBored 6y agoThats cute, but why wouldn't something like your favorite database's XML import get you were you want to go without the rust. Presumably that is after all where this data actually comes from. The state has it all in some ancient database, and they opened it up by using the "dump to xml" option. Your favorite database probably has utf8/16 column and/or conversion functions too.
- jstrong 6y agomy guess is it would take me a lot longer to finagle the database to do what I want, but that's me.
- hyperman1 6y agoI assume it depends on what knowledge you have. If you know your DB tooling and have basic XML and ETL experience, StillBored is probably right. If you have to look it all up but write software every day, a custom program is probably faster to get right. You might still want to invest in DB/XML/ETL knowledge if more of this kind of questions come up.
- muststopmyths 6y agoyeah, the source ftp site has a DB schema description right at the top level. You could set up a table to import data into. Then export it out in whatever format you want, or query it however. If the schema changes from one month to another it would be far easier to just update the DB table schema for the next import than change parsing code. Everything can be solved with code, but not everything should be. Interesting as a coding exercise and perhaps that's the point of the article. Not the way I'd do it if I wanted a stable process for consuming the data regularly.
- bawolff 6y agoFtp, multi-gb xml, utf-16. That's a bit of a blast from the past. That said, pretty much every programming lang in existence has sax libraries [since this sort of thing was much more common in the 2000s], many of them will transparently handle the utf-16 encoding, so it doesn't seem like something to make that much of a fuss over.
- bjoli 6y agoI will steal that unicode+book of genesis joke! A question about XML in rust: Does anybody know of any XML library in any language that can do tree-folds instead of stateful callbacks? I used SSAX in scheme and I now becomes sad every time I have to do any complex things using regular SAX parsers with stateful callbacks.
- adamretter 6y agoThis sounds interesting. I haven't heard of "tree-folds" on XML before - do you have any links to further reading and/or examples?
- bjoli 6y agoI have only ever used it in scheme in Oleg Kiselyov's SSAX. Oleg's site: http://okmij.org/ftp/Scheme/xml.html#XML-parser http://okmij.org/ftp/Scheme/xml.html#XML-parser Guile manual https://www.gnu.org/software/guile/manual/html_node/SSAX.html https://www.gnu.org/software/guile/manual/html_node/SSAX.htm... (see that section and the next one describing transformations)
- michaelhkay 6y agoAccumulators in XSLT 3.0 are related to tree folds; certainly they are an attempt to solve the same problem, of providing pure functional (stateless) access to data obtained incrementally through tree navigation.
- bjoli 6y agoAnybody who has gone looking for a good XSLT 3 implementation would probably recognize your name! Thank you for Saxon.
- brundolf 6y agoThat "Rust Evangelism Strike-Force" logo is fantastic. Tucking it away for future use.
- chrismorgan 6y agoHowever, it shouldn’t be 1.5MB. As a web developer, here are some tips on how this could be improved. It’s being served as an 1832×1465 PNG, but on screen it’s capped to 400×320, so the substantial majority of the pixels are wasted. Reducing it to a 2× image, 800×640, still PNG, drops the size by about 1.1MB. If you wanted to get fancier you could go lossy WebP and easily bring it below 50KB, but you’d probably still want the PNG as a fallback for older browsers (and even recent-but-not-quite-current Safari). At these sizes it’s probably not even worth splitting 1× and 2× into different files with srcset, though you could if you wanted to. The full markup with both of these techniques and four image files would be this: <picture> <source srcset="resf.webp, resf-2x.webp 2x" type="image/webp"> <img src="resf.png" srcset="resf-2x.png 2x" width="400" height="320" alt="“Rust Evangelism Strike Force” logo, in vaporwave aesthetics"> </picture> But the first step is the most important, just shrinking the PNG from its original size to the size that you actually want to use it at. The rest is just sugar coating to make things even better, at a certain cost to you the developer if you haven’t got tooling to automate it.
- ysleepy 6y agoI really don't get the supposed problem with this. Steaming XML is a mature thing, libraries in most every language for decades. Oh! you had to add a "UTF-16LE", the suffering is real... How is this hell? Did you want streaming json? it would end up being essentially the same code. YAML has to be it! (Is there a streaming parser?) This totally feels like conjured drama.
- qz2 6y agoCompletely a drama. I was reading 200gb or so xml files about 15 years ago in Java using their event driven xml parser, the name of which I forget. It didn’t use a whole lot of RAM or require any thought. Basically buffer 100 rows into RAM, then process and bulk insert into our SQL box. It just took time to run and most of that was waiting for the DB. A huge chunk of businesses are based on doing exactly this and not a lot more.
- bassman9000 6y agoSAX
- usr1106 6y agoWhat's the fuss over UCS-2? Isn't a fair amount of data processing still done on Windows systems on the user end and in Java in the corporate space? Has either of them ever converted away from 16 bit characters? (Sorry the naive but honest question. Haven't used Java since around 2005 (remember J2ME?) and Windows since 2010.)
- ygra 6y agoNeither Windows, nor .NET, nor Java have moved away from 16-bit code units (the latter two have made concessions for strings in other encodings, either to save space or to improve speed in scenarios where UTF-16 will never appear, like HTTP headers). It's unlikely they ever will; that's a hell of a backwards-compatibility break for very little gain.
- qz2 6y agoYeah also to note that strings in powershell are 16-bit chars.
- f1refly 6y agoI'm not convinced rust was in any way better than other languages here. I'd spin up a quick implementation to proof my point, but the site is timing out for me, so that's probably not going to happen.
- bawolff 6y agoYeah, pretty sure even php would have been much easier.
- Dylan16807 6y ago> As you can imagine, 65,536 characters did not turn out to be nearly enough for all the world’s languages—much less the wave of weird characters yet to appear. Eh, it was nearly enough. There were too many obscure Chinese characters but 17 bits would have fit just about everything. And for things like emoji it's not "much less", it's "a smidgen less", they're quite tiny in comparison. And that timeline is a mess. UTF-8 came before UTF-16.
- bawolff 6y agoWow TIL: UTF-8 goes back to 1992 (!) Which makes the mess that is UTF-16/UCS-2 even more unfortunate because its not like unicode had made all that many in-roads by that point. People were still mostly using various legacy code pages at that point. Imagine if we could have skip all the fuss over wide character encodings (i suppose there was already wide character encodings due to asian language encodings)
- ygra 6y agoWell, at least Windows' use of Unicode goes back to the late 1980s and back then the 16-bitness of Unicode still seemed like a sensible idea and UTF-8 definitely didn't exist yet.
- bawolff 6y agoHmm. This stuff is hard to google, but based on https://docs.microsoft.com/en-us/previous-versions/cc194797(v=msdn.10)?redirectedfrom=MSDN https://docs.microsoft.com/en-us/previous-versions/cc194797(... it looks like windows NT was the first version to support UCS-2 (in 1993), so i think utf-8 did exist at that point in time, but its pretty hard to blame microsoft for going with ucs-2 at that point in time.
- chrismorgan 6y agoI find it in myself to blame them. They introduced a whole new completely incompatible encoding (or two, because of endianness—and that led eventually to the BOM), rather than going with a scheme that would maintain backwards compatibility with ASCII. Even if it was not yet apparent that UCS-2 would be insufficient (not sure about that one way or the other), UCS-2 should have compared unfavourably to UTF-8, with fixed encoding length being its only benefit—and that, of course, was quickly stripped away with the disaster that is UTF-16 that ruined Unicode for everyone with surrogates, and it took about another 15 years before the UCS-2/UTF-16 mess was resolved nearly everywhere.
- hyperman1 6y agoAs XML has gone from everybody's darling to everybodys hate, it might be interesting to tell why we liked it so much in the past. What was the problem it solved? Encoding is the first one: Most programs made assumptions about file encoding. You got an american program and the user wanted parts of it translated/configured in its own language? Encoding problems were almost guaranteed, and you'll have to live with losing some characters. That's in Western Europe, the rest of the world had it worse. When XML came in, you had a good chance of getting UTF-8 or else one of the other UTF variants. Finally. Next up: every program used its own config and data file formats. Or multiple different formats for the same program. You take the manual and learn a new one every 2 weeks. Hooray, XML has only 1 set of rules to learn. Escaping rules were different for each file format, or even incomplete or non-existent. Feeding data form 1 program to another might work most of the time, and some day it breaks because someone, for the first time in 3 years, decided to start a sentence with a space. Hooray, XML lets you escape any text you can think of, yes even PCDATA. As a programmer, we stopped writing parsers for custom formats. Hooray, take any of the shelf XML parser and you're mostly done. Byebye, yacc and flex. The whole schema thing was a godsend. You can validate the basic structure of your files before sending it off to a third party or putting it in a production machine. There are tools to autocomplete or even generate test cases based on schemas. Take that, custom formats from hell. As a bonus, XML was big enough for an entire ecosystem of tooling to grow up around it, most of it cheap or free. I can open an XML file in notepad++, click on 'reformat', have syntax highlighting, validation... In the 90's this was only possible for the most common of formats, and might cost 10 000 $ or more for an extremely limited, buggy tool delivered only by the vendor. So XML was a very big step forward. Things weren't perfect, of course. Spaces, entities, namespaces had enough pitfalls to keep you busy. The fact that he takes an 38GB file and condenses it to 92MB speaks of it verbosity (Note how the author had to sacrifice TAB/CR/LF to get there and invented a basic custom format, i.e. the 90's attitude) . Personally, I'd liked to see JSON extended with comments and dates, a bit better numeric support, and a default usage of schemas. Then let TOML and YAML die in their respective little cradles. It would have left us with exactly 1 format for data transfer and configuration that had XMLs advantages without its verbosity and enterprisyness. But alas, too late, that oportunity has passed.
- 6y ago
- _a1_ 6y agoWith SAX (Simple API for XML) api, both UTF16 to UTF8 conversion and unlimited XML size can be parsed even under 1mb of RAM. It's harder than DOM processing, but SAX was created for dealing with huge amounts of data inside XML.
- adamretter 6y agoThe author is obviously technically capable, but makes some ridiculous/ignorant claims about XML and Character Encoding. Anyone who works with texts today as a programmer who doesn't understand the basics of character encoding is likely in trouble, I find it quite ignorant to claim "Only 90s Kids Will Remember". Also, most modern XML parser will handle/detect the correct encoding for you, so whilst you should understand character encoding, you don't need to do the heavy lifting yourself. There are also inaccuracies in this article, the author says: "Instead, we’ll use a streaming approach (the XML folks call this “SAX”, for reasons I didn’t care enough to research)." Erm... actually there are multiple streaming approaches available. SAX is just one of them (which was pioneered in Java). The author also says: "I decided to go with the quick-xml library because it had good documentation, good performance, and roughly the API I was looking for." Erm... but... quick-xml provides a StAX like API, NOT a SAX API. The former is pull and the latter is push; quite different things, with different performance characteristics depending on your needs. I guess when everything looks like a nail to the author they reach for the rusty hammer. The amount of time and code they put into solving their problem in a limited way is rather staggering. I would have suggested using - the correct tool for the correct job! If you have a small amount of XML (38GB isn't big these days), then work with it as XML. Alternative options would have been to: 1. Load the XML document into a Native XML Database, which takes all of about 10 minutes whilst you make a cup of Tea. There are several Open Source ones if that's a requirement for you. Write probably less than 10 lines of query, and voila you have your CSV. 2. XSLT 3.0 has a streaming mode. You could write a small amount of XSLT 3.0, execute it using an Open Source processor from the command line (e.g. Saxon), and voila you have your CSV. 3. If you wanted to go really fast, and can spare a few dollars, they could have rented a cloud instance with more than enough memory for 1 hour, stuck it all in RAM and processed it quickly using whatever temporary scripting language they liked... If there was an article down-vote button, I would be using it!
- zczc 6y agoXSLT 3.0 streaming with Saxon requires Saxon-EE (closed-source Enterprise Edition), see https://www.saxonica.com/html/documentation/sourcedocs/streaming/xslt-streaming.html https://www.saxonica.com/html/documentation/sourcedocs/strea... Are there Open source xstl 3.0 processors with streaming support?
- lsllc 6y agoInteresting article, I love this quote: > If you ever want proof of Solzhenitsyn’s assertion that “the line separating good and evil passes… right through every human heart”, consider that one man is directly responsible for both UTF-8 and the Go programming language.