3 ms·
If anyone needs to handle something like this in Python I've had good results using the (sadly under-documented) XMLPullParser class in the Python standard libr
by simonw 6y ago
If anyone needs to handle something like this in Python I've had good results using the (sadly under-documented) XMLPullParser class in the Python standard library.
I've used it to parse a multiple GB XML file using less than 60MB of RAM - details here: https://github.com/dogsheep/healthkit-to-sqlite/issues/7 https://github.com/dogsheep/healthkit-to-sqlite/issues/7
- guitarbill 6y agoI believe lxml also supports streaming/incremental parsing (https://lxml.de/parsing.html#incremental-event-parsing https://lxml.de/parsing.html#incremental-event-parsing). Nothing wrong with the standard library, but I seem to always turn to lxml for non-trivial XML work.
- masklinn 6y agoThese are part of the stdlib’s elementtree so also present there. And somewhat inconveniently you can’t hand a null builder to iterparse so for large documents you must remember to clear the elements it builds lest you explode your memory.
- raverbashing 6y agoAnything for streaming JSON in Python?
- simonw 6y agoYes, ijson. I figured out how to use that a few days ago: https://github.com/simonw/sqlite-utils/issues/173 https://github.com/simonw/sqlite-utils/issues/173