3 ms·
XML would fix the max rows issue, but open you up to OOM issues instead!
by secondcoming 6y ago
XML would fix the max rows issue, but open you up to OOM issues instead!
- andor 6y agoSorry, but if you run into OOM issues by parsing an XML file, you're using the wrong API. The DOM for a large XML document will of course take tons of space in memory. The key to parsing XML files quickly and with low memory consumption is to only keep in memory what's necessary, by streaming over the elements. https://en.wikipedia.org/wiki/Simple_API_for_XML https://en.wikipedia.org/wiki/Simple_API_for_XML
- llarsson 6y agoCorrect. And to add to this: apparently the lost data was due to the data that exceeded the 16k rows XLS supports, so the amount of data per file was apparently not huge to begin with. So even a shitty XML parser should do just fine here.