4 ms·
Listhub has its own problems -- specifically when MLS'es update data, Listhub has to ensure that all of the applications that use its data are updating their in
by imroot 11y ago
Listhub has its own problems -- specifically when MLS'es update data, Listhub has to ensure that all of the applications that use its data are updating their information as well. Listhub is also one HUGE xml file that is a pain in the ass to process. (We explored with a python based parser that chunked the file up into smaller listings and then imported that into Elasticsearch; but ultimately gave up and resorted into a C based parser that threw everything into MongoDB.
- icebraining 11y agoI'm curious, did you look into streaming XML parsing, StAX-style? It's not hard to process infinitely-long files with constant memory: http://stackoverflow.com/questions/9809469/python-sax-to-lxml-for-80gb-xml/9814580#9814580 http://stackoverflow.com/questions/9809469/python-sax-to-lxm...