3 ms·
Last weekend I did some work with the OSM planet file - the thing with the XML format is it took several hours just to decompress it - even though I was reading
by michaelt 14y ago
Last weekend I did some work with the OSM planet file - the thing with the XML format is it took several hours just to decompress it - even though I was reading it from RAM on an EC2 m2.2xlarge instance. And after that it still took an age to parse all the XML. All told it took 24 hours just to decompress the file and do a three-pass parse.
With the benefit of this experience, I decided it was worth switching to OSM's alternative 'PBF' format [1]. It's a dense binary format that doesn't require additional compression. It's also reportedly 6 times faster to read than gzipped XML. Honestly it seems very complicated to parse, but if you're willing to work with Java or C there's a parser already available. [2]
[1] http://wiki.openstreetmap.org/wiki/PBF_Format http://wiki.openstreetmap.org/wiki/PBF_Format
[2] https://github.com/scrosby/OSM-binary https://github.com/scrosby/OSM-binary
- klaustopher 14y agoYeah, next time we will use the PBF, but there's currently no ruby parser for this. If we continue to regularly parse the data, we will have to write a wrapper around the C parser. Thanks for the link
- pygy_ 14y agoOne of the best protobuf parsers is Haberman's upb. There are no Ruby bindings, but it is built with dynamic languages in mind. There are already Lua and Python bindings, you could use them as example. https://github.com/haberman/upb/wiki https://github.com/haberman/upb/wiki https://github.com/haberman/upb/tree/master/bindings https://github.com/haberman/upb/tree/master/bindings