3 ms·

Yeah, it basically is ... We use osmosis to preparse the data and than just parse the XML ... It is a pain. I think what we can try is to import the data in th

Yeah, it basically is ... We use osmosis to preparse the data and than just parse the XML ... It is a pain. I think what we can try is to import the data in the same format they have it in OSM and use smart indexes within the database to issue queries quickly. I think this will be the major investigation when going forward.

robmil 14y ago
Sounds interesting — it would definitely be worth a follow-up post if you do manage to get that working!
michaelt 14y ago
Last weekend I did some work with the OSM planet file - the thing with the XML format is it took several hours just to decompress it - even though I was reading it from RAM on an EC2 m2.2xlarge instance. And after that it still took an age to parse all the XML. All told it took 24 hours just to decompress the file and do a three-pass parse. With the benefit of this experience, I decided it was worth switching to OSM's alternative 'PBF' format [1]. It's a dense binary format that doesn't require additional compression. It's also reportedly 6 times faster to read than gzipped XML. Honestly it seems very complicated to parse, but if you're willing to work with Java or C there's a parser already available. [2] [1] http://wiki.openstreetmap.org/wiki/PBF_Format http://wiki.openstreetmap.org/wiki/PBF_Format [2] https://github.com/scrosby/OSM-binary https://github.com/scrosby/OSM-binary
- klaustopher 14y ago
  Yeah, next time we will use the PBF, but there's currently no ruby parser for this. If we continue to regularly parse the data, we will have to write a wrapper around the C parser. Thanks for the link
  - pygy_ 14y ago
    One of the best protobuf parsers is Haberman's upb. There are no Ruby bindings, but it is built with dynamic languages in mind. There are already Lua and Python bindings, you could use them as example. https://github.com/haberman/upb/wiki https://github.com/haberman/upb/wiki https://github.com/haberman/upb/tree/master/bindings https://github.com/haberman/upb/tree/master/bindings

robmil 14y ago

Sounds interesting — it would definitely be worth a follow-up post if you do manage to get that working!

michaelt 14y ago

Last weekend I did some work with the OSM planet file - the thing with the XML format is it took several hours just to decompress it - even though I was reading it from RAM on an EC2 m2.2xlarge instance. And after that it still took an age to parse all the XML. All told it took 24 hours just to decompress the file and do a three-pass parse. With the benefit of this experience, I decided it was worth switching to OSM's alternative 'PBF' format [1]. It's a dense binary format that doesn't require additional compression. It's also reportedly 6 times faster to read than gzipped XML. Honestly it seems very complicated to parse, but if you're willing to work with Java or C there's a parser already available. [2] [1] http://wiki.openstreetmap.org/wiki/PBF_Format http://wiki.openstreetmap.org/wiki/PBF_Format [2] https://github.com/scrosby/OSM-binary https://github.com/scrosby/OSM-binary

klaustopher 14y ago
Yeah, next time we will use the PBF, but there's currently no ruby parser for this. If we continue to regularly parse the data, we will have to write a wrapper around the C parser. Thanks for the link
- pygy_ 14y ago
  One of the best protobuf parsers is Haberman's upb. There are no Ruby bindings, but it is built with dynamic languages in mind. There are already Lua and Python bindings, you could use them as example. https://github.com/haberman/upb/wiki https://github.com/haberman/upb/wiki https://github.com/haberman/upb/tree/master/bindings https://github.com/haberman/upb/tree/master/bindings