3 ms·
Neat! I worked on a project where CSV parsing was a significant bottleneck (https://openaddresses.io/ https://openaddresses.io/, it processes street addresses).
by NelsonMinar 6y ago
Neat! I worked on a project where CSV parsing was a significant bottleneck (https://openaddresses.io/ https://openaddresses.io/, it processes street addresses). Roughly 50% of the processing in the time is spent parsing CSV and decoding to Unicode. I dinked around a bit and tuned it to read at about 30 MB/second and figured that was good enough. So this is a 100x speedup in reading (assuming all the stars align) which would cut that 50% down to nearly zero. Just as soon as it works transparently with a single Python import statement I'll use it ;-)
- whitten 6y agoWould a change of representation (away from CSV) significantly change parsing speed ?
- ingalls 6y agoThat's a good question! The new V4 platform https://batch.openaddresses.io/data https://batch.openaddresses.io/data outputs data as line delimited GeoJSONs, but haven't gotten time to test throughput in any common tools other than jq