2 ms·
I really should write up how we did delimiter and quote detection in this library: https://github.com/capitalone/DataProfiler https://github.com/capitalone/Dat
by lettergram 5y ago
I really should write up how we did delimiter and quote detection in this library:
https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler
It turns out delimited files IMO are much harder to parse than say, JSON. Largely because they have so many different permutations. The article covers CSVs, but many files are tab or null separated. We’ve even seen @ separated with ‘ for quotes.
Given the above, it should still be possible to use the method described. I’m guessing you’d have to detect the separators and quote chars first, however. You’d have to also handle empty rows and corrupted rows (which happen often enough).
- bob1029 5y ago> It turns out delimited files IMO are much harder to parse than say, JSON. I have to agree, having recently done a deep-dive into the new Utf8JsonReader in .NET. Tracking state in a JSON parsing context can actually be quite elegant. CSV is a minefield of edge cases in my experience.