3 ms·
The real answer here is take whatever the R data.table package is doing and copy that. It's even written in C. It's by far the best csv reading package I've eve
by nonameiguess 5y ago
The real answer here is take whatever the R data.table package is doing and copy that. It's even written in C. It's by far the best csv reading package I've ever used and will sort as it reads, sometimes extremely fast since it can pretty intelligently guess if a radix sort is possible. The only downside is I believe it can only handle csvs that fit in memory, but memory is getting pretty damn big these days. I used it as inspiration a long while back when I needed to write a library to read in ARPA-formatted n-gram language models, which is a fairly obscure file format I could not find any existing libraries for.
Of course, data.table has existed for over a decade and you're not going to come up with something comparable in 15 minutes, but that's why you'd want to just copy it and not come up with your own solution in the first place.