4 ms·
Huh. I am working on a little utility to deduplicate csvs using sqlite. Seems like this would likely be better. I used sqlite in the projects name. This makes
by nonethewiser 4y ago
Huh. I am working on a little utility to deduplicate csvs using sqlite. Seems like this would likely be better.
I used sqlite in the projects name. This makes me think of an article I read recently that suggested not using descriptive names for projects. For reasons such as this.
- ivvve 4y agoThis is probably dumb intervention but forgive me because it made me giddy to find: the pandas drop_duplicates() method is pretty great, if you're in a pinch. Again -- forgive me if you know about it!
- ALittleLight 4y agoYou may want to look at pandas too (unless the issue is your CSVs are too large to fit in memory).
- karteum 4y agoif the lines are strictly identical, you may just use cat foo.csv | sort | uniq