3 ms·
Put into a database, most databases can import CSV files directly. From there you can start create SELECT queries and depending how much processing you need to
by tored 5y ago
Put into a database, most databases can import CSV files directly.
From there you can start create SELECT queries and depending how much processing you need to do you can create intermediate views for multiple steps. After that your can export the data directly to CSV.
https://www.mysqltutorial.org/import-csv-file-mysql-table/ https://www.mysqltutorial.org/import-csv-file-mysql-table/
https://www.mysqltutorial.org/mysql-export-table-to-csv/ https://www.mysqltutorial.org/mysql-export-table-to-csv/
https://www.sqlitetutorial.net/sqlite-import-csv/ https://www.sqlitetutorial.net/sqlite-import-csv/
https://www.sqlitetutorial.net/sqlite-export-csv/ https://www.sqlitetutorial.net/sqlite-export-csv/
- nullbytesmatter 4y agoI managed to "dedupe and merge" by importing the CSV into sqlite and add an index on a column, and then scanning the index by distinct values and doing an index lookup for each key and merging the rows. Not ideal at all, but it seems fast enough. I calculated it will complete in ~50 hours.
- nullbytesmatter 4y agoI was wrong, it took around 5 days, but it finished!