I managed to "dedupe and merge" by importing the CSV into sqlite and add an index on a column, and then scanning the index by distinct values and doing an index

I managed to "dedupe and merge" by importing the CSV into sqlite and add an index on a column, and then scanning the index by distinct values and doing an index lookup for each key and merging the rows. Not ideal at all, but it seems fast enough. I calculated it will complete in ~50 hours.