5 ms·
I would write a script in PHP to iterate over the CSV (though most languages would work here). In the first pass, scan for the rows that should be merged. On th
by Minor49er 5y ago
I would write a script in PHP to iterate over the CSV (though most languages would work here). In the first pass, scan for the rows that should be merged. On the second pass, when a merged row is encountered, write it if it hasn't been written yet. If you do this, just be sure to make a demo or two first to ensure that the behavior is what you expect. Also make use of buffering since reading and writing a file line-by-line will probably be very slow compared to reading it in at blocks at a time
- nullbytesmatter 5y agoInteresting idea to do two passes and build a merge list ahead of time. I am not sure if there's enough memory to build up such a list but it's worth a try. Edit: I tried, and OOM'd by the OS. Apparently 64GB of RAM isn't enough to key off the unique column.
- Minor49er 5y agoIf you are finding that many matches that early, it might help to do multiple passes. On the first pass, identify up to X number of duplicates. Once you have X number of duplicates or reach the end of the file, iterate over the CSV and merge those rows. Then start again identifying a new set of duplicates, then apply those updates, over and over, until the file has no more duplicates