heres the way to do it incrementally in a way that plays nice with having more text than you have ram, programmatically: first split the file into chunks that

heres the way to do it incrementally in a way that plays nice with having more text than you have ram, programmatically: first split the file into chunks that will comfortably fit in ram, depending on the encoding of the file and the language's default encoding, allow maybe ~2x blow up in memory size, though experimenting is more accurate than guessing. Then sort each of these files in place. then have a program then opens all these smaller sorted files, and does an incremental line by line merge that compares them over all these files, with the case of two lines being equal to drop one, and write the lines that are unique to the result file, and then tada! i actually first dealt with this problem on a programming interview, and I quite liked how instructive an example of out of core programming it was.