I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it a

I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it at the time and it did allow me to deduplicate using Python and no external dependencies.