Right. I really should have been better about data provenance, because I know better than that. On my disk, I do indeed see `2007.csv`, `2008.csv`, ..., `2013.

Right. I really should have been better about data provenance, because I know better than that. On my disk, I do indeed see `2007.csv`, `2008.csv`, ..., `2013.csv`. Combined, they are 4.1GB uncompressed. It appears to contain 100,488,054 lines (50,244,026 CSV records). Since I only cared about indexing the URLs, I extracted them from the CSV data (this command just pulls out the second column from the CSV file, which is also one of my creations: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv --- but a simple Python script should do nicely as well). xsv select 2 20*.csv > urls-unsorted And then remove quotes (because xsv correctly encodes CSV data): sed -i 's/^"\(.\+\)"$/\1/g' urls-sorted Finally, sort and dedupe them: LC_ALL=C sort -u urls-unsorted > urls-sorted I just ran this procedure again, and urls-sorted is byte-for-byte equivalent to the data I used in the blog post. Judging by the timestamps on the original CSV files, I did this around the beginning of August. In fact, I had completely forgotten that I had to do this at all! Sorry about that. When I get a chance, I'll update the blog post.