3 ms·
I'd like to try compressing DOI urls; what was the file used? https://archive.org/details/doi-urls https://archive.org/details/doi-urls has 11 files; on my mach
by kmike84 11y ago
I'd like to try compressing DOI urls; what was the file used? https://archive.org/details/doi-urls https://archive.org/details/doi-urls has 11 files; on my machine they are 4.3GB combined (after uncompressing), and there is no a file which is 2.8 GB uncompressed.
- burntsushi 11y agoRight. I really should have been better about data provenance, because I know better than that. On my disk, I do indeed see `2007.csv`, `2008.csv`, ..., `2013.csv`. Combined, they are 4.1GB uncompressed. It appears to contain 100,488,054 lines (50,244,026 CSV records). Since I only cared about indexing the URLs, I extracted them from the CSV data (this command just pulls out the second column from the CSV file, which is also one of my creations: https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv --- but a simple Python script should do nicely as well). xsv select 2 20*.csv > urls-unsorted And then remove quotes (because xsv correctly encodes CSV data): sed -i 's/^"\(.\+\)"$/\1/g' urls-sorted Finally, sort and dedupe them: LC_ALL=C sort -u urls-unsorted > urls-sorted I just ran this procedure again, and urls-sorted is byte-for-byte equivalent to the data I used in the blog post. Judging by the timestamps on the original CSV files, I did this around the beginning of August. In fact, I had completely forgotten that I had to do this at all! Sorry about that. When I get a chance, I'll update the blog post.
- kmike84 11y agoThanks for the details! A really nice article, by the way.