3 ms·
I am downloading the file now. I was a little surprised to see that you went with gzip. Given the target audience it is not that hard to imagine that the end u
by dfc 14y ago
I am downloading the file now. I was a little surprised to see that you went with gzip. Given the target audience it is not that hard to imagine that the end user will have access to bzip2 or xz. With the size of the file why choose gzip? Hopefully `gzip -9` was used. I am curious about how much smaller xz/bzip2 will be. I will update this post once the file is downloaded.
- earthrise 14y agoI actually didn't compare gzip and bzip2 before uploading, and I probably should have. I did use `gzip -9`. I'm compressing it with bzip2 -9 now to see if it gets any smaller. I'll post a reply here when I know.
- dfc 14y agoUse xz. Debian, Gnome, Archlinux, Gentoo, Fedora have all switched to xz, and for good reason. I expect xz will offer a 20-30% reduction in file size. That will add up quick.
- earthrise 14y agoThanks, i'll run xz too!
- dfc 14y agoMy download has slowed to a crawl around 90% completion. I am doing 3k a second. When I read that your list was not sorted case sensitively I wondered what difference that may make. I did a little experiment with the american-wordlist-insane wordlist and sorting in the interim. I used msort which does a case sensitive sort[1] and sort which does not. Here are the results: 6830085 insane.msort 1461684 insane.sort.xz 1376548 insane.msort.xz It is not a big difference but it is still a little better. For future releases you may want to do a case sensitive sort. [1] Zebra comes before aardvark
- earthrise 14y agobzip2 saved about 200mb: gzip: 4293 bzip2: 4097 xz is still running