Page size varies, but when I was doing the initial speed tests, pages were around 20k each (guestimate). Yea, the crawler uses libcurl, which supports gzip com

Page size varies, but when I was doing the initial speed tests, pages were around 20k each (guestimate). Yea, the crawler uses libcurl, which supports gzip compressed content. My comcast link is (theoretically) 7Mbit down, 385Kbit up.