3 ms·
What's your average page size? Does your crawler support gzip compressed transfers? What speed is your Comcast link?
by natch 18y ago
What's your average page size?
Does your crawler support gzip compressed transfers?
What speed is your Comcast link?
- scumola 18y agoPage size varies, but when I was doing the initial speed tests, pages were around 20k each (guestimate). Yea, the crawler uses libcurl, which supports gzip compressed content. My comcast link is (theoretically) 7Mbit down, 385Kbit up.