4 ms·
wget -p -r -l 0 -nc -U "I can't believe it's not Googlebot/2.1" -i "$FILE" ... and you wouldn't have.
by textfiles 17y ago
wget -p -r -l 0 -nc -U "I can't believe it's not Googlebot/2.1" -i "$FILE"
... and you wouldn't have.
- jacquesm 17y agoHehe, that's a good trick :) Reminiscent of the Compaq one: BIOS NOT (C) IBM 1982 Or something to that effect! Here is what I use, now modified with your trick: wget -r -nv -np -nc -i $URLFILE With a separate process putting file in batches of 50,000 files. That way you get one wget process to do a boatload of work instead of firing up a new one for every file. That also helps in re-using the connections.
- soult 17y agoLast time I checked, Geocities didn't support HTTP keep-alive. They do however support gzip compression, which wget doesn't support. Also, as you noticed, including "Googlebot" in the User-Agent works around the bw limit. Did you archive regional versions (de.geocities.com, etc.) too? I see that (any-subdomain).reocities.com currently is just an alias for reocities.com. I have ~ 12 GB of archived data from de.geocities.com (couldn't find more links and didn't have the time later on), saved in the same format as wget (mtime set according to Last-Modified header etc.) if you are interested. Anyway, good work!