3 ms·
Are you planning on open-sourcing the downloader part? I'm very interested.
by pdimitar 8y ago
Are you planning on open-sourcing the downloader part? I'm very interested.
- dredmorbius 8y agoLiterally just a Bash while-read loop over community IDs. It's embarrassingly trivial. I'm planning on posting the data, probably to https://social.antefriguserat.de/ https://social.antefriguserat.de/ and will include procssing scripts. This is the fetch-script, which saves both the HTML and HEAD responses: #!/bin/bash sample_file=$1 comm_path='community-pages' base_url='https://plus.google.com/communities' i=0 time sed -e 's,^.*/,,' $sample_file | while read commid; do i=$((i+1)) echo -e "\n>>> $i $commid <<<" 1>&2; url="${base_url}/${commid}" commfile="${comm_path}/${commid}.html" commhead="${comm_path}/${commid}.head" echo "curl -s -o '${commfile}' -D '${commhead}' '${url}'" done The sample file is simply a list of G+ community IDs or URLs, e.g.: 100000056330101053659 100000310247038604843 100000355641542704509 100000408644688836681 100000537266485621548 100000813948204546252 100001055751908082772 100001158162744298957 100001173291703462139 100001193552641351693
- noisy_boy 8y agoThis seems to be a perfect use case for GNU Parallel[0] to download and process, say 10 ids, in parallel. If you have already downloaded/processed and have no need to do it again, then probably doesn't matter now. [0]: https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/
- dredmorbius 8y agoXargs, actually, though saturating my dinky Internet connection was trivial. Ten concurrencies kept any one request from stalling the crawl though. That's why the script echoed the curl commant rather than run it directly. It fed xargs. The other problem was failed or errored (non 3xx/4xx, or incomplete HTML -- no "</html>" tag found) responses. There was no runtime detection of these. Instead, I checked for those on completion of the first run and re-pulled those in a few minutes, a few thousand from the whole run, most of which ended up being 4xx/3xx ultimately.