thery simple really, but so is UNIX :-) parallel -j 1000 -a urls.txt {wget --recursive {}|lynx --dump|(other unix shell power tools) > url.txt} gets you a pow

thery simple really, but so is UNIX :-) parallel -j 1000 -a urls.txt {wget --recursive {}|lynx --dump|(other unix shell power tools) > url.txt} gets you a powerfull web scraper that maxes out your internet connection. Throw some machine learning on top of it and you can do a lot with that. Can't tell you what exactly I did with the ML that being trade secret about to get a patent, but the rest doesn't involve the parallel anyway.