6 ms·
Show HN: A Python Script to Download Thousands of Wallpapers at Once
- kpenc 8y agoEvery function has side effects. Could've written the whole thing in one function. Also, PEP8.
- apetresc 8y agoOne function? Could've written the whole thing in one `wget` call.
- pknerd 8y agoteach me how.
- shubb 8y agoI guess wget has a useful spidering function that could probably page through the websites search results page, downloading all the preview and real images. You'd have to do the login bit as a different call and fetch the first search pages url yourself?
- apetresc 8y agoHere's a start: for x in $(curl -s https://alpha.wallhaven.cc/random | pcregrep -o1 "https://alpha.wallhaven.cc/wallpaper/(\d+)" | sort | uniq) ; do wget "https://wallpapers.wallhaven.cc/wallpapers/full/wallhaven-$x.jpg" ; done
- akx 8y agobash, curl, pcregrep, sort, uniq and wget is not what I'd call "one wget call".
- slenk 8y agoTo the hardcore bash users, I think they call that 'easy'. I have someone like that on my team - holy crap some of the bash stuff they can come up with
- zo1 8y agoI hope none of that stuff makes it into your main codebase? If there is even a remote chance of requiring maintenance by someone other than the person that wrote it, that is. Which is probably 99.99% the chance, unless it's temporary/throwaway code.
- lou1306 8y agoHardcore bash sure sounds like fun, but when things start getting too big or messy I usually find a Python script with some `subprocess` [0] tricks to be way easier on the eye. [0]: https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html
- mynewtb 8y agoEvery code has side effects. Splitting code into appropriate functions makes those side effects much more managable.
- jfz 8y agoUsing i and j in loops makes code much less self-explanatory, especially when you could use "page_index" and "image_index" instead.
- lou1306 8y agoEven better, replacing for i in range(len(imgid)): and similar lines with: for i, img in enumerate(imgid): would allow one to get rid of all these list accessors.
- myroon5 8y agoIt looks like this downloads them one-by-one instead of downloading at the same time. Is there a simple Python equivalent to Java's .parallelStream().forEach() that would allow these calls to easily be run in parallel?
- mynewtb 8y agoSure thing... from multiprocessing import Pool with Pool(8) as p: p.map(function, sequence))
- pddubs 8y agoYou could just use threads here too - CPython releases the GIL during IO. from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor as executor: executor.map(function, sequence)
- myroon5 8y agoNeat. I see that you explicitly specify 8 as the number of processes here. In Java, parallelStream() will pick a sane default for you if you haven't previously specified (based on the number of available processors, I believe). Is something like that possible in Python?
- jstarfish 8y agoThe ThreadPool class picks a sane default (number of cores), but I believe it uses Python threads instead of processes.
- shubb 8y agoYou might want to async the downloading part. It is often faster to download 5 images at once than to download them one after another. In the choices section you use input("text") properly one place but not others. You use a couple different ways of decoding codes, and the dict way is nicer, but also consistency is nice. Also, I am not sure you handle bad input (default all?)... Personally, I'd pull the meat out of the loops in main() into functions - GetImageList() and GetImage(). Relatively complex there so it would be easier to read and spot errors in those bits of code in isolation.
- ASalazarMX 8y agoUnless you're in dire need of thousands of wallpapers (most of them you are going to delete anyway) it's better not to hammer the website. I'd even limit the download rate.
- ejdanderson 8y agoAgreed - I looked at the code hoping for a good example of async in python as suggested by the “at once” in the title.
- slenk 8y agoSide question - are there any websites like wallhaven, but with less people and anime? I'm thinking the type of content Chromecast uses, or /r/TechnologyPorn?