Something that I rarely see mentioned: only scrape what you need. If the site provides some means of limiting what it returns to you (a particular path, catego

Something that I rarely see mentioned: only scrape what you need. If the site provides some means of limiting what it returns to you (a particular path, category, or search), take advantage of that. If you know that the information is on a limited number of pages and you only need to acquire it once, there it is perfectly fine to download those pages manually then have your program extract the required information automatically. (That may even save you from some work.)