1. The precedent (so far) is scraping is legal if the scraped data is publicly available[A]. 2. I guess the best approach depends on what data you're scraping.

1. The precedent (so far) is scraping is legal if the scraped data is publicly available[A]. 2. I guess the best approach depends on what data you're scraping. Some data it's fine to first convert to plain text, then scrape scrape that. For structured data like tables and HTML, you're better off using the structure of the HTML itself. I suppose you could design a framework that covers all the common tasks, then feed the framework parameters for each site. It's not just handling different sites: the same site will change over time, and there will be oddities between pages/items on the same site. [A]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=scraping%20legal&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...