Some major issues from my experience web scraping: 1. Changes in data structures. If some site randomly decides to alter the format of their json/xml objects f

Some major issues from my experience web scraping: 1. Changes in data structures. If some site randomly decides to alter the format of their json/xml objects for their frontend api it may brake your scraper and anything that relies on that scraper’s output. 2. Security controls like rate limiting, captcha, ip blacklisting, auth systems. 3. Html which is rendered via complicated client side JavaScript blobs or web sockets. You’ll need a Headless browser engine like selenium and some site-specific parsing logic. 4. Legal issues.