4 ms·
I learned scraping with python and beautiful soup. My biggest challenge was that on certain sites, the html I would get from requests was different than what wo
by c_o_n_v_e_x 5y ago
I learned scraping with python and beautiful soup. My biggest challenge was that on certain sites, the html I would get from requests was different than what would be seen in chrome.
I tried using selenium to get around this but was never successful. The issue has really handicapped my ability to scrape.
- guskel 5y agoTry faking your User-Agent header.
- banana_giraffe 5y agoThe article touches on that a bit in the "Avoid being blocked" section. Sometimes user-agent isn't enough, I've run into other headers that can trigger a block or a change in behavior. The last one I ran into was gating on accept-language to not control the language, but serve up a honey-pot type page for automated crawlers.
- guskel 5y agoHaha, that's devious but ultimately futile against a targeted scraper.
- shmoogy 5y agoUser agent spoofing, also chrome adjusts/fixes some html so sometimes copying css or xpath directly will not work and requires modification. Good to work in Jupyter notebook locally to test and optimize scrapers
- maltz 5y agoYou should probably take a look at this. https://www.zenrows.com/blog/stealth-web-scraping-in-python-avoid-blocking-like-a-ninja https://www.zenrows.com/blog/stealth-web-scraping-in-python-...