3 ms·
I was just seeing someone else's tutorial on scraping Amazon prices. They also ran into an issue where they needed to scrape twice instead of once. Not sure tha
by intellijdd 6y ago
I was just seeing someone else's tutorial on scraping Amazon prices. They also ran into an issue where they needed to scrape twice instead of once. Not sure that it's the same issue you're facing but I thought I might drop my two cents.
- catwind7 6y agoI actually did notice that issue, even with using a stateful client like mechanize. Sometimes I had to scrape > 5 times in order to get through the "anti robot" page. Other times, I get no issue at all. It's weird - maybe they're doing some pattern matching on request metadata on their end?