3 ms·
The IP-level block you experienced after trying to get around the login modal is an anti-scraping measure. I doubt it is based on anything like “detecting that
by fwlr 3y ago
The IP-level block you experienced after trying to get around the login modal is an anti-scraping measure. I doubt it is based on anything like “detecting that you’ve opened web inspector”, it’s much more likely that Instagram is running some heuristic that tries to bucket you into either “potential signup” or “scraper trying to download as many images as possible”, and once your behavior pattern fits into the second bucket they apply a (very) temporary IP block.
It comes off as extreme and insulting to a regular user, but it is also one of the few methods that scrapers can’t trivially dodge (it’s fairly easy to maintain a roster of IPs, but that is merely easy, while e.g. changing your user agent to pretend to be a different browser is trivial).
- roughly 3y agoInsta definitely uses IP blocks to force login walls, but I don’t think it’s anti-scraping, because the limit seems to be like 3 images before you get redirected to log in - it’s way too low and too much of a pain in the ass to normal users to be intentionally anti-scraping and unintentionally anti-logged-out-users.
- fwlr 3y agoThe fact that it pushes anonymous viewers to log in or sign up is definitely a positive (for their conversion rate metric, if nothing else), and that might be the main reason the number of images is so low. But the specific mechanism of IP blocking that is used to achieve restriction - that is anti-scraping, pure and simple.
- mhuffman 3y ago... except, that "pro" scrapers use parallelized calls from rolling ip addresses. It is one of the only ways to scrape things like Youtube... so I hear!
- fwlr 3y agoNobody is going to be able to stop motivated, well-resourced, professional scrapers. IP blocks are effective for stopping most trivial scraping.