4 ms·
If you've ever tried scraping Facebook you'd know that it's nearly impossible to do so reliably. They have a formidable anti-scrape strategy. Instagram is curre
by katzgrau 8y ago
If you've ever tried scraping Facebook you'd know that it's nearly impossible to do so reliably. They have a formidable anti-scrape strategy. Instagram is currently ridiculously easy to scrape - but I doubt it's going to be that way for long.
- retube 8y agowhat do they do that makes it so hard?
- eastendguy 8y agoThey (randomly?) change IDs and the DOM structure, and thus break scripts. One workaround is to automate visually (KantuX, Sikuli,...) but that is much slower and makes only sense for low-volume scraping.
- epmaybe 8y agoThey randomize the tag ids and classes a lot of the time. At least that has been my experience with the Facebook messenger website.
- slow_donkey 8y agoIf you're doing messenger, there's a community api that emulates the browser on github
- dvh 8y agoGoogle also does it although I suspect it may be accidental when they pass it through gwt or something else. I wanted to read value from simple table in Google developer console only to discover it is 27 levels deep in dom.
- aleph- 8y agoAny links on their anti scrape strategy? Been developing a distributed scraper with baked in proxy per process ability to harvest data for a data project I'm working on. So this would be good to know.
- katzgrau 8y agoI don't think it's something they would publicize otherwise it would be an invite to work around it. The best route might be to try and scrape a public business page (better yet, several hundred) - that'll show you everything you need to know