45 ms·
It's really easy to scrape Instagram from outside the browser as a non-user. No API or Access Token needed. Two steps: 1. Get the profile ID, 2. submit query
by 1vuio0pswjnm7 6y ago
It's really easy to scrape Instagram from outside the browser as a non-user. No API or Access Token needed. Two steps: 1. Get the profile ID, 2. submit query via GraphQL.
For example, if the username is "fish" and we want the first 36 entries.
1. x=$(wget -O- -q https://www.instagram.com/fish/?__a=1|grep -o "profilePage_[^\"]*" |sed 's/profilePage_//');
2. wget -O- -q "https://www.instagram.com/graphql/query/?query_hash=e769aa130647d2354c40ea6a439bfc08&variables={\"id\":\"$x\",\"first\":36,\"after\":\"\"}"
- annadane 6y agoDo you enter these two commands sequentially?
- 1vuio0pswjnm7 6y agoYes the profile ID is needed for the GraphQL query
- iggldiggl 6y agoWhile the API access isn't quite as limited as their regular website, don't they eventually block your IP after too many queries? At least that has been my experience when browsing via Imginn – eventually picture loading stalls and in the devtools you can see that in that case the GraphQL requests redirect to a login page instead of returning the desired data. At least that's where having a dynamic IP comes in handy :-P, and with Imginn some minimal fiddling with the page inspector (unhiding the "Load more" button again) lets you retry the query, so your infinite scrolling progress isn't lost.
- 1vuio0pswjnm7 6y agoIn their robots.txt, no crawl delays are given, someone seems to have overlooked that ?__a=2 works just as well as ?__a=1, and overall, whether they intended to or not, they appear to be saying they only want to allow certain bots. Perhaps try changing your user-agent string to match the ones in robots.txt. Generally I do not use middleman sites like Imginn as I find I can perform for myself whatever "service", e.g., downloading resources from some other website, they are offering.