6 ms·
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search resu
by userbinator 8y ago
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results.
300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obscure things. It seems almost hostile to those searching for IC part numbers, specific and very exact phrases, and just "non mainstream" content in general.
How sad it is then, that we are told and have internalised the notion that we should use search engines like Google to find things, and yet it prevents us from "trying too hard" to find what we're looking for...
- jardah 8y agoYes it is, and since it's IP based, it's even easier if you are for example working from an office and there are multiple people using google. But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D
- buzer 8y ago> But that is why they only show recaptcha, you fill it in and you will get extemption cookie for 30 more requests :D Does that actually work? Whenever I have been searching some obscure things and managed to get the captcha after 6-10 pages, it just goes to loop where it keeps giving it constantly. Though it stops giving it if I change the search terms.
- confounded 8y agoWith a VPN on Brave on iOS, Google will only show me infinite captchas.
- dx034 8y agoThe rule set must be more complex. I often use VPN which results in captchas on many pages but I never get one on Google. I guess the 300 queries/IP only count if other parameters indicate crawling.
- jardah 8y agoYea, it was a very general example, since there is at least one rule that is based on rate limiting too, and this 300/IP limit is what have seen on average.
- cm2187 8y agoBut it's a bit clunky. I was running searches through an embedded webbrowser in a c# application, which is really an embedded internet explorer and was very quickly presented with a captcha. It was a human viewing the results, but a script constructing the query string, but that was enough the be labelled as a crawler.
- jocoda 8y agooften all it takes is to be using webbrowser. I would love to use webbrowser for small search utilities because it's so easy to use, but it seems to be a magnet for problems.
- nerdponx 8y agoUse DDG. It's fine for most things. Use Google as a fallback.
- drewmol 8y ago>Use Google as a fallback. To do this from DDG, prepend your search using !g to search Google
- epanchin 8y agoWith a search as you type feature (Google instant?) that limit could be hit in a few searches...
- 0x00000000 8y agoInstagram is the worst I have come across. If you are on a page with 1000+ pictures trying to find something near the bottom, you have to let it load each new group sequentially, then after a while it starts timing you out for like 60 seconds or longer every couple times you load more. God forbid you accidentally navigate away while scrolling you have to start all over again from the top. Due to recent events it seems they got scared, locked down their API, then tightened down the request limit to prevent scraping to the point it is hardly usable on desktop anyway.
- textmode 8y agoThis sounds like an issue that is specific to Javascript-controlled browsers. If using a traditional, non-Javascript tcp/tls/http client it is trivial to extract the image urls and other information from the page using a single HTTP request (and from each successive page using more HTTP requests in a single connection, if "has_next_page" is "true"). No "API" needed. Can you provide an example of a single page with 1000+ images?
- 0x00000000 8y agohttps://www.instagram.com/ryuji513 https://www.instagram.com/ryuji513 It looks like it just hits https://www.instagram.com/graphql/query/.. https://www.instagram.com/graphql/query/... every time you scroll down so if you scroll too fast it just hammers it and throttles your requests to that endpoint.
- deleted 8y ago[deleted]
- textmode 8y ago1. Fetch 1st page. Note id of user (e.g. 1954202703). this is the "id": value in the url. Note end_cursor. This is used for the "after": value in the url Note rhx_gis. This is used to create the "X-Instagram-GIS:" header. Looking at archive.org, it seems as recently as last year, end_cursor was once all that was needed. 2. Fetch js from ProfilePageContainer url in 1st page (e.g., https://www.instagram.com/static/bundles/base/ProfilePageContainer.js/d63ea6652d48.js https://www.instagram.com/static/bundles/base/ProfilePageCon...) Note queryId (e.g. 42323d64886122307be10013ad2dcc44) This is used for "query_hash" in the url. 3. Create header "X-Instagram-GIS:" Apparently this is some MD5 hash of rhx_gis and the query string variables according to this source: https://www.diggernaut.com/blog/how-to-scrape-pages-infinite-scroll-extracting-data-from-instagram/ https://www.diggernaut.com/blog/how-to-scrape-pages-infinite... However a little experimentation revealed that generation of rhx_gis or this hash must also incorporate the user-agent string -- change any character in the user-agent string and the request will fail. They also put IP address and a Unix time value in a cookie but the cookie can be deleted and the request still succeeds. For example the final url for the first 12 photos is: https://www.instagram.com/graphql/query/?query_hash=42323d64886122307be10013ad2dcc44&variables={"id":"1954202703","first":12,"after":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"} https://www.instagram.com/graphql/query/?query_hash=42323d64... Overall, seems not too much work for someone who really wants to automate retrieval of Instagram photos. These requests for successive groups of 12 can be RFC 2616 pipelined over a single connection. Not long ago and for some number of years, it was even easier (e.g. just use end_cursor value as "max_id" in url).
- greglindahl 8y agoFrom my experience at blekko, 99.9% of the "people" who go deep into the results pages for a single query are actually bots. You're a very unusual user, and there are a lot of bots.
- danmg 8y agoI run up against this all the time. My browser is fast from blocking all third party trackers and scripts. My searches are faster, so it thinks they're automated.
- xenomachina 8y agoThere's a difference between going deep into the results, and progressively refining a query. The former is pretty indicative of bot behavior -- humans rarely go past even the first page of results. I do the latter all the time, and this frequently gets me Google's captcha, especially if I'm doing something like using site: and inurl: operators.
- reitanqild 8y agoI've managed to trigger Google's bot detector too while trying to find documentation for a certain bank api (legitimate reasons, we were supoosed to integrate and their docs didn't make sense).
- avip 8y agoJust be a proud false positive.
- duxup 8y agoOften when I fire up my VPN I get CAPTCHAed on the first search.