3 ms·
Google search team, I wouldn't want to be in your place today. My frustration with Google Search is it's degree of protection against bots. If you, Google, can
by visarga 9y ago
Google search team, I wouldn't want to be in your place today. My frustration with Google Search is it's degree of protection against bots. If you, Google, can spider everything webmasters post, why can't webmasters spider everything Google returns?
In my case, I would like to use Google search to bootstrap a few minor search engine indexes and collect data for NLP projects. But the free version is too limited and the paid version prohibitively expensive, so, no luck.
- fixermark 9y ago> why can't webmasters spider everything Google returns? I'm sure you could start client-side caching every search you ever do to Google. But if you're searching enough to eat up Google's bandwidth, they're paying for that data and they're under no obligation to keep serving you as a client (much as any server is under no particular obligation to serve a search spider).
- trendia 9y ago> But if you're searching enough to eat up Google's bandwidth, they're paying for that data and they're under no obligation to keep serving you as a client Do you not see the irony?
- cbr 9y agoSites allow search engines to index them (instead of telling them to go away with robots.txt) because the search traffic is worth it to them. Search engines don't allow people to scrape them (resorting to blocking after scrapers ignore robots.txt) because they don't get anything similarly valuable in return. (Disclosure: I work for Google, though not on search.)
- fixermark 9y agoNo, I don't. Can you help clarify it for me? Search engines crawling millions of sites each with---on average---a few MB of data distributes cost globally. Extracting terabytes of index data from a single search engine's repository consolidates the cost on the back of that repository's bandwidth provision. These are not symmetrical cost structures.
- hartator 9y agoIt’s exactly what we do: https://serpapi.com https://serpapi.com Shoot me an email, I’ll hook you up with 1,000 credits. It’s funny one of my first use was from ML as well, but it end up being used mostly for SEO and web marketing.