4 ms·

This comes at the expense of normal users who have to solve three iterations of the puzzle. One co-op term I wrote a simple VBA script to open up Google and se

This comes at the expense of normal users who have to solve three iterations of the puzzle. One co-op term I wrote a simple VBA script to open up Google and search for some addresses. It was a small 2-3 day research project. Well unfortunately for me, I subsequently had to solve captchas every day for the next month and a half, and although about half of them would be instantly solved, when I had to redo CAPTCHAs I could feel my sanity slipping away.

deleted 9y ago
[deleted]
visarga 9y ago
Google is stingy with search results, even though its index exploits the content we all create. They run millions of automated requests on my server - in fact Google is using more CPU than my regular visitors, but I can't run on their servers even a few thousand automated queries. They don't allow even mild automated searching, the price per thousand searches is prohibitive and you can't get past the 1000'th result so you need to use combinations of words and other imperfect means to go deeper. I would have liked to do deep web searches in order to build datasets for ML, but unfortunately there's no search engine (or is too expensive) for bots. It's 2017, years after the Snowden leaks, and we still use Google and don't have a f*#ng search engine for creators, we only have search engine for 'customers'. We need our own search engine for many things - privacy, control over filtering, mass search, deep search, for creating novel projects, to be sure Russians haven't tampered with it, and so on. I'd say Google's refusal to accept bots is akin to a lack of net neutrality and places limits on what kind search based open source projects can be created - I know it sounds entitled, and maybe I should just build my own crawler and index - but you don't solve a more complex problem as a preamble to solving a simpler problem and we don't want thousands or millions of private search engines pulling the same content and duplicating indexes - we still need a community solution. If they love net neutrality, they should support search neutrality as well, for the maker community. It's the same argument as for net neutrality: to protect innovation from incumbents. Search is like the last mile and Google is like a nasty pipe throttling ISP.
- chatmasta 9y ago
  I’ve often felt the same as you, and I think there is room to disrupt google in this area: a search engine with an API, and a search engine that covers the “deep web” (not Tor, but content that is not currently indexed, like a lot of content in apps). FWIW Bing has an API.
- bogomipz 9y ago
  >"They don't allow even mild automated searching, the price per thousand searches is prohibitive and you can't get past the 1000'th result so you need to use combinations of words and other imperfect means to go deeper." Can you elaborate on what you mean by price per 1K searches and a 1K search result limit? Is this a B2B Google offering for search? Is this offering posted somewhere?
  - chatmasta 9y ago
    I think he means that the rate limiting when scraping google gets way more aggressive past the first page.

deleted 9y ago

[deleted]

visarga 9y ago

Google is stingy with search results, even though its index exploits the content we all create. They run millions of automated requests on my server - in fact Google is using more CPU than my regular visitors, but I can't run on their servers even a few thousand automated queries. They don't allow even mild automated searching, the price per thousand searches is prohibitive and you can't get past the 1000'th result so you need to use combinations of words and other imperfect means to go deeper. I would have liked to do deep web searches in order to build datasets for ML, but unfortunately there's no search engine (or is too expensive) for bots. It's 2017, years after the Snowden leaks, and we still use Google and don't have a f*#ng search engine for creators, we only have search engine for 'customers'. We need our own search engine for many things - privacy, control over filtering, mass search, deep search, for creating novel projects, to be sure Russians haven't tampered with it, and so on. I'd say Google's refusal to accept bots is akin to a lack of net neutrality and places limits on what kind search based open source projects can be created - I know it sounds entitled, and maybe I should just build my own crawler and index - but you don't solve a more complex problem as a preamble to solving a simpler problem and we don't want thousands or millions of private search engines pulling the same content and duplicating indexes - we still need a community solution. If they love net neutrality, they should support search neutrality as well, for the maker community. It's the same argument as for net neutrality: to protect innovation from incumbents. Search is like the last mile and Google is like a nasty pipe throttling ISP.

chatmasta 9y ago
I’ve often felt the same as you, and I think there is room to disrupt google in this area: a search engine with an API, and a search engine that covers the “deep web” (not Tor, but content that is not currently indexed, like a lot of content in apps). FWIW Bing has an API.
bogomipz 9y ago
>"They don't allow even mild automated searching, the price per thousand searches is prohibitive and you can't get past the 1000'th result so you need to use combinations of words and other imperfect means to go deeper." Can you elaborate on what you mean by price per 1K searches and a 1K search result limit? Is this a B2B Google offering for search? Is this offering posted somewhere?
- chatmasta 9y ago
  I think he means that the rate limiting when scraping google gets way more aggressive past the first page.