12 ms·
Google Deactivates Web Search API
- woodruffw 11y agoSince I can't post both a title and text: What's interesting about this is that the API has been officially deprecated since 2010 (and "offline" since September 2014), but today is the first day that it's actually become unavailable. Edit: Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, etc) were using this API because of its simplicity (and absence of registration/authentication).
- bhouston 11y agoThis is actually the second time they have reduced their Web API for search results. Google used to offer a very full featured REST API, but then took it offline and replaced it with a weak AJAX one: http://googlesystem.blogspot.ca/2008/04/google-search-rest-api.html http://googlesystem.blogspot.ca/2008/04/google-search-rest-a... You could actually go cool things with the original API. I used it to power a predecessor to https://friskr.com https://friskr.com, but we had to switch web search engines after that change.
- woodruffw 11y agoThat's interesting, I had no idea. It's a shame that the trend at Google has been towards closing down APIs like these in favor of specialized/pay-as-you-go endpoints.
- incongruity 11y agoI feel like that's a larger trend. APIs are introduced to drive platform growth and then pulled or cut back once that growth has been achieved. Twitter is another example of that phenomenon.
- tokenizerrr 11y ago> Additionally, there are probably going to be big repercussions to the web/average person's browsing experience as a result of this. A massive number of sites (and programs, bots, etc) were using this API because of its simplicity (and absence of registration/authentication). Yep. My harmless little IRC bot can no longer Google search anymore :(
- fixermark 11y agoWhy not? There's a custom search API. https://developers.google.com/custom-search/json-api/v1/overview https://developers.google.com/custom-search/json-api/v1/over...
- throwanem 11y agoYes, and it's scoped by either domain or (rarely used) types from the schema.org ontology. You can't use it to just search the web.
- fixermark 11y agoYou can use it to search the web; you don't get the exact same experience as the old API. https://support.google.com/customsearch/answer/2631040?hl=en https://support.google.com/customsearch/answer/2631040?hl=en
- 11y ago
- xbryanx 11y agoCould you provide an example or two of the sites that will be impacted, and how it will effect an average user? I'm not that familiar with this area.
- JustUhThought 11y ago#DuckDuckGoLooksUnprofessional It seems pety, but I really feel the duck look is holding it back.
- jlblatt 11y agoThis should be (and is currently) downvoted but it did make me look up DDG's API.
- supergreg 11y agoHow so? It doesn't look that different from Google or Bing. Less clutter even makes it look better.
- ocdtrekkie 11y agoYou mean like the original Google home page? With it's clashing bright colored logo and terrible font use? http://www.jaygeiger.com/index.php/2007/01/21/original-google-homepage/ http://www.jaygeiger.com/index.php/2007/01/21/original-googl...
- soared 11y agoNo, Google had a nonsense name with a logo, like Bing. DDG is a silly name with a cartoon duck.
- martinald 11y agoSlightly OT: Does anyone know how RankScanner works? My guess was this but it still seems to work. Their site seems to suggest they have loads of servers, but seems unlikely, I'd have thought Google would have captached them to hell by now. Very good product. I'm curious how it works though.
- woodruffw 11y agoI'm not acquainted with RankScanner, but my educated guess is that they use one of Google's (many) authenticated APIs[1] for search/content categorization/ranking analysis. [1]: https://developers.google.com/custom-search/ https://developers.google.com/custom-search/
- martinald 11y agoThanks. I was under the search impression that this was only for building your own internal search. do you know how much the pricing is?
- woodruffw 11y agoThere are some prices listed here: https://developers.google.com/custom-search/json-api/v1/overview#pricing https://developers.google.com/custom-search/json-api/v1/over... These seem to be for products in a slightly different domain than plain old search, though.
- Jake232 11y agoHaving built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option. Simply put, the API never returned the results in the exact same order as the actual search results did. The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of pages from google over the years, and with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily. This is especially true with the dropping cost of proxies. I'm obviously an exception considering I run several scraping SaaS and have generally specialized in it for years, but I can he hitting x.com with 30,000 different IP's within the hour.
- mozumder 11y agoWhy do API developers add a HTTP status code in their JSON response, when the HTTP status code is already in the header?
- wmichelin 11y agoSay you're making an ajax request to this API, you don't necessarily know the status, especially if you're in an "always" handler or something.
- taylorhughes 11y agoThis but with jsonp might make sense; with xhr you can always get the http response code
- woodruffw 11y agoProbably because HTTP status codes are perceived to be at a different layer of the response than the body. A lot of developers (myself included, for trivial things) treat HTTP bodies as strings with no associated metadata.
- jsmthrowaway 11y agoIt's debatable, but that's probably antithetical to how HTTP is supposed to be perceived when you start thinking about what a body is.
- woodruffw 11y agoI would tend to agree. Speaking beyond convenience, this isn't a practice that's easy to justify ;)
- X-Cubed 11y agoDepending on the technology you're using to access the API, the HTTP status code may not be exposed from the underlying stack. For example, Microsoft Silverlight only returned 200 or 404 status codes: https://msdn.microsoft.com/en-us/library/system.net.httpstatuscode(v=vs.95).aspx#Anchor_1 https://msdn.microsoft.com/en-us/library/system.net.httpstat...
- SwellJoe 11y agoWhat are the alternatives for a full web search API? Yahoo BOSS closed down a few weeks ago. Bing seems the last man standing. And, unfortunately, none of the ones I've tried are very good; one project I'm working on needs to prioritize newer results over older ones, and only Google had a sort by date option (which I think was also removed in the past).
- bake 11y agoGoogle Custom Search is one option, but it has a limit of 10,000 queries per day and is very expensive compared to Bing ($5 per 1,000 queries). I've migrated to Bing, myself -- no volume limits, generous free tier, and much cheaper than Google.
- SwellJoe 11y agoI'm not seeing how to make GCS perform a full web search. It won't allow creation of a search without at least one schema defined, and I need schemaless data (or rather, I need the whole web, including a variety of schemas). Am I missing something?
- jrochkind1 11y agoCan you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. Bing has a decent search api? Interesting, somehow I missed that. Want to provide a URL to a page for the service you're talking about? MS is really bad at documenting their services and prices. I eventually found this page, at this ridiculous URL: https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44 https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57... . I think that's what you're talking about? Note the link on that page to "Bing API FAQ" -- is broken. If this wasn't MS, the poor docs would make me think it was surely a terrible product.
- woodruffw 11y ago>Can you use Google Custom Search for a full index web search?Last I looked, I thought you could only use it configured with a certain list of domains to include in the search. You can, refer to this SO answer: http://stackoverflow.com/a/11206266/3354209 http://stackoverflow.com/a/11206266/3354209
- bake 11y agoDoes anyone know of an API that returns pages related to any given URL? I'm looking for an alternative to Google's "related:..." search, which was only reliably accessible through Google Web Search (Google Custom Search is very expensive and has a limit of 10K queries per day). For example: https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fblog%2F2013%2F08%2Fwriting-thread-safe-django-code%2F https://www.google.com/#q=related:http:%2F%2Fagiliq.com%2Fbl... It returns pages related to any given page, and works at the specific page level (not just at the domain level, like SimilarWeb's API). I've migrated to Bing for all other web search needs (way cheaper and without volume limits like Google Custom Search), but Bing does not offer a "related:..." equivalent as far as I can tell.
- reefoctopus 11y agoI ran into this problem several years ago while writing SEO software. There are a bunch of companies who provide google search results, but there's usually at least a couple minute delay between when the data is requested and when it arrives at your callback page. I had success with Authority Labs. You might want to check them out. They'll raise the query limit if you ask them.
- partiallypro 11y agoWhile you're looking for this feature in the meantime, ping the Bing team and ask for it as an added function of their API. Microsoft is a lot more open to feedback these days.
- bake 11y agoAny idea on how best to get in touch with them?
- partiallypro 11y agoI don't know a great way, that's not intrusive, but try bingdevcenter@outlook.com or the MSDN forum. I know that Microsoft employees frequent HackerNews, so I'm sure someone will see this as well. https://blogs.msdn.microsoft.com/bingdevcenter/ https://blogs.msdn.microsoft.com/bingdevcenter/
- nfriedly 11y agoI wrote a PageRank library a while back and just got a report yesterday that it also no longer works. (PageRank never actually had an official API, but it was exposed for the Google toolbar. They had announced that this would go away a while back but just dropped the axe in the last day or two.)
- woodruffw 11y agoI guess they're cleaning house.
- jayjay1010 11y agoThis is terrible news and with Pagerank also disabled recently, now 2 features of our domain health testing service has been crippled. Pagerank was easy to replace with moz rank but this data was irreplaceable due to the fact that it was specific and unique to google. Rip one of our main tools.
- chrismorgan 11y agoPageRank had been completely useless for a couple of years, and practically useless for several years before that.
- jayjay1010 11y agoThat depends on your reason of usage, as a guide to web ranks its been obsolete as you say for years, but as a cheap and quick measure of a domains potential (for passing link juice or to compare link juice between two domains) in comparison against another, Pagerank had some value.
- visarga 11y agoWe need a search engine that allows for deep search. It should be an open and cooperative project, and users could possibly run an instance of the spider/indexer as payment for executing searches, so it could be like a cross between bittorrent and Tor. A free search engine would enable API calls and also boost privacy and freedom from the likes of Google. We have built a lot of experience about search engines since 2000, we have access to scientific papers, cheap cloud servers and a huge interest in freeing search, so I think the open source community do it.
- deleted 11y ago[deleted]
- sourcd 11y agoPerhaps something built on top of http://commoncrawl.org/ http://commoncrawl.org/ ? I hope "OpenSearch" becomes a thing, like OpenAI.
- sylvinus 11y agoSomething like http://commonsearch.org http://commonsearch.org ?
- pgeorgi 11y agoThe second paragraph seems to describe http://yacy.net/ http://yacy.net/
- sdrinf 11y agoWriting an open, and distributed web crawler / indexer is a nice programming exercise. Writing an "objective" ranking function (for any values of "objective") in an open, and distributed manner is structurally not favoured by humanity's current incentive structure. As in: * a dev team have to agree on signals, and weights: "SERP quality" has dedicated teams of people assigned for specific verticals @ Google; replicating this in a distributed manner will be played politically * Assuming any significant usage, the second you submit ranking code to public github repo, the algo will be played by thousand SEO scammers to their advantage * Executing custom ranking function on other people's computer not only introduces security risks, but will have scammers setting up honeypots for collecting other people's ranking signals, and playing accordingly.
- genop 11y agoThe YouTube API used to be open too - no regstration or authentication. That ended years ago. I don't really need an "API" to do searches and transform the results into a more useful format than HTML. These API's make it easier and are addictive, for sure. It's foolish to rely on them however.
- amelius 11y agoI don't want to sound negative, but this is why I don't use external APIs. You never know when they will suddenly shut down.
- oneeyedpigeon 11y ago"Use" != "depend". What's your alternative in this case, run your own search behemoth?
- mseebach 11y agoNo, it's only been deprecated for six years. There was just no way to see it coming.
- masswerk 11y agoPhew! -- Just managed to reactivate some applications[1][2] by providing cached search results. The Google REST API was a great way to provide audiences with a known tasks in order to connect them to a historic setup or interface. RIP. [1] http://www.masswerk.at/googleBBS/ http://www.masswerk.at/googleBBS/ [2] http://www.masswerk.at/google60/ http://www.masswerk.at/google60/
- allard 11y agoI believe Google will replace it — the Google Search Appliance is going away too. Something will pick up those customers. An announcement at I/O, which is pretty soon?
- gizzlon 11y agoIt has been deprecated for years, disallowing new customers. No sign of any replacement
- miduil 11y agoWe had some trouble when the Youtube API was disabled for non-registered users. We were using it for some shared & local web-playlist, which is based on youtube-dl/mpv. Sadly we couldn't use DDG because their API won't let you filter for video results only (Source: Asked on IRC, got a reply from DDG staff + haven't found any documentation mention this either). We ended up using searx [0] which takes little more time for search results (about 1-3 seconds) but we gain more video-sources (such as vimeo or dailymotion). I'm aware that youtube-dl got a search functionality build in as well, but the requirement that the client needs to get results without being prepared by our embedded computer wouldn't fit. [0]: https://github.com/asciimoo/searx https://github.com/asciimoo/searx
- givan 11y agoWith more people placing value on privacy, the api was used by disconnect search along with tor and other privacy tools, google has just planted the seed that will become their biggest competitor.
- sourcd 11y agoThis needs our love & support : commonsearch.org Prev discussion (few weeks back) https://news.ycombinator.com/item?id=11281700 https://news.ycombinator.com/item?id=11281700 Credit : @sylvinus
- dajohnson89 11y agoWhat is the rationale for Google doing this? How much money was this service costing them?
- skrowl 11y agoI believe the problem was that if you did /google or !google on your bots for various chat platforms, you were able to find the top hit or two without ever looking at Google's Ads. Most of their revenue is ads.
- dajohnson89 11y agoSo the question becomes, how much potential money were they losing through the API? Does Google think that by removing the API, people are now going to go to another window, perform a google search, and intently study the advertising before reviewing the search results?
- mwfunk 11y agoYou just made a textbook strawman. It doesn't do anyone any good to theorize about what the dumbest possible reason might be for Google to do this. It is infinitely more useful to think about what rational, informed reasons they might have for doing it (regardless of whether you agree with those reason). People reflexively jumping to strawman arguments is probably #2 or #3 on the top ten list of Why We Can't Have Nice Things, in tech, politics, and just about anything else in the world where different groups of people have imperfect knowledge about each other's activities and motivations.
- fastest963 11y agoAfaik this has been broken for me for December 8, 2015 when I first received: {"responseData": null, "responseDetails": "This API is no longer available.", "responseStatus": 403}
- TazeTSchnitzel 11y ago`"responseStatus": 403`, but the actual HTTP response is 200 OK?!
- yolesaber 11y agoBecause the request went through! /s