10 ms·
Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searc
by 01jonny01 9mo ago
Google quietly announced that Programmable Search (ex-Custom Search) won’t allow new engines to “search the entire web” anymore. New engines are capped at searching up to 50 domains, and existing full-web engines have until Jan 1, 2027 to transition.
If you actually need whole-web search, Google now points you to an “interest form” for enterprise solutions (Vertex AI Search etc.), with no public pricing and no guarantee they’ll even reply.
This seems like it effectively ends the era of indie / niche search engines being able to build on Google’s index. Anything that looks like general web search is getting pushed behind enterprise gates.
I haven’t seen much discussion about this yet, but for anyone who built a small search product on Programmable Search, this feels like a pretty big shift.
Curious if others here are affected or already planning alternatives.
UPDATE: I logged into Programmable Search and the message is even more explicit: Full web search via the "Search the entire web" feature will be discontinued within the next year. Please update your search engine to specify specific sites to search. With this link: https://support.google.com/programmable-search/answer/12397162 https://support.google.com/programmable-search/answer/123971...
- throwaway_20357 9mo agoWhat are some of the niche search engines build on Google's index affected by this?
- doublerabbit 9mo agoKagi
- deleted 9mo ago[deleted]
- marginalia_nu 9mo agoThey published this the other day: https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search Which saw some discussion on HN.
- embedding-shape 9mo ago> some discussion ~450 score, ~247 comments and still on /best ("Most-upvoted stories of the last 48 hours"): https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678 - "Waiting for dawn in search: Search index, Google rulings and impact on Kagi"
- pell 9mo agoI think Kagi buys search engine results from SERP vendors who typically scrape Google’s results and offer an API experience on top of it.
- echelon 9mo agoNo wonder Kagi is angry. Google is a monopoly across several broad categories. They're also a taxation enterprise. Google Search took over as the URL bar for 91% of all web users across all devices. Since this intercepts trademarks and brand names, Google gets to tax all businesses unfairly. Tell your legislators in the US and the EU that Google shouldn't be able to sell ads against registered trademarks (+/- some edit distance). They re-engineered the web to be a taxation system for all businesses across all categories. Searching for Claude -> Ads in first place Searching for ChatGPT -> Ads in first place Searching for iPhone -> Ads in first place This is inexcusable. Only searches for "ChatGPT versus", "iPhone reviews", or "Nintendo game comparison" should allow ads. And one could argue that the "URL Bar" shouldn't auto suggest these either when a trademark is in the URL bar. If Google won't play fair, we have to kill 50% of their search revenue for being egregiously evil. If you own a trademark, Google shouldn't be able to sell ads against you. -- Google's really bad. Ideally we'd get an antitrust breakup. They're worse than Ma Bell. I wouldn't even split Google into multiple companies by division - I'd force them to be multiple copies of the same exact entity that then have to compete with each other: Bell Systems -> {BellSouth, Bell Atlantic, Southwestern Bell, ...} Google -> {GoogleA, GoogleB, GoogleC, ...} They'd each have cloud, search, browser, and YouTube. But new brand names for new parent companies. That would create all-out war and lead to incredible consumer wins.
- marginalia_nu 9mo agoCould probably argued that search access is an essential facility[1], though it doesn't appear antitrust law has anywhere near the same sort of enforcement it did in the past. [1] https://en.wikipedia.org/wiki/Essential_facilities_doctrine https://en.wikipedia.org/wiki/Essential_facilities_doctrine
- onetokeoverthe 9mo ago[dead]
- throwaway290 9mo agowhat stops Kagi from indexing internet and makes them pay some guys to scrape search results from Google? one guy at Marginalia can do it and entire dev team at a PAID search engine can't?
- nemosaltat 9mo ago> Kagi This seems to be true, but more indirectly. From Kagi’s blog [0] which is a follow up to a Kagi blog post from last year [1]. [0]> Google: Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model.[^1] [0]> The current interim approach (current as of Jan 21, 2026) [0]> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations. I’m an avid Kagi user, and it seems like Kagi and some other notable interested parties have _already_ been unable to do get what they want/need with Google’s index. [0]> The fact that we - and companies like Stanford, Nvidia, Adobe, and the United Nations - have had to rely on third-party vendors is a symptom of the closed ecosystem, not a preference. Hopefully someone here can clarify for me, or enumerate some of these “third-party vendors” who seem like they will/might/could be directly affected by this. [0] antibabelic > relevant https://blog.kagi.com/waiting-dawn-search https://blog.kagi.com/waiting-dawn-search [1] https://blog.kagi.com/dawn-new-era-search https://blog.kagi.com/dawn-new-era-search > [^1]: A note on Google’s existing APIs: Google offers PSE, designed for adding search boxes to websites. It can return web results, but with reduced scope and terms tailored for that narrow use case. More recently, Google offers Grounding with Google Search through Vertex AI, intended for grounding LLM responses. Neither is general-purpose index access. Programmable Search Engine is not designed for building competitive search. Grounding with Google Search is priced at $35 per 1,000 requests - economically unviable for search at scale, and structured as an AI add-on rather than standalone index syndication. These are not the FRAND terms the market needs
- tpetry 9mo agoI believe they try to indirectly say they are using SerpApi or a similar product that scrapes Google search results to use them. And other big ones use it too so it must be ok... That must be the reason why they limit the searches you can do in the starter plan. Every SerpApi call costs money.
- monooso 9mo agoKagi does not use Google's search index. From their post which made the front page of HN yesterday [1]: > Google does not offer a public search API. The only available path is an ad-syndication bundle with no changes to result presentation - the model Startpage uses. Ad syndication is a non-starter for Kagi’s ad-free subscription model. [1]: https://news.ycombinator.com/item?id=46708678 https://news.ycombinator.com/item?id=46708678
- jsnell 9mo agoThey then go on to say that they pay a 3rd party company to scrape Google results (and serve those scraped results to their users). So their search engine is indeed based on unauthorized and uncompensated use of Google's index. But since they're not using/paying for a supported API but just taking what they want, they indeed are unlikely to be impacted by this API turndown.
- DangitBobby 9mo agoCongrats on saying that in the most one-sided way possible. Google makes it literally impossible for them to pay for access to search results to make the product they want (customizable subscription search with no ads), and Google also is the de-facto globally sanctioned crawler because they are the only search engine anyone gives a shit about, and also sites need to be indexed by them to survive. In short, Google owns the river and sells the boats, and the public built a wall around it. Google is in a monopoly position in search.
- nova22033 9mo ago>In short, Google owns the river and sells the boats, and the public built a wall around it. That would be a monopoly if there was only 1 river in the whole world.
- DangitBobby 9mo agoYeah I mean think whatever you need to for the metaphor to work.
- deleted 9mo ago[deleted]
- saltysalt 9mo agoI built my own web search index on bare metal, index now up to 34m docs: https://greppr.org/ https://greppr.org/ People rely too much on other people's infra and services, which can be decommissioned anytime. The Google Graveyard is real.
- orf 9mo agoNumber of docs isn’t the limiting factor. I just searched for “stackoverflow” and the first result was this: https://www.perl.com/tags/stackoverflow/ https://www.perl.com/tags/stackoverflow/ The actual Stackoverflow site was ranked way down, below some weird twitter accounts.
- saltysalt 9mo agoI don't weight home pages in any way yet to bump them up, it's just raw search on keyword relevance.
- orf 9mo agoSure, but the point is results are not relevant at all? It’s cool though, and really fast
- saltysalt 9mo agoI'll work on that adjustment, it's fair feedback thanks!
- direwolf20 9mo agoUnfortunately this is the bulk of search engine work. Recursive scraping is easy in comparison, even with CAPTCHA bypassing. You either limit the index to only highly relevant sites (as Marginalia does) or you must work very hard to separate the spam from the ham. And spam in one search may be ham in another.
- 9mo ago
- deleted 9mo ago[deleted]
- salawat 9mo agoIt's been clear for the last decade that we have to wean ourselves off of centralized search indexes if only to innoculate the Net against censorship/politically motivated black holing. I can only weep at this point, as the heroes that were the Silent and Greatest generations (in the U.S.), who fought hard to pass on as much institutional knowledge as possible through hardcore organization and distribution via public and University library, have had that legacy shit on by these ad obsessed cretins. The entirety of human published understanding; and we make it nigh impossible for all but the most determined to actually avail themselves of it.
- raincole 9mo ago> “search the entire web” TIL they allowed that before. It sounds a bit crazy. Like Google is inviting people to repackage google search itself and sell it / serve with their own ads.
- MrGilbert 9mo agoYou know, back in the days, the web used to be more open. Also - just because you CAN do something, doesn't mean you HAVE to.
- shevy-java 9mo agoIt basically means that Google is now transitioning into a private web. Others have to replace Google. We need access to public information. States can not allow corporations to hold us here hostage.
- whs 9mo agoI tried it and contributed to searx. It didn't give the same result as Google, and it also have 10k request rate limit (per month I believe). More than that you'll have to "contact us"
- zitterbewegung 9mo agoI know that duckduckgo uses Microsoft Bing Custom search and honestly it is a much more robust system since you don't have to worry about Google axing it. https://www.customsearch.ai https://www.customsearch.ai
- embedding-shape 9mo agoInstead you worry about Microsoft axing it? Sure, it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month, but hardly either are long-term solutions.
- aylmao 9mo ago> it might take 3 years instead of 6 months, and the shutdown period would be 1 year instead of 1 month This matters much more than people (and evidently those within Google) realize
- embedding-shape 9mo agoIt matters more the shorter your future planning is. Neither works if you're looking forward 3-4 years, for example.
- deleted 9mo ago[deleted]
- aylmao 9mo agoWhy wouldn’t it work if you plan ahead 3-4 years? Planning ahead doesn’t mean you never suffer from unexpected events, including depreciations. And long timelines help you as much as anyone else.
- sumtechguy 9mo agoHaving been downsizing my horde of computer junk I have several large boxes of full MSDN disc sets. There is a graveyard of MS stuff that is no longer supported. The only thing with MS is they seem to give you a better off ramp usually than 'oh well sucks to be you'.
- 1vuio0pswjnm7 9mo agoIf its Motion for a Partial Stay is denied, or if it loses on appeal, then under this Final Judgement Google will be forced to offer syndicated "full web" search to Qualified Competitors https://dn710204.ca.archive.org/0/items/gov.uscourts.dcd.223205/gov.uscourts.dcd.223205.1462.0.pdf https://dn710204.ca.archive.org/0/items/gov.uscourts.dcd.223...
- vagab0nd 9mo agoDamn, I just wrote a note "search is free" in my aggressively-automate-everything-using-llms personal project plan.md. I guess not anymore.
- 01jonny01 8mo agoImportant Email Update from Google: Dear Programmable Search Engine user, Thank you for contacting us via the Web Search Products Interest Form. We have received your feedback and are actively reviewing the specific use cases you shared. We are writing to share important details regarding the transition plan and the available solutions for your search needs. 1. For Unrestricted Web Search: Future Web Search Service For partners requiring unrestricted "Search the entire web" capabilities, we are developing a new enterprise-grade Web Search Service. As you evaluate your future needs, please be aware of the commercial terms planned for this new service: Pricing: USD $15 CPM (Cost Per Mille / 1,000 requests). Minimum Commitment: A minimum monthly fee of USD $30,000 will apply. We’ll release more information on this service later in 2026. Existing 'Search the entire web' engines remain functional until January 1, 2027. 2. For AI & Advanced Search: Google Vertex AI We strongly encourage you to explore Google Vertex AI as another option for partners who need enterprise search and AI capabilities across 50 or fewer domains. Vertex AI offers powerful capabilities for: Grounded Generation: Connecting your AI agents to your own data and/or to Google Search to provide accurate, up-to-date responses. Custom Data Search: Building enterprise-grade search engines over your own data and specific websites. This solution is available now and is designed to scale with your specific application needs. Clarification on Current Service Status: While you evaluate which path fits your business needs, please remember the timeline for your current implementations: Existing Projects: If you have an existing Programmable Search Engine configured to "Search the entire web," your service will continue to function until January 1, 2027. You have the full year to plan your migration. New Projects: As of January 20, 2026, new engines created in the Programmable Search Engine admin console are restricted to "Site Search Only" (specific domains only). Later in 2026 we’ll provide you with more updates regarding the new Web Search Service and the means to express your desire to use it to power your web search experiences. Sincerely, Programmable Search Engine Team