10 ms·
Nilay Patel has been talking about "Google Zero" - the moment when Google effectively stops sending any traffic to other sites - for a few years now: https://ww
by simonw 6mo ago
Nilay Patel has been talking about "Google Zero" - the moment when Google effectively stops sending any traffic to other sites - for a few years now: https://www.theverge.com/24167865/google-zero-search-crash-housefresh-ai-overviews-traffic-data-audience https://www.theverge.com/24167865/google-zero-search-crash-h...
- voidUpdate 6mo agoHow does google get the information to create an LLM-generated response without sending traffic to any other sites?
- dust-jacket 6mo agoBy scraping them, same as always. The difference is where once they scraped, wrote a summary and invited users to go to the site, now they just provide the summary.
- ekidd 6mo agoWhich as some running a website raises a fascinating question. If Google is just going to crawl my sites and present information as an AI summary on their site, then what exactly do I gain by allowing Googlebot to crawl my sites?
- whazor 6mo agoWebsites tend to be updated and considered to be the source as well.
- Andrex 6mo agoFree speculation: I could see a future where Google populates a footer on results with the website logos of the sources. Presumably, the new web will require users to memorize websites/brands and go directly to those sites if they see a lot of their results are being provided by one source. Websites may go back to being simply labors of love.
- victor106 6mo ago>I could see a future where Google populates a footer on results with the website logos of the sources. ChatGPT/Claude does this today. I barely click or care for the source when they already have me the info I wanted. My speculation is all information worth anything is going to be behind some kind of wall.
- nozzlegear 6mo ago> ChatGPT/Claude does this today. I barely click or care for the source when they already have me the info I wanted. Maybe I'm just #builtdifferent, but I click these a lot. Especially if I'm trying to research or make a decision on something, I want the actual source and not the potentially-fudged summary.
- victorbjorklund 6mo agoI click those all the time if it is something that matters and I wanna verify that the AI got it correctly.
- microtherion 6mo agoAbsolutely! And I also click them to get an idea how good the source is.
- dbetteridge 6mo agoSeems like a great way to end up knowing no real information and with no ability to analyse literature or think for yourself? Not to mention the hallucinations
- runako 6mo ago> Websites may go back to being simply labors of love. The situation may be even worse. Back in the labor of love era, at least webmasters could get feedback from readers. In the LLM era, readers may not even know that the site exists. Without feedback/community, the overall quality of those sites will decrease over time.
- pflenker 6mo agoA couple of years back I worked with a company which maintained specific data which was the main traffic driver on that page. Google approached them and wanted to pay for the rights to get the data and display it on top of the search results, a feature which was fairly new back then. This was an interesting dilemma because it was very clear that the money was way less than the loss in ad revenue due to traffic drop, but it was also clear that if we wouldn’t take the deal, a more desperate competitor would, which would result in the same traffic loss but without the extra google money. So the company took the deal. History repeats itself here, with the difference that instead of paying for the data, the ai crawlers simply take it for free.
- iamtedd 6mo agoReal-world Prisoner's Dilemma.
- jonshariat 6mo agoIt always comes back to game theory haha
- khana 6mo ago[dead]
- drcongo 6mo ago"Nice data you got there, it'd be a shame if something were to happen to it"
- vb-8448 6mo agoThe fact is that internet is already "tech giants own realm": the power is way beyond public imagination and affects all of us in real life on daily bases, but there are still people thinking they are not the "evil one" here.
- thelastgallon 6mo agoThis reminds me of Walmarts squeezing strategy with all the manufacturers. Business with us at the price we say or out of business.
- thedelanyo 6mo ago> then what exactly do I gain by allowing Googlebot to crawl my sites? Mention
- lacewing 6mo agoIt's worse than that. They train their models preferentially on what they consider to be high-quality data. But if you look at the usual "references" on search queries, they're often just a post-hoc BS justification that links to spam blogs or Tiktok videos.
- swarnie 6mo agoAllow? Deep down, do you think you have a choice? Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of books being gobbled up by TheZuck but ultimately do you really have a choice? Rules and laws don't exists for you.
- alt227 6mo agoYes, Google advertises its crawler IP ranges and it is quite easy to keep track of this and block them. But only if you control the infrastructure that your site runs on of course.
- rolph 6mo agostego your site, google sees the red herring version, intended users see the payload. this has been done before, quite often, but toward ends morally askew.
- account42 6mo agoYou think they'll stick to those IPs any more than robots.txt when it comes to that?
- alt227 6mo agoYes I do, I also think they will continue identifying themselves in the user-agent header. https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers https://developers.google.com/crawling/docs/crawlers-fetcher...
- coldpie 6mo ago> allowing Googlebot to crawl my sites As far as I know, you don't have a choice. They have no obligation to respect your wishes, and LLMs are legally allowed to scrape & republish your content.
- margalabargala 6mo ago> They have no obligation to respect your wishes I have no obligation to not send all scraper-looking traffic to a black hole full of zip bombs.
- charcircuit 6mo agoYou do have an obligation because what you are describing is illegal, at least in the US under the CFAA.
- margalabargala 6mo agoOkay, nix the zip bombs. What's my obligation to treat bot-shaped traffic as something I should reply to?
- charcircuit 6mo agoYour obligation is not to cause harm to the requester.
- chadgpt3 6mo agoSending them a zip bomb didn't cause harm. It was their choice to unzip it. Is jwz liable if a child sees his testicle eggcup macro when visiting via HN?
- charcircuit 6mo agoWhy send a zip bomb if your goal is not to cause some amount of a denial of service attack to the crawler. Showing porn to a minor is not legal either.
- tedd4u 6mo agoVastly less but still more traffic than if you didn’t participate. I’m sure they will calibrate it just so.
- wvenable 6mo agoIt's a catch-22. Without google crawling your site, you don't get any new traffic. But with google crawling your site, you also might not get any traffic. AI summarization has already causes issues for sites like rtings where people are no longer visiting the site but still making use of the data presented there. Leading to rtings not getting enough traffic to continue to post their data. It is an existential crisis for websites and when they go away it'll be an existential crisis for AI.
- monooso 6mo agoThat's some catch, that catch-22.
- BubbleRings 6mo agoI see everything twice!
- Boxxed 6mo agoIt's the best there is.
- elevation 6mo ago> Without google crawling your site, you don't get any new traffic What about the stories of marketing managers who learned months after the fact that their credit card had expired and their google ad spend had ceased with no affect on traffic? Google isn't always an effective promotional vehicle.
- deaton 6mo agoWhat you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt
- loloquwowndueo 6mo agoThere are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.
- pokot0 6mo agoInternet is more and more becoming a commercialization platform. If you are selling something on your website, you still want Google (or ChatGPT for that matters) to expose customers to your product. The gate is the actual delivery of the product is behind a purchase/signup. Google and others want to control the entire customer journey, to the point the your website is simply a way to pass metadata to them. They are actually achieving this! this kills the entire internet vibe of the 90s, early 2k
- nine_k 6mo ago> is more and more becoming a commercialization platform FTFY: "couple of decades since has become". The vibes of passion-driven 1990s started to be overwhelmed by the din of money right when the Internet has become a major commerce venue, some time in early 2000s.
- 1718627440 6mo agoThe content has long been commercialized, it's the platform itself, that it is now undergoing that change.
- soco 6mo agoMaybe it's time to think about alternative ways to market your products, if search engine ranking and SEO got broken. I have no idea how, I don't need or do that, but it seems we're past breaking point.
- franze 6mo agowell its already happening and people are fighting over traffic crumbs already, they call it GEO
- UltraSane 6mo agoCan you actually prevent Google from crawling your site?
- prinny_ 6mo agoYou're allowed to exist on the web. The alternative is you are pushed out, your site is not indexed and google / chrome labels it as a security risk when people are trying to reach it directly. The mandate is clear: give up the data or give up the spot.
- jefftk 6mo agoI write things on the internet because I want to share ideas. If someone reads my post and tells a friend, that's great. If an AI crawls my posts and passes along the ideas that's great too. (It doesn't work for ad-funded writing, but while I have substantial sympathy there this has historically been an unpopular argument on HN)
- kennywinker 6mo agoSetting aside ad-driven revenue - the ideas, when spat out by an llm, are disconnected from the author. If people like your ideas, they aren’t becoming fans/followers/long-term-readers. That means good luck leveraging some interesting writing into a book, a speaking tour, a podcast, or even any kind of consistent readership. The llm slurps up your content and monetizes it while you get nothing.
- jefftk 6mo agoI'm not interested in a book, speaking tour, or podcast. I've never had consistent readership because I write about too many unrelated things. I blog because I have ideas I want to share; I don't feel at all ripped off.
- arcatech 6mo agoAnd do you think that’s how everyone should feel? If not, how is it relevant to people not liking what Google is doing?
- jefftk 5mo agoPeople are certainly welcome to feel a lot of different ways, not trying to be prescriptive here. My parent asked: "what exactly do I gain by allowing Googlebot to crawl my sites?" and I was describing what I get out of it, in the hope that others might feel similarly.
- kennywinker 6mo agoFair enough, sounds like you won’t be impacted. But the vast majority of people i read online are able to write the content i enjoy because there are paths to earn a living off it. I expect the future of llm search will leave only hobbyists and slop producers standing.
- victorbjorklund 6mo agoMaybe you want your ideas to spread? If your sites purpose is getting ad impressions then yea no point. But if your purpose is to spread ideas then it is still useful.
- nicbou 6mo agoI have spent nine years putting out free information. Surely you realise that I have to pay rent and buy food while I do it? My income isn’t ads, just getting a cut of the sale on the complex products I help you buy. Even that sort of curation takes time and effort. Even for all the things I do for free without any revenue whatsoever - most of it, really - I do want to feel some recognition. I don’t want the interaction to be mediated by an advertising company.
- account42 6mo agoGoogle is not going to be spreading ideas that they don't approve of.
- nine_k 6mo agoIf your site is all about disseminating information (like Wikipedia), then Google would provide a free mirror of sorts. If your site is about your product, Google won't be able to serve the sign-up page from AI; the traffic would come your way. Same for a site that sell something: the traffic you're interested in would arrive at your checkout page. Paid-content sites and ad-supported sites are screwed though, on top of their being screwed by archive.is and ad blockers.
- afavour 6mo agoThe really confusing part about the ad-supported sites is that most of them are supported by Google's ad products. So Google is eating their own lunch here.
- rchaud 6mo agoSearch Engine Result Page (SERP) ads shown on Google itself are far more valuable than display ads that get shown on random websites. Google has been slashing payouts to those sites for over a decade. More recentlt, they've been slashing search impressions to those sites as well. With search engine ads + Youtube ads + Play Store ads, they can probably cut out the third - party site ads business altogether and not miss a beat.
- watwut 6mo ago> Google would provide a free mirror of sorts. That is not what "free mirror" is. Like, that is not the same thing at all.
- esseph 6mo ago> what exactly do I gain by allowing Googlebot to crawl my sites? Site traffic
- nicbou 6mo agoThat’s the thing, they have altered the deal. You still feed the machine but you get no traffic. Keep writing that helpful stuff though!
- rdedev 6mo agoSites pay good money to appear on top search results. Looks like the future is going to be sponsored AI sources. It's going to be even more difficult to figure out if google is presenting you with actual information instead of just an ad
- try-working 6mo agoThat's Google making way for its disruptor. We'll see who that is. Imagine a search engine that just presents search results. Groundbreaking.
- daemin 6mo agoMore likely you're going to get a search engine which returns results as short 5 second AI generated video clips with an infinite scroll. (Torment Nexus rules apply here)
- duskdozer 6mo agoAs it turns out I found out some people use tiktok as a search engine. How exactly, I am not sure.
- jazzypants 6mo agoYou search for a term and then you watch videos until you're satisfied. I don't think it's a very good way to learn about a topic, but millions of people seem to disagree with me. It makes more sense when you remember that 21% of the American population is functionally illiterate. https://en.wikipedia.org/wiki/Literacy_in_the_United_States https://en.wikipedia.org/wiki/Literacy_in_the_United_States
- hansmayer 6mo agoIt was exactly my opinion after thinking about this for a while. This is essentially Google making their search engine into yet another website. Sure, there is certain inertia - people used to using Google - but it will fade out with new generations. They've shot themselves in the foot, they just don't see it yet, and it will take some time for it to become obvious.
- chadgpt3 6mo agoThanks for the optimistic view. We could use more of that. I can only hope it comes about...
- gerdesj 6mo ago(You misspelled someone as some) Google has always crawled your site and been an arse! Now you get to decide whether they are hallucinating! You can drop pointers on Masto and other socials to your sites - that has not changed. Do we need something else? ie you drop a link to somewhere else.
- lifis 6mo agoThe expected purpose of websites is to spread information, so whether users get it by making a request to your website or to Google is irrelevant. In fact, if they get it from Google it's better because it reduces website load. If instead the purpose of your website is to manipulate users for financial gain (for instance by showing media attempting to manipulate their purchasing decisions, after receiving a bribe from a vendor), and the information is just a way to lure users, then maybe this malicious business model will finally be no longer possible.
- AlexandrB 6mo agoOn the one hand, this is an interesting observation. The internet as it exists today is filled with product placement slop and real information is a rare commodity. The loss of these kinds of sites is a blessing. On the other hand, Google played a big role in creating this problem in the first place. The search results have trended downwards towards this kind of SEO slop for the last 10 years and Google has been unable (or unwilling) to fix it. Plus, the AI results Google shows are not free from commercial influence and will probably only get worse in this regard. Except now this money will flow to Google instead of $random_internet_spammer. I don't know if that's any better. The idea that Google won't eventually "manipulate users for financial gain" with Gemini is comically naive. That's how they're going to make money from this thing.
- ymolodtsov 6mo agoDepends on what you're trying to do. Sell ads over your content? Probably not great. Sell goods? Still good for you. Become influential and spread ideas? Good for you. It's the news media that will suffer the most.
- saddist0 6mo agoHonestly, a majority of news media are just AI generated content (echo chambers). I personally feel Dead Internet Theory is true and has arrived already.
- ymolodtsov 6mo agoIt's like saying that most YouTube videos are bad. Sure, but there's a power law and most people consume the top content.
- conradfr 6mo agoWho can lobby the EU for the right to abstain a website from being used for training data?
- 1718627440 6mo agoCopyright already exists, the issue is that these companies are doing it legally anyway. For me it is the same issue as with privacy: I'm deeply uncomfortable with the current situation, but there is no political fight for me to fight, because the law is already how I want it to be, it's the public perception, that needs to change, but that is hard to influence, without being rich.
- duskdozer 6mo agoIt should have been a clear extension of the intent of existing copyright/licensing that training would be disallowed without consent, but "move fast and break things"/"possession is nine-tenths of the law" win out
- gblargg 6mo ago> then what exactly do I gain by allowing Googlebot to crawl my sites? Making the information available that you put up your site for?
- larodi 6mo agoExactly nothing, nada, zero. And given it does not only crawl, it trains on it, so that never ever should people come back to search for it again.
- vdelpuerto 6mo agohonest question here, why there are lot of comments about SEO is dead?
- chadgpt3 6mo agoSearch engines are dead. The industry has moved to GEO.
- vdelpuerto 6mo agoThanks. I'll take it a look
- le-mark 6mo ago> then what exactly do I gain by allowing Googlebot to crawl my sites? The counter argument is that sites are becoming more AI slop or may intentionally provide poison they don’t want to train on. There may be a cut off date after which training must be carefully curated; and the main body of data has already been collected. Sites may still get traffic from agents searching for current information. Maybe even the resurgence of RSS? One can dream.
- chadgpt3 6mo agoThe opportunity to feed it false information?
- gsky 6mo agoI asked the exact question back in 2011 when they were showing images directly leaving me pay for bandwidth out of my pocket
- Minor49er 6mo agoIt's the Google AMP controversy all over again
- watwut 6mo agoNothing, but idea is that you wont be able to prevent it from happening. In that case, the consequence will be that people will stop having webs. It is already happening with personal and niche sites.
- SoftTalker 6mo agoThe web as we know it is over. Websites will die on the vine if LLMs intermediate all the content. The "website" of the future will be an API optimized for LLM crawlers, serving plain-text content that no end-user will ever view directly. The SEO game will change to LLMAO.
- LetsGetTechnicl 6mo agoThat sounds like absolute hell
- ben_w 6mo ago> Websites will die on the vine if LLMs intermediate all the content. The current zeitgeist of them will, but I think not all. My first website (GeoCities) was either before Google existed or very close to it. Connected to people via WebRings and directory listings. More recently, RSS feeds.
- SoftTalker 6mo agoYeah there will likely continue be a small underground of old-style websites I guess. But you'll have to be in the loop on how to find them, and very few people will pay to advertise on them.
- mrec 6mo ago> very few people will pay to advertize on them That sounds like an unalloyed plus. The perverse incentives caused by advertising have been the biggest driver of the web's decline, IMO.
- BrunoBernardino 6mo agoAlternatively, we can collectively "fight back" by not using Google and teaching others around us to do so as well. There are plenty of decent [1] and great (better) alternatives, where you're not the product [2]! [1]: https://alternativeto.net/software/google-search/?license=free https://alternativeto.net/software/google-search/?license=fr... [2]: https://alternativeto.net/software/google-search/?license=commercial https://alternativeto.net/software/google-search/?license=co...
- CSMastermind 6mo agoThis was the promise of Bing that never materialized.
- winterschon 6mo agoHe also spins a lot of trash talk about an industry he's never personally worked in as any kind of engineer at all. He's a "Journalist Covering Tech" without a degree in journalism, so he's not even a "Tech Journalist"; might as well be the blogger character from Silicon Valley. > Bachelor of Arts degree in political science from the University of Chicago. His hot takes are best ignored, is just convenient click bait for their entire negativity angle.
- jreed91 6mo agoBrendan Carr is that you?
- nicbou 6mo agoIt has reduced traffic to my website by around 65%. I live from that website. My income is a function of the traffic it gets. I spent 9 years of my life putting hard-earned information on the internet, and now big tech uses it to enrich themselves while putting me out of work. Even my backup plan - software development - is being devalued to hell. It's so damn depressing. We'll get the internet that we deserve.
- salomonk_mur 6mo agoTo be fair, the traffic you had was mostly routed by them as well. Google giveth, Google taketh away.
- ryanisnan 6mo agoCallous take. You make it seem like only Google was giving here. If Google was routing users to OP's site, surely OP had something beneficial to give.
- atoav 6mo agoNo. OP made the content Google copied with AI to their landing page, which reduced the traffic. But I forgot that they convinced you to accept AI as a magic "make copyright disappear"-technology.
- CalRobert 6mo agoHad they been correctly punished for anticompetitive practices there would be other search engines with meaningful market share.
- jmusall 6mo agoIn my humble opinion it was balanced when Google gave traffic to sites and these sites gave good results for Google to show
- intended 6mo agoGoogle giveth jack shit here. Google users wanted to find that info. Google served its users.
- deleted 6mo ago[deleted]
- andsoitis 6mo agoIsn’t Google’s ads flywheel depend on them sending traffic to websites?
- jcattle 6mo agoYes, but this just shows how google is not repeating mistakes of the Kodaks and Blockbusters of the world. They keep innovating even if it means cannabilizing their main revenue stream. Which increases the chances that you will not still be stuck producing film emulsions while everyone else is slowly making bank doing digital.
- snowwrestler 6mo agoOver the past year to 1.5 years, in the sites I run, I have seen a drop in traffic from Google, which leveled off, and is now slightly rising. I think if you look through this thread you’ll see a lot of skepticism of the AI results, and I think that is a fairly broadly held opinion. The obvious way to check the AI answer is to click through to some sources. I think for Google to stop sending me traffic, it would have to be essentially perfect at AI answers. It will never get there, especially as so many searches are opinion-based like “what is the best mobile phone right now.”
- dyauspitr 6mo agoThey will not do that because their cash cow ad revenue depends on it
- ex-aws-dude 6mo agoTake it one step further, the ultimate endgame is everyone consuming things through their own LLM assistant In the future I don't even use Google but my bot does
- spencerflem 6mo agoI think the idea is the google is the bot
- ex-aws-dude 6mo agoYeah but google will inject ads, my future LLM bot will filter them out Why would I want to be restricted to google’s “view” of the info
- popcorncowboy 6mo agoAssumes you'll be able to parse out, or even discern the ads. "May contain product placement". The content is the ad.
- tardedmeme 6mo ago(this bot exists and is called Kagi)
- illiac786 6mo agoThat article is not even two years old… “for a few years” make it sound like he saw it coming long before it started. He’s just describing a trend and its ineluctable destination. But I do agree it will be or already is a paradigm shift. And a painful one.
- simonw 6mo ago> That article is not even two years old" From that article: > There’s a theory I’ve had for a long time that I’ve been calling “Google Zero” [...] Regular Decoder listeners have heard me talk a lot about Google Zero in the last year or two. But the point here is not that he predicted it first, it's that he coined a term for it and has been extensively covering it.