14 ms·

google.com/goto: Google's anti-scraping update

678 points by 1e1a 23d ago

JohnFen 23d ago
If I hadn't already ditched Google long ago, this would be more than enough to chase me away.
- scottyah 23d ago
  Why? I don't really see why this is bad, other than if your company blocks url shorteners/redirects. What are you losing?
  - nextaccountic 23d ago
    I use a Firefox extension to rewrite those Google redirects into plain links, to remove Google tracking of which links I click. The extension is broken now
  - JohnFen 23d ago
    Because I wouldn't want Google to know what links I click on. When I used to use Google, and they added such redirects that included the destination URL as a parameter, I'd edit the link to make it just that URL before resolving it. This new scheme would make that impossible.
    - what 23d ago
      They already knew which links you clicked on though…
      - googapologist 23d ago
        [flagged]
      - deleted 23d ago
        [deleted]
      - JohnFen 23d ago
        How? I never allowed Javascript or anything.
      - smart_asslop 22d ago
        Even with JS enabled, they did not. Linked below is an explanation which includes confirmation of that fact by a member of the uBlock Origin team. https://news.ycombinator.com/item?id=49680338 https://news.ycombinator.com/item?id=49680338
    - brainwad 23d ago
      They own the JS on the page, they don't need the redirects to know where you clicked...
      - googapologist 23d ago
        [flagged]
      - deleted 23d ago
        [deleted]
      - smart_asslop 22d ago
        Hello! It appears that you are disseminating misinformation. A uBlock Origin developer now confirms, regarding Google results click tracking: It existed in a way or another, but it could be blocked. Now it can't. Which corroborates the mysteriously flagged sibling comment. https://old.reddit.com/r/uBlockOrigin/comments/1we7491/how_to_bypass_googles_goto_url_rewrite_for_search/p9fnjzq/ https://old.reddit.com/r/uBlockOrigin/comments/1we7491/how_t... Next time, please consider listening to people who know how the web works and avoid overconfidently parroting what Big Tech gatekeepers want you to think. Thanks!
        brainwad 22d ago
        It's not misinformation. By default Google has been able to click track with JS. If you disable JS, they don't serve you search (https://serpapi.com/blog/google-now-requires-javascript/ https://serpapi.com/blog/google-now-requires-javascript/). Yes, if you are tricky, you can block just the part of the JS that does click tracking, probably. But most users can't do that. They therefore were already getting 99% complete click tracking before this change. This change is not about click tracking, it's about blocking scrapers. And btw about the result filtering the uBlock dev wants to do: of course this can still be done, if you preload all the goto links on the page and then inline the real URLs returned by the preload requests. Perhaps not easy to do as an extension but at a browser level (Brave?) totally possible. Though Google might then think your users are scrapers...
        22d ago
  - Fabricio20 23d ago
    Blocking URL shorteners and google-ad links yes! Personally for me it's also the fact that this is effectively an unresolvable URL shortener, store that link somewhere and it will most likely be dead. Can't copy link anymore and paste it on a notepad or chat app to check it out later as there isn't a guarantee it will load at all (ie: the problem with url shorteners).
  - exlr 23d ago
    [flagged]
- tarkin2 23d ago
  They are forcing my further and further towards kagi. I do hope the fawning on here is at least partly justified.
  - JohnFen 23d ago
    Kagi is what I settled on a couple of years ago. I do think sometimes the fawning is over the top, but it is a solid search engine and tended to give me a bit better results than Google out of the box. The real win, though, is that you can give various sites a weight, so the search results will prefer or avoid sites according to your desires. Once I had that going, my search results tended to be much better than Google.
    - tarkin2 23d ago
      Yeah, that's the thing for me: filtering out the SEO crap that Google happily serves up. Google's actual search results are a waste of time visiting, both for the mindless CEO content and the ad-laden, analytics-happy, javascript-heavy websites. So I tend to use the AI overview. But plugging myself into the all-seeing corporate oracle, that grew on all the web's content, and now seeks to supplant it seems unseemly. It's just sad that kagi will likely only be a fringe thing, and google will continue to promote these foul, foul, mindless websites and then supplant them with its AI.
    - Zambyte 23d ago
      Yup, same here. Been using Kagi for years. It's boring, it just works. Hoping they can stay that way.
      - lukeify 23d ago
        My ongoing concern with Kagi is they always seem to be focused on sidequests like their Orion browser and their LLM-powered Translate tool. Maybe that's interesting for some people, but I can't help but feel like I just want a damn search engine than works.
        23d ago
1e1a 23d ago
Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url=<opaque base64 string>. The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL. Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.
- darknavi 23d ago
  > Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating. Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced. I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.
  - mentalpiracy 23d ago
    Wow - this exact same bug has been happening to me too. I gave up on troubleshooting it after the first few attempts came up with nothing, assumed it was just unique to me.
    - Maxion 23d ago
      I searched for this a few months ago and found some mentions of this bug, but yes it affects me too. Can stall up to 20 seconds+ sometimes. Chrome is fine.
    - darknavi 23d ago
      I had no idea others hit this, I just assumed it was some wonky setup I have locally. I hope we both figure it out one day haha (or Firefox does)
  - Starlevel004 23d ago
    > On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, Have you tried disabling HTTP/2?
deleted 23d ago
[deleted]
liquid_thyme 23d ago
I've been using the 'ClearURLs' FF addon for quite a while now, and it gives you the URLs back in the results. https://docs.clearurls.xyz/ https://docs.clearurls.xyz/
- joeblubaugh 23d ago
  That wouldn’t work at all with this url format - the url can only be resolved by requesting the goto link. I recommend reading more than the headline
  - deleted 23d ago
    [deleted]
  - Dylan16807 23d ago
    > That wouldn’t work at all with this url format - the url can only be resolved by requesting the goto link. What is "that" you say wouldn't work? The extension can work fine by requesting every goto link upfront. As far as your google searches go, this is just as private as before. I would recommend you avoid judging a comment because of a metric it neither said nor implied.
    - hurfdurf 23d ago
      Previously the clear target URL of a search result was added to the result-link (or div?) as an additional data-tag (aria or whatever, can't remember), and thus was able to be used with a userscript or similar to replace the Googleified tracking redirect link. And it was also part of the googliefied tracking URL, either way you were able to see the clear target URL. That no longer is possible, you must ask google.com for consent to visit the result link before seeing what the final target URL is.
      - Dylan16807 23d ago
        But the extension doesn't need to use that exact mechanism, and the person that linked it didn't mention specific mechanisms. The purpose of the extension is putting the real URLs back, and it can still do that. And it can still prevent google from knowing which search results you click on, even though OP didn't mention that feature.
nullbio 23d ago
The enshittification continues. Nothing good ever comes from businesses desperately trying to protect their moats rather than making their products better so they don't need to.
- stackghost 23d ago
  Thank Prabhakar Raghavan for that. After destroying Yahoo’s search product he failed up and did the same at Google. In reward for making Google Search materially worse in every way except ad revenue, he failed up again and was promoted to a cushy do-nothing role. Everything wrong with the tech industry, embodied in a single person.
  - ulfw 23d ago
    Well one has to look a level up and ask who the people are who hire such people and why
  - sumedh 23d ago
    If Google's ad revenue is going up he is keeping the shareholders happy.
  - bitpush 23d ago
    Do you have any source outside of the Ed Zitron article (which is highly specualtive?)
  - brainwad 23d ago
    He wasn't the one who merged ads and search into the same org. The blame has to fall on Sundar, surely. > promoted to a cushy do-nothing role The people who this happens to are not perceived as successes. Everyone knows they are gentle firings.
- fnord77 23d ago
  it's metastasized at this point
donmcronald 23d ago
I use ChatGPT for almost all my searches now. I’m not joking.
- nextaccountic 23d ago
  I wish one of those free AIs made a search product already. I don't want my search results to be interspersed with text I guess that the web chat can have a search skill to remove the prose and give only links, plus maybe an excerpt of each result
  - deleted 23d ago
    [deleted]
  - InsideOutSanta 23d ago
    Z.ai has a search MCP server, it should be trivial to use that to build a basic search UI on top.
- entropie 23d ago
  Yeah, its way faster. Its not because chatgpt is so superior. Its just because google search is dogshit. They work on killing the web as we knew it and I fear its kinda working.
  - taurath 23d ago
    Can’t sell Gemini if they were to make Search good. Streaming services already adding in ads to “ad-free” tiers they’ve now named “premium”. Quality of life on the internet has gotten shitty while Reality Classic stays mostly the same, though more expensive.
- Marsymars 23d ago
  Whenever I've tried this, it seems okay if you need an answer to a question, but plain bad if I'm looking for a specific page. e.g. I'm just now looking for the menu for a local restaurant. "restaurantname menu" in Kagi (Google would presumably be similar) returns a link to the menu as the first result in about a second. Or "restaurantname menu !" goes directly to the menu in about a second. Meanwhile, searching "restaurantname menu" in chatgpt takes about 5 seconds to return an embedded map from mapbox showing the location of the restaurant. If I click the restaurant pin on the map, there's no menu link, the 667 reviews have no link or way to view, and the restaurant description literally says "I don't have enough information to identify which local business <restaurantname> refers to." Below the map there's some text: "If you mean <restaurantname> in <place>, here’s the current menu. <restaurantname>". The <restaurantname> link just opens the same card as clicking the pin on the map. After that there's a bullet point list of the menu that ommits a ton of detail and options. After that there's finally a link... that I can click to open up a popup at the bottom of the page with an actual link to the menu. This was literally the first thing that popped into my head, I didn't have to put any effort into finding a query where chatgpt falls on its face.
demibabs 23d ago
Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.
- Cider9986 23d ago
  They were already tracking everything you click but for example if you want to send a link to someone you can't copy the link from the Google result and send it to them, you'd either send them the Google tracking link or go to the website yourself.
- dd8601fn 23d ago
  [flagged]
- rootsudo 23d ago
  Tracking. Tied to use account, length of stay, how many reclixks, etc for marketing by/ads and surveillance.
  - demibabs 23d ago
    But what’s stopping Google from doing this already?
    - 1e1a 23d ago
      JavaScript being disabled - Google was already sending analytics pings when search result links were clicked on, using JS.
      - Cider9986 23d ago
        It doesn't work without JS though.
        deleted 23d ago
        [deleted]
        23d ago
Cider9986 23d ago
Brave search (free) or Kagi (paid) are able to replace Google and not feel like I'm missing out. Brave has its own independent index which is cool.
- deleted 23d ago
  [deleted]
- ibejoeb 23d ago
  I use kagi exclusively because it works. I get the results I need every time, and it's not annoying. I haven't used Google in years. Simply no need. I really hope people keep paying kagi so they stick around.
  - expedited123 23d ago
    I hope you also know they're using Yandex as one of their indexes. https://kagifeedback.org/d/5445-reconsider-yandex-integration-due-to-the-geopolitical-status-quo https://kagifeedback.org/d/5445-reconsider-yandex-integratio... Uruky is the way. 2x cheaper also.
- taspeotis 23d ago
  +1 for Kagi, even if Maps and Shopping and Images aren’t as good as Google (tbh their Images search is good enough 8/10 times) they’ve got the actual web search stuff working quite well.
pbhjpbhj 23d ago
ISPs can presumably correlate the Google query string with the request following the response to the goto and so make a search index? I guess they would charge too much. Do any large ISPs use visit data to feed into a search index?
- TZubiri 23d ago
  ISPs do not see query strings since Google uses HTTPS. ISPs can only see the domain and IP address you are connecting to.
  - zenoprax 23d ago
    Can they even see the domain? Assuming you're not using the ISP's DNS of course.
    - vhcr 23d ago
      Unless you're using DNS over HTTPS they can see the unencrypted DNS traffic. There's also Encrypted Client Hello, but they can also see which IP you're connecting to.
      - dwedge 23d ago
        Also if the domain responds on the bare IP or there's nothing else they've seen on that IP it's a fair assumption. This is without even wondering if their router sends telemetry
        TZubiri 22d ago
        Telemetry isn't a concept that would apply to a router. The information that your router sends, they send to an ISP controlled router, they receive this information for operational purposes, so they don't need to install some client-device application to further collect data.
    - TZubiri 23d ago
fps-hero 23d ago
It hasn’t provided direct URLs for decades? Not exactly new behaviour. I’ve got something that will blow your mind. Google now has tracking analytics, for get this, your business’s phone number. Some “Adsense partner” convinced our web admin to install a little script which changes your phone number on your website so they track phone call enquires back to search engine leads / advertising spend. Yeah no thanks, that was creepy as hell and had it rolled back ASAP. You’ve got to realise the power these tech companies hold over your business. Don’t show up in the search results, someone lists your business as closed in maps, a tracking phone number goes dead so they can’t call you, you might as well have shut up shop and ceased to exist.
- 414techie 23d ago
  Would it be different to you if a third company (not Google) provided that same phone number tracking service?
  - philipov 23d ago
    Would that company then sell that data back to Google? The consolidation of control of information under a single actor is certainly a factor, but it's not that simple, and it's not the only factor. You shouldn't be so eager to outsource as much of your business intelligence as possible.
    - 414techie 23d ago
      Does whether or not they sell to Google matter? What’s the alternative? Roll your own custom phone analytics system?
      - philipov 23d ago
        It matters. The alternative is to use a product that doesn't leak your business intelligence to third parties.
- thrirhrhdbejejs 23d ago
Avicebron 23d ago
It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.
- fc417fc802 23d ago
  Not better, just much _much_ more convenient. And it doesn't have to be a frontier lab. I'm happy to ask any oracle that returns sufficiently good results which is an ever increasing number of them.
water-drummer 23d ago
Gemini does something similar
ur-whale 23d ago
Google Search is dying, and it is showing all the standard symptoms.
rmunn 23d ago
I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look. DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.
- golly_ned 23d ago
  Personalized search results are not ipso facto unreliable.
  - rmunn 23d ago
    It means they're capable of burying news stories that would contradict your worldview and pushing news stories that support your pre-existing biases, leading to more engagement from you (a win from their point of view) but also burying you in an echo chamber. And unless you were in the habit of doing the occasional search in Incognito Mode, you wouldn't know. (And even then, they probably would be able to put together enough clues to figure out your identity even without your Google login cookie). Yes, the fact that they could do it does not prove that they were doing it, not right away. I ditched them as soon as I found out that they could do it, because I was absolutely certain that eventually, they would end up doing it. And I wanted neutral search results, not biased ones, even ones biased towards my own point of view.
    - rileymat2 23d ago
      I subscribe to The New York Times, if I am searching for a news event, I’d appreciate a site that’s not paywalled and I trust be the first result if it is reasonable.
      - CobrastanJorji 22d ago
copemaxxxing 23d ago
I've been using Brave search for a year now, because there is no way to turn off Google AI search. I don't miss Google at all. Goodbye you shit company.
- ranger_danger 23d ago
  > there is no way to turn off Google AI search &udm=web works fine for me, you can use a browser shortcut or extension to enforce it
- brainwad 23d ago
  Adding -ai to the query works fine. Or like the other responder said, use the web tab instead of the all tab.
Lvl999Noob 23d ago
I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?
- 1e1a 23d ago
  It looks like it isn't yet rolled out to some users, try with a different browser session.
- austhrow743 23d ago
  Are you logged in to a Google account?
- genezeta 23d ago
  What I see now, and it's been like this for a while, is this: You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one. So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes. So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.
- stickfigure 23d ago
  I was also super confused because it doesn't do this if you are logged in (to google). I ran the same search in an incognito window and it showed the /goto links.
  - Lvl999Noob 23d ago
    Thanks! This was it. Saw the /goto links in private window.
firefoxd 23d ago
At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information. [0]: https://news.ycombinator.com/item?id=49665572 https://news.ycombinator.com/item?id=49665572
- adtac 23d ago
  but you can click the link and reveal the information. what is being hidden? from who?
  - brainwad 23d ago
    Like the OP says, it's being hidden form scrapers. Like you I don't see a reason for normal users to be concerned.
    - Dylan16807 23d ago
      Unless I want to see what the link is before clicking. Or copy the link.
      - 1e1a 23d ago
        Or navigate to the link without waiting for however long it takes for Google's redirect endpoint to respond.
      - deleted 23d ago
        [deleted]
    - googapologist 23d ago
      Why are you defending the ruination of one of the basic principles of hypertext? Why are Google apologists flagging my other comments?
      - vachina 23d ago
        Ask the grifters and bots. If you don’t adapt get ready to get bulldozed.
Animats 23d ago
Bing has done that for years. Hated it.
faangguyindia 23d ago
Most people i see using ChatGPT for any search related work in everyday life. Maybe Google is observing this?
- keiferski 23d ago
  Yeah seems pretty obvious to me that most people are not going to be using a search engine in 5 years. In the sense of searching for something and combing through the results to find the answer.
  - DarmokTanagra 23d ago
    [dead]
MiroslavPokorny 23d ago
No longer ? Google has been encoding the target url for years.
adtac 23d ago
why is this such a bad thing? it's not really any different from using a uuid as a user facing key, which basically everyone does. and trying to protect your moat isn't automatically a bad thing. they clearly feel it's helping competition, so they're closing a hole. competition is good doesn't mean help your competitors. - coming from someone who's been using fastmail as my personal for ~10 years because i don't want my emails to be backprop fodder
- vish045 23d ago
  I guess someone made a website which google crawled and adding a senf made uuid to it is like google trying to own it rather than just being a true search engine just having index to it.
Dwedit 23d ago
I use Google maybe 10 times a month.
nomilk 23d ago
> Combined with earlier moves like removing &num=100 Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.
- hexagonwin 23d ago
  would you mind sharing such other tool?
  - nomilk 23d ago
    I refer to tools like LLMs, unfortunately (not search engines). If anyone knows of a search engine that returns 100 results, I'm very keen to learn of it
gxs 23d ago
My monthly reminder to use Kagi instead I genuinely forget I’m not using Google until I come across articles like this
lubujackson 23d ago
As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites. There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.
- rkagerer 23d ago
  surprising to see how much they have stripped from our view Don't forget cached pages.
  - 4gotunameagain 23d ago
    Which being public wealth, should be publicly available. Data that were sourced from the public, should be public. We need this legislated, or wealth inequality and fiefdoms will keep growing with the historically known outcomes.
    - brainwad 23d ago
      Didn't Google take it down because of copyright? With the rise of paywalls it became a backdoor around them, but Google doesn't actually have a license to republish the copyrighted text they scraped.
      - embedding-shape 23d ago
        It's not a backdoor is the sites intentionally served Google a version without the paywall so the indexing got "better" than what real users got. The sites kind of dug their own hole here, Google wouldn't even sit on the content unless it was offered up to them in that way.
sehw 23d ago
[dead]
MiroslavPokorny 23d ago
Dont worry in a year or two, Google wont even redirect you to the actual true url, instead everything will be a page with all links rewritten so all http is tunnelled thru them.
- HappMacDonald 23d ago
  Doesn't this just describe AMP?
  - karel-3d 23d ago
    what happened to that anyway? I remember seeing it everywhere, and it just... stopped?
    - seba_dos1 23d ago
      False start.
    - collinmanderson 21d ago
      They used to give preferential treatment to AMP search results, but November 2020 they announced switching to "Core Web Vitals" signal instead. https://developers.google.com/search/blog/2020/11/timing-for-page-experience https://developers.google.com/search/blog/2020/11/timing-for...
  - MiroslavPokorny 22d ago
    Not really because AMP is a cutdown of a fully interactive HTML app.
userbinator 23d ago
I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this... For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article. Almost exactly a year ago, Google stopped working without JS. I stopped using Google. Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out. Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.
ma2kx 23d ago
At least they seem still provide results for my searxng instance. I mean sure, they are horrible but duckduckgo just blocks most queries (and I'm the only person using the ip / seraxng instance)... Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.
deleted 23d ago
[deleted]
deleted 23d ago
[deleted]
JsonDemWitOster 23d ago
While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling. While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.
- Jakob 23d ago
  Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash https://kapeli.com/dash It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.
  - skinfaxi 23d ago
    What do you use now?
    - ormax3 23d ago
      https://devdocs.io/ https://devdocs.io/
- CobrastanJorji 23d ago
  Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.
- orbital-decay 23d ago
  https://yacy.net/ https://yacy.net/
angry_octet 23d ago
This is such an incredibly annoying and deliberate defect. I want an extension that will let me resolve the true url without visiting the site, or even rewrites the entire page to show the url.
aucisson_masque 23d ago
So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots. Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.
- orbital-decay 23d ago
  Yeah. I might be wrong but I think they only served direct links for a relatively short time in their history, early on in the 90's and in recent years with ping, which they used to track clicks anyway. At least half of their history they used either the 302 redirects or onmousedown link rewriting (which was terrible). And I'm not even starting on AMP.
- deleted 23d ago
  [deleted]
- jolmg 23d ago
  Privacy-wise, it enables them to track the result you click on. Also, though more niche, it would make archived search result pages (e.g. on the Wayback Machine) less useful.
  - maxchehab 23d ago
    I think they always knew what link you clicked on (when most of the web has a ga4 tag on the other side of the link)
    - jolmg 23d ago
      You can block Google Analytics with your browser. You can't block needing to ask Google for the URL of the result you want to visit.
      - nprateem 23d ago
        If you're that concerned, use another search engine
skarlso 23d ago
Who the hell uses Google still? Kagi is the way to go! Even though it’s paid it’s worth it.
parisiansam 23d ago
Been using Brave Search now for a while including its Brave AI and aside of sporadic times I never needed Google (albeit Brave Search is slower to Google, you get used to it)
JohnTHaller 23d ago
Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.
- googapologist 23d ago
  Note: Irrelevant. The reported behavior exists, does it not? Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.
  - vachina 23d ago
    It’s very relevant because the author has a conflict in interest. They can wax poetic about “open internet” but really their business depend on it.
    - mitxela 23d ago
      Kagi also uses this
  - JohnTHaller 23d ago
    It's relevant to know when the source of information is biased due to a conflict of interest. The source's business is apparently scraping Google (and Bing, etc).
Founderarcstone 23d ago
wow thanks for sharing
ChrisArchitect 23d ago
Append (for logged out users) to the title.
rkagerer 23d ago
I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course. I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms. I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought. I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today. I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.
- fc417fc802 23d ago
  The right click copy thing annoys me to no end.
  - seba_dos1 23d ago
    I switched to DuckDuckGo around 2018 or so, which was when I realized that Google's results quality has deteriorated so much that I won't lose anything of value by doing so. This thread is how I learn about the atrocities that Google Search is committing these days. Which is an elaborate way to say: you don't have to live like this.
    - 23d ago
deleted 23d ago
[deleted]
on_the_train 23d ago
I'm confused. Google has done this for several years already. I have a Firefox extension installed that reverts it - it's several years old.
- josephcsible 23d ago
  They changed how they do it, such that it's no longer possible for extensions to revert it.
dalton74 23d ago
The udm=web parameter has been doing the job for a while now. It's a shame you have to know about an undocumented flag to get the old behavior.
- brainwad 23d ago
  That flag is the result of clicking on the "Web" tab of the results page, so it kinda is a documented feature. All the tabs have their own udm value (e.g. image search is 2, AI mode is 50, etc.).
1317 23d ago
> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. what? the Location header is the redirect, no?
stncls 23d ago
When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off. I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API. What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?
- telotortium 23d ago
  I’m guessing that the knowledge of the techniques to support large-scale web search have diffused out of Google - it has been a few decades after all. Not to mention that the distributed system knowledge that used to live only in Google was either published by Google or cloned in other projects, usually made by ex-Googlers. And you can now rent capacity at scales that 20 years ago required Google to build lots of their own data centers. Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.
karel-3d 23d ago
Why can't agents just click on the link and follow it? I don't get it
Almondsetat 23d ago
"google.com/goto considered harmful"
millicentricism 23d ago
Hopefully this also helps against the rampant CTR manipulation that’s made some search results purely a measure of spend.
bomewish 23d ago
I really like the idea of turning the search index into a public utility. It is one of those natural monopoly coordination problem things. Just quasi nationalise it for economic efficiency. Ofc google can still sell adds against their own ui (like everyone else). Hopefully this move will move that idea closer to reality.
- dgellow 23d ago
  With the current administration? They would manipulate the results to ridiculous extent
  - kursus 23d ago
    So would the previous and the next one, this is not situational.
ArcHound 23d ago
It's funny. Recently I looked into what Google is doing (see at https://blog.miloslavhomer.cz/how-google-sees-your-site/ https://blog.miloslavhomer.cz/how-google-sees-your-site/). It's a lot of work to get the data to build an index. Why would they give it to everyone for free?
- kolinko 23d ago
  Becuse they are a monopolist and we can require certain things from monopolists. Similarly, Bell Labs was kind of required to release transistor for anyone to license - it was a part of social contract that they were allowed to maintain their monopoly in exchange for releasing certain parts of technology. Alternatively, they could be split up and their indexing division made an independent company selling to anyone on a free market.
  - ArcHound 23d ago
    I tend to agree - businesses should give back. But they can also keep some secrets. R&D has steep cost. Publishing it all gives your competitors an advantage.
    - kolinko 23d ago
      It has steep costs, but with monopolists they have other ways of extracting value from the inventions. Awesome book about the history if Bell Labs - virtually all semiconductor tech we use today was created there (transistors, ics, solar, lasers, fiber optics, telecom satellites…), and they had to license it to be allowed to maintain their monopoly status. https://www.amazon.pl/Idea-Factory-Great-American-Innovation/dp/1594203288 https://www.amazon.pl/Idea-Factory-Great-American-Innovation...
      - ArcHound 23d ago
        Thanks for the recommendation, I do see your points. We should be treating monopolies differently.
osquar 23d ago
Is it a move to sell more of the paid Google Search API calls? If so, that's a sign of distress. Surely, the cost of serving search results to bots isn't that high
tobinfricke 23d ago
This seems good. I'm not sure why I should be upset that Google is preventing abuse of its service.
- kolinko 23d ago
  I think it’s the other way around - why should you care about a monopolist’s interests?
grey-area 23d ago
If you don’t like this, just don’t use Google folks. There are alternatives, use them.
Pxtl 23d ago
"There's nothing to explain. You're trying to kidnap what I've rightfully stolen."
nottorp 23d ago
Of course, this move is user hostile because you don't see what you're being sent to.
- tantalor 23d ago
  You can see the domain printed on the page, right?
  - nottorp 23d ago
    I had to check :) I use duckduckgo mostly. They still display the domain, at least, yes. Wonder for how long.
avadodin 23d ago
What are you going to steal from Google? The Internet has been dead for years and, after the Scrapocalypse, the small living remnants are behind a login wall. Google can not provide you anything you couldn't find on either your local ZIM archive or the Media you consume.
shevy-java 23d ago
Google wants to build up a private web here. We already know this from AMP before. Also, the search results are total garbage now. Just give it a try and you see how useless the UI is, with AI results first, then tons of commercial go-to entries and only then a few links that are often also totally useless and irrelevant. Google optimised towards crap. We really need to get rid of Google. Qwant results are now a bit better than before, but still not that great, and I hate that its default UI is a clone of Google. We need more alternatives. Let's get rid of Google once and for all - it has disappointed too many people now.
deleted 23d ago
[deleted]
Hendrikto 23d ago
> The url parameter uses a custom, Google-specific encoding. So that will be figured out, making the whole exercise void. Google knows this. What is the real goal here?
- Uzazo 23d ago
  Imagine it being the ID of the database row holding the actual URL. There's no way to figure it out without access to that database.
- tgsovlerkhgsel 23d ago
  "Figuring it out" doesn't help if it's encrypted with a key only Google holds or a reference to some database record that you don't have. In either case, the only way to resolve this is to ask Google, which means they can track it as a click (and rate limit etc.).
mrkramer 23d ago
They are allowed to scrape everybody else but get their feelings hurt when they get scraped....oh yea they respect robots.txt. Guess what, scraping everything that is public is legal.
jasonjmcghee 23d ago
I've never heard of this particular SERP provider, but some marketer is very excited they wrote this blog post right now (500+ upvotes on an seo blog post). And their fix here - they just resolve all the urls- which I suppose could make the service slightly more expensive? But otherwise isn't that what every similar provider/ crawler etc will do and this change will only hurt users?
1vuio0pswjnm7 23d ago
1789154298 | Google's Emissions Climbed 48% Since 2019 due to AI | https://www.gadgetreview.com/googles-emissions-climbed-48-since-2019-ai-is-why https://www.gadgetreview.com/googles-emissions-climbed-48-si... | https://news.ycombinator.com/item?id=49663855 https://news.ycombinator.com/item?id=49663855 | 0 comments
gvieri 23d ago
I had tried now but I don't see the goto. Is it possible that has been deployed only for USA users ? (I work and live in UE). If so: probably a vpn can help you for a while. In the future: I think I'll really go for payed search engine.
pmarreck 23d ago
Is there any market for an open-source internet search engine that is paid for by honest ads?
chews 23d ago
now is time to start the ai to spam/reverse the - google.com/goto for url resolution, it can't be that hard to reverse engineer how the hash works. It was doable for twitter and YouTube, it will be doable here too...
herpdyderp 23d ago
That's it. I'm done with Google search. I just found you can add Kagi to Safari: https://apps.apple.com/us/app/kagi-for-safari/id1622835804 https://apps.apple.com/us/app/kagi-for-safari/id1622835804
lofaszvanitt 23d ago
Divide et impera. People need to learn. Get together, do things together, create alternatives, use that, maintain position, do not let outsider saboteurs neuter the project. Problem solved.
txheilmann 23d ago
[dead]
nijave 23d ago
Hmm that's actually pretty clever. They can serve each result page slightly different encrypted links and it should be obvious right away if it's a SERP bot (trying to grab a page of links) or a human that just picks a few here or there. I wonder if this would also work on other sites getting hammered with bots. Allow each anonymous user 1 "real" page load then turn the rest into encrypted links that the web server can decrypt. If a session cookie with reputation exists, stop screwing with the links. Kind of annoying but it'd allow tracking if the same agent/bot is churning through IPs/User Agents.
hivixop 23d ago
[dead]
pvillano 23d ago
If you were scraping only Google with all the IPs you can get, then this change really slows you down. If you're trying to fight scrapers on a small site, delay links can only flatten bursts. If bots can only scrape at human speed per IP, they can just scrape 100x as many sites at the same time. Once every bot operator does that, total traffic will return to the original level.
thwarted 23d ago
What does this mean? This text appears on this page https://www.autom.dev/blog/google-search-goto-links https://www.autom.dev/blog/google-search-goto-links > The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. And this text appears on this page https://www.autom.dev/blog/google-goto-url-fix https://www.autom.dev/blog/google-goto-url-fix > Do not follow the redirect. Read Location. That's what a redirect is, reading the value of the location header and then requesting it. How do you not follow the redirect by reading the location header? Once you've made the request to the /goto url, with GET or HEAD, to get the location header, google knows you're interested in whatever it is putting in the location header and can assume you're going to go there, if you're letting the User Agent (curl or the browser) go there for you or not.
echoangle 23d ago
> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. What does this mean? Isn’t the location header the redirect? Am I not following the redirect by requesting the location header url?
rat_on_the_run 23d ago
Can confirm that when using google not logged in. Now when sharing a link from google search, I won't get the actual link. This will certainly help google's tracking. Google has gone so bad over the past few years. You only get like 8 results per page. I remember there was a time that I wonder how a site get reached if it ranked on the second page, when I can set the number of results to be 50. The censorship is also really bad, and google doesn't even tell you the results are censored, returning totally nonsense results while other search engines work normally. I've been supporting Brave search which returns 20 results per page and has other features. It used to be not good a few years ago, but now the results are often better than google's.
1vuio0pswjnm7 23d ago
"Combined with earlier moves like removing &num=100 and tightening BotGuard/SearchGuard, Google is steadily raising the cost of naive SERP scraping." Another "move" is suing companies like Autom, e.g., SerpApi Google's Amended Complaint from their suit against SerpApi https://ia801008.us.archive.org/25/items/gov.uscourts.cand.461513/gov.uscourts.cand.461513.45.0.pdf https://ia801008.us.archive.org/25/items/gov.uscourts.cand.4... "30. Copyright holders have authorized Google to implement access controls like SearchGuard for the content they license to Google, and in some cases insisted that Google do so. Googles authorization takes many forms. For example, Google has an agreement with a prominent licensing partner that holds copyrights to millions of works that it licenses Google to use in its Search results. Under the parties agreement, versions of which date back to 2017, Google is not only authorized, it is obligated to use commercially reasonable efforts to safeguard the licensed content against unauthorized third-party access. Other license agreements contain similar obligations. For example, another major content provider requires that Google ensure the content it licenses will not be available for download by third parties, thereby authorizing the implementation of technical access controls." "31. In other cases, Googles authorization to implement access control measures like SearchGuard is part and parcel of the grant of licenses themselves, as Google and its licensors recognize that the value of the licensed rights would be undermined if others were free to access, take and resell the licensed content without restriction. For example, Google has a licensing agreement with Reddit, under which Reddit licenses Google to use the copyrighted content of both Reddit and its users in Search Services." "32. Googles licensing partners have also expressly requested that Google prevent unauthorized access to licensed content. For example, when Reddit suspected that scrapers like SerpApi were accessing, taking, and reselling the content that Reddit had licensed to Google, it specifically asked Google to employ technical measures to prevent such unauthorized appropriation." But this does not account for material that is not covered by the "license with a prominent licensing partner", its license with "another major content provider" or its agreement with Reddit Google not only uses SearchGuard on SERPs containing links to the content covered by these licenses, it uses SearchGuard on _all_ SERPs Google needs more than a "goto" update. It needs to update its terms to require _all_ copyright holders for the materials it has indexed and cached to give Google authorisation to use "technological protection measures" to deny access to certain members of the public, e.g., Google's perceived competitors including any Google user who "searches too fast"
ValentineC 22d ago
One really annoying thing that I just found in practice is that this change breaks copying and pasting links directly from the Google search results. Now, I need to click on the link and access the page to “unfurl” the URL so that I can paste a nice, human-parseable URL into a chat. Google may have just made me actively want to switch away from it.
daft_pink 22d ago
so glad i use kagi
NishanStepak 22d ago
I am curious what the difference between scraping and linking. I have a project that links to many different sites, but does not scrape the content. I have been accused of scraping because of the number of links. Some sites have rate limits which limit the number of times you can go to a site from another site. Gutenberg does this. Internet Archive does not have rate limits. My site is not a link farm, however, this can be a pattern of unwanted backlinks in some cases. I am curious how this fits in with Googles Anti-scraping updated. On August 18-20, Google did a massive automated anti-spam update with the algorithm. There was no way to contest what your site was. There is a change being done in how Google is deciding what sites are spam or not. From what I am noticing many sites that with low traffic that are academic or personal were affected by this update.
latcom007 21d ago
[flagged]
bitbasher 21d ago
I've found duckduckgo to be a pretty good replacement these days. A few years ago I found myself using !g often to hop over to google, but these days I've been getting by without any !g ... I'm not sure if it's because duckduckgo has gotten better or google has gotten worse, but either way it's welcome.
javier2 20d ago
Oh, Google does not like others stealing and indexing their content?! How dare they!
nerdyadventurer 16d ago
While they can scrape all whole web, ignoring copyright content. Sometimes Gemini even output images with watermarks from a popular licensed media seller. Mean while Aron Swartz was prosecuted for scraping[1] and finally he end up taking his life. 1: https://blog.curiousquail.com/im-upset-again-about-a-co-creator-of-rss-being-prosecuted-for-something-meta-is-doing-with-little-consequence/ https://blog.curiousquail.com/im-upset-again-about-a-co-crea...

678 points by 1e1a 23d ago

JohnFen 23d ago

If I hadn't already ditched Google long ago, this would be more than enough to chase me away.

scottyah 23d ago
Why? I don't really see why this is bad, other than if your company blocks url shorteners/redirects. What are you losing?
- nextaccountic 23d ago
  I use a Firefox extension to rewrite those Google redirects into plain links, to remove Google tracking of which links I click. The extension is broken now
- JohnFen 23d ago
  Because I wouldn't want Google to know what links I click on. When I used to use Google, and they added such redirects that included the destination URL as a parameter, I'd edit the link to make it just that URL before resolving it. This new scheme would make that impossible.
  - what 23d ago
    They already knew which links you clicked on though…
    - googapologist 23d ago
      [flagged]
    - deleted 23d ago
      [deleted]
    - JohnFen 23d ago
      How? I never allowed Javascript or anything.
    - smart_asslop 22d ago
      Even with JS enabled, they did not. Linked below is an explanation which includes confirmation of that fact by a member of the uBlock Origin team. https://news.ycombinator.com/item?id=49680338 https://news.ycombinator.com/item?id=49680338
  - brainwad 23d ago
    They own the JS on the page, they don't need the redirects to know where you clicked...
    - googapologist 23d ago
      [flagged]
    - deleted 23d ago
      [deleted]
    - smart_asslop 22d ago
      Hello! It appears that you are disseminating misinformation. A uBlock Origin developer now confirms, regarding Google results click tracking: It existed in a way or another, but it could be blocked. Now it can't. Which corroborates the mysteriously flagged sibling comment. https://old.reddit.com/r/uBlockOrigin/comments/1we7491/how_to_bypass_googles_goto_url_rewrite_for_search/p9fnjzq/ https://old.reddit.com/r/uBlockOrigin/comments/1we7491/how_t... Next time, please consider listening to people who know how the web works and avoid overconfidently parroting what Big Tech gatekeepers want you to think. Thanks!
      - brainwad 22d ago
        It's not misinformation. By default Google has been able to click track with JS. If you disable JS, they don't serve you search (https://serpapi.com/blog/google-now-requires-javascript/ https://serpapi.com/blog/google-now-requires-javascript/). Yes, if you are tricky, you can block just the part of the JS that does click tracking, probably. But most users can't do that. They therefore were already getting 99% complete click tracking before this change. This change is not about click tracking, it's about blocking scrapers. And btw about the result filtering the uBlock dev wants to do: of course this can still be done, if you preload all the goto links on the page and then inline the real URLs returned by the preload requests. Perhaps not easy to do as an extension but at a browser level (Brave?) totally possible. Though Google might then think your users are scrapers...
        22d ago
- Fabricio20 23d ago
  Blocking URL shorteners and google-ad links yes! Personally for me it's also the fact that this is effectively an unresolvable URL shortener, store that link somewhere and it will most likely be dead. Can't copy link anymore and paste it on a notepad or chat app to check it out later as there isn't a guarantee it will load at all (ie: the problem with url shorteners).
- exlr 23d ago
  [flagged]
tarkin2 23d ago
They are forcing my further and further towards kagi. I do hope the fawning on here is at least partly justified.
- JohnFen 23d ago
  Kagi is what I settled on a couple of years ago. I do think sometimes the fawning is over the top, but it is a solid search engine and tended to give me a bit better results than Google out of the box. The real win, though, is that you can give various sites a weight, so the search results will prefer or avoid sites according to your desires. Once I had that going, my search results tended to be much better than Google.
  - tarkin2 23d ago
    Yeah, that's the thing for me: filtering out the SEO crap that Google happily serves up. Google's actual search results are a waste of time visiting, both for the mindless CEO content and the ad-laden, analytics-happy, javascript-heavy websites. So I tend to use the AI overview. But plugging myself into the all-seeing corporate oracle, that grew on all the web's content, and now seeks to supplant it seems unseemly. It's just sad that kagi will likely only be a fringe thing, and google will continue to promote these foul, foul, mindless websites and then supplant them with its AI.
  - Zambyte 23d ago
    Yup, same here. Been using Kagi for years. It's boring, it just works. Hoping they can stay that way.
    - lukeify 23d ago
      My ongoing concern with Kagi is they always seem to be focused on sidequests like their Orion browser and their LLM-powered Translate tool. Maybe that's interesting for some people, but I can't help but feel like I just want a damn search engine than works.
      - 23d ago

1e1a 23d ago

Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url=<opaque base64 string>. The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL. Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.

darknavi 23d ago
> Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating. Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced. I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.
- mentalpiracy 23d ago
  Wow - this exact same bug has been happening to me too. I gave up on troubleshooting it after the first few attempts came up with nothing, assumed it was just unique to me.
  - Maxion 23d ago
    I searched for this a few months ago and found some mentions of this bug, but yes it affects me too. Can stall up to 20 seconds+ sometimes. Chrome is fine.
  - darknavi 23d ago
    I had no idea others hit this, I just assumed it was some wonky setup I have locally. I hope we both figure it out one day haha (or Firefox does)
- Starlevel004 23d ago
  > On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, Have you tried disabling HTTP/2?

deleted 23d ago

[deleted]

liquid_thyme 23d ago

I've been using the 'ClearURLs' FF addon for quite a while now, and it gives you the URLs back in the results. https://docs.clearurls.xyz/ https://docs.clearurls.xyz/

joeblubaugh 23d ago
That wouldn’t work at all with this url format - the url can only be resolved by requesting the goto link. I recommend reading more than the headline
- deleted 23d ago
  [deleted]
- Dylan16807 23d ago
  > That wouldn’t work at all with this url format - the url can only be resolved by requesting the goto link. What is "that" you say wouldn't work? The extension can work fine by requesting every goto link upfront. As far as your google searches go, this is just as private as before. I would recommend you avoid judging a comment because of a metric it neither said nor implied.
  - hurfdurf 23d ago
    Previously the clear target URL of a search result was added to the result-link (or div?) as an additional data-tag (aria or whatever, can't remember), and thus was able to be used with a userscript or similar to replace the Googleified tracking redirect link. And it was also part of the googliefied tracking URL, either way you were able to see the clear target URL. That no longer is possible, you must ask google.com for consent to visit the result link before seeing what the final target URL is.
    - Dylan16807 23d ago
      But the extension doesn't need to use that exact mechanism, and the person that linked it didn't mention specific mechanisms. The purpose of the extension is putting the real URLs back, and it can still do that. And it can still prevent google from knowing which search results you click on, even though OP didn't mention that feature.

nullbio 23d ago

The enshittification continues. Nothing good ever comes from businesses desperately trying to protect their moats rather than making their products better so they don't need to.

stackghost 23d ago
Thank Prabhakar Raghavan for that. After destroying Yahoo’s search product he failed up and did the same at Google. In reward for making Google Search materially worse in every way except ad revenue, he failed up again and was promoted to a cushy do-nothing role. Everything wrong with the tech industry, embodied in a single person.
- ulfw 23d ago
  Well one has to look a level up and ask who the people are who hire such people and why
- sumedh 23d ago
  If Google's ad revenue is going up he is keeping the shareholders happy.
- bitpush 23d ago
  Do you have any source outside of the Ed Zitron article (which is highly specualtive?)
- brainwad 23d ago
  He wasn't the one who merged ads and search into the same org. The blame has to fall on Sundar, surely. > promoted to a cushy do-nothing role The people who this happens to are not perceived as successes. Everyone knows they are gentle firings.
fnord77 23d ago
it's metastasized at this point

donmcronald 23d ago

I use ChatGPT for almost all my searches now. I’m not joking.

nextaccountic 23d ago
I wish one of those free AIs made a search product already. I don't want my search results to be interspersed with text I guess that the web chat can have a search skill to remove the prose and give only links, plus maybe an excerpt of each result
- deleted 23d ago
  [deleted]
- InsideOutSanta 23d ago
  Z.ai has a search MCP server, it should be trivial to use that to build a basic search UI on top.
entropie 23d ago
Yeah, its way faster. Its not because chatgpt is so superior. Its just because google search is dogshit. They work on killing the web as we knew it and I fear its kinda working.
- taurath 23d ago
  Can’t sell Gemini if they were to make Search good. Streaming services already adding in ads to “ad-free” tiers they’ve now named “premium”. Quality of life on the internet has gotten shitty while Reality Classic stays mostly the same, though more expensive.
Marsymars 23d ago
Whenever I've tried this, it seems okay if you need an answer to a question, but plain bad if I'm looking for a specific page. e.g. I'm just now looking for the menu for a local restaurant. "restaurantname menu" in Kagi (Google would presumably be similar) returns a link to the menu as the first result in about a second. Or "restaurantname menu !" goes directly to the menu in about a second. Meanwhile, searching "restaurantname menu" in chatgpt takes about 5 seconds to return an embedded map from mapbox showing the location of the restaurant. If I click the restaurant pin on the map, there's no menu link, the 667 reviews have no link or way to view, and the restaurant description literally says "I don't have enough information to identify which local business <restaurantname> refers to." Below the map there's some text: "If you mean <restaurantname> in <place>, here’s the current menu. <restaurantname>". The <restaurantname> link just opens the same card as clicking the pin on the map. After that there's a bullet point list of the menu that ommits a ton of detail and options. After that there's finally a link... that I can click to open up a popup at the bottom of the page with an actual link to the menu. This was literally the first thing that popped into my head, I didn't have to put any effort into finding a query where chatgpt falls on its face.

demibabs 23d ago

Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.

Cider9986 23d ago
They were already tracking everything you click but for example if you want to send a link to someone you can't copy the link from the Google result and send it to them, you'd either send them the Google tracking link or go to the website yourself.
dd8601fn 23d ago
[flagged]
rootsudo 23d ago
Tracking. Tied to use account, length of stay, how many reclixks, etc for marketing by/ads and surveillance.
- demibabs 23d ago
  But what’s stopping Google from doing this already?
  - 1e1a 23d ago
    JavaScript being disabled - Google was already sending analytics pings when search result links were clicked on, using JS.
    - Cider9986 23d ago
      It doesn't work without JS though.
      - deleted 23d ago
        [deleted]
      - 23d ago

Cider9986 23d ago

Brave search (free) or Kagi (paid) are able to replace Google and not feel like I'm missing out. Brave has its own independent index which is cool.

deleted 23d ago
[deleted]
ibejoeb 23d ago
I use kagi exclusively because it works. I get the results I need every time, and it's not annoying. I haven't used Google in years. Simply no need. I really hope people keep paying kagi so they stick around.
- expedited123 23d ago
  I hope you also know they're using Yandex as one of their indexes. https://kagifeedback.org/d/5445-reconsider-yandex-integration-due-to-the-geopolitical-status-quo https://kagifeedback.org/d/5445-reconsider-yandex-integratio... Uruky is the way. 2x cheaper also.
taspeotis 23d ago
+1 for Kagi, even if Maps and Shopping and Images aren’t as good as Google (tbh their Images search is good enough 8/10 times) they’ve got the actual web search stuff working quite well.

pbhjpbhj 23d ago

ISPs can presumably correlate the Google query string with the request following the response to the goto and so make a search index? I guess they would charge too much. Do any large ISPs use visit data to feed into a search index?

TZubiri 23d ago
ISPs do not see query strings since Google uses HTTPS. ISPs can only see the domain and IP address you are connecting to.
- zenoprax 23d ago
  Can they even see the domain? Assuming you're not using the ISP's DNS of course.
  - vhcr 23d ago
    Unless you're using DNS over HTTPS they can see the unencrypted DNS traffic. There's also Encrypted Client Hello, but they can also see which IP you're connecting to.
    - dwedge 23d ago
      Also if the domain responds on the bare IP or there's nothing else they've seen on that IP it's a fair assumption. This is without even wondering if their router sends telemetry
      - TZubiri 22d ago
        Telemetry isn't a concept that would apply to a router. The information that your router sends, they send to an ISP controlled router, they receive this information for operational purposes, so they don't need to install some client-device application to further collect data.
  - TZubiri 23d ago

fps-hero 23d ago

It hasn’t provided direct URLs for decades? Not exactly new behaviour. I’ve got something that will blow your mind. Google now has tracking analytics, for get this, your business’s phone number. Some “Adsense partner” convinced our web admin to install a little script which changes your phone number on your website so they track phone call enquires back to search engine leads / advertising spend. Yeah no thanks, that was creepy as hell and had it rolled back ASAP. You’ve got to realise the power these tech companies hold over your business. Don’t show up in the search results, someone lists your business as closed in maps, a tracking phone number goes dead so they can’t call you, you might as well have shut up shop and ceased to exist.

414techie 23d ago
Would it be different to you if a third company (not Google) provided that same phone number tracking service?
- philipov 23d ago
  Would that company then sell that data back to Google? The consolidation of control of information under a single actor is certainly a factor, but it's not that simple, and it's not the only factor. You shouldn't be so eager to outsource as much of your business intelligence as possible.
  - 414techie 23d ago
    Does whether or not they sell to Google matter? What’s the alternative? Roll your own custom phone analytics system?
    - philipov 23d ago
      It matters. The alternative is to use a product that doesn't leak your business intelligence to third parties.
thrirhrhdbejejs 23d ago

Avicebron 23d ago

It's sad that instead of searching things other people put up, we're basically asking sam or dario oracle to tell us the truth. The people should be furious. But we've internalized this idea that they are somehow better.

fc417fc802 23d ago
Not better, just much _much_ more convenient. And it doesn't have to be a frontier lab. I'm happy to ask any oracle that returns sufficiently good results which is an ever increasing number of them.

water-drummer 23d ago

Gemini does something similar

ur-whale 23d ago

Google Search is dying, and it is showing all the standard symptoms.

rmunn 23d ago

I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look. DDG's search results have been poorer recently than they used to — I often see completely unrelated results (to the point of my saying "Why in the world did that come back as a search result??!?") starting from page 2. And yet, I still use them, simply because they aren't Google.

golly_ned 23d ago
Personalized search results are not ipso facto unreliable.
- rmunn 23d ago
  It means they're capable of burying news stories that would contradict your worldview and pushing news stories that support your pre-existing biases, leading to more engagement from you (a win from their point of view) but also burying you in an echo chamber. And unless you were in the habit of doing the occasional search in Incognito Mode, you wouldn't know. (And even then, they probably would be able to put together enough clues to figure out your identity even without your Google login cookie). Yes, the fact that they could do it does not prove that they were doing it, not right away. I ditched them as soon as I found out that they could do it, because I was absolutely certain that eventually, they would end up doing it. And I wanted neutral search results, not biased ones, even ones biased towards my own point of view.
  - rileymat2 23d ago
    I subscribe to The New York Times, if I am searching for a news event, I’d appreciate a site that’s not paywalled and I trust be the first result if it is reasonable.
    - CobrastanJorji 22d ago

copemaxxxing 23d ago

I've been using Brave search for a year now, because there is no way to turn off Google AI search. I don't miss Google at all. Goodbye you shit company.

ranger_danger 23d ago
> there is no way to turn off Google AI search &udm=web works fine for me, you can use a browser shortcut or extension to enforce it
brainwad 23d ago
Adding -ai to the query works fine. Or like the other responder said, use the web tab instead of the all tab.

Lvl999Noob 23d ago

I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?

1e1a 23d ago
It looks like it isn't yet rolled out to some users, try with a different browser session.
austhrow743 23d ago
Are you logged in to a Google account?
genezeta 23d ago
What I see now, and it's been like this for a while, is this: You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one. So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes. So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.
stickfigure 23d ago
I was also super confused because it doesn't do this if you are logged in (to google). I ran the same search in an incognito window and it showed the /goto links.
- Lvl999Noob 23d ago
  Thanks! This was it. Saw the /goto links in private window.

firefoxd 23d ago

At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information. [0]: https://news.ycombinator.com/item?id=49665572 https://news.ycombinator.com/item?id=49665572

adtac 23d ago
but you can click the link and reveal the information. what is being hidden? from who?
- brainwad 23d ago
  Like the OP says, it's being hidden form scrapers. Like you I don't see a reason for normal users to be concerned.
  - Dylan16807 23d ago
    Unless I want to see what the link is before clicking. Or copy the link.
    - 1e1a 23d ago
      Or navigate to the link without waiting for however long it takes for Google's redirect endpoint to respond.
    - deleted 23d ago
      [deleted]
  - googapologist 23d ago
    Why are you defending the ruination of one of the basic principles of hypertext? Why are Google apologists flagging my other comments?
    - vachina 23d ago
      Ask the grifters and bots. If you don’t adapt get ready to get bulldozed.

Animats 23d ago

Bing has done that for years. Hated it.

faangguyindia 23d ago

Most people i see using ChatGPT for any search related work in everyday life. Maybe Google is observing this?

keiferski 23d ago
Yeah seems pretty obvious to me that most people are not going to be using a search engine in 5 years. In the sense of searching for something and combing through the results to find the answer.
- DarmokTanagra 23d ago
  [dead]

MiroslavPokorny 23d ago

No longer ? Google has been encoding the target url for years.

adtac 23d ago

why is this such a bad thing? it's not really any different from using a uuid as a user facing key, which basically everyone does. and trying to protect your moat isn't automatically a bad thing. they clearly feel it's helping competition, so they're closing a hole. competition is good doesn't mean help your competitors. - coming from someone who's been using fastmail as my personal for ~10 years because i don't want my emails to be backprop fodder

vish045 23d ago
I guess someone made a website which google crawled and adding a senf made uuid to it is like google trying to own it rather than just being a true search engine just having index to it.

Dwedit 23d ago

I use Google maybe 10 times a month.

nomilk 23d ago

> Combined with earlier moves like removing &num=100 Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.

hexagonwin 23d ago
would you mind sharing such other tool?
- nomilk 23d ago
  I refer to tools like LLMs, unfortunately (not search engines). If anyone knows of a search engine that returns 100 results, I'm very keen to learn of it

gxs 23d ago

My monthly reminder to use Kagi instead I genuinely forget I’m not using Google until I come across articles like this

lubujackson 23d ago

As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". It is surprising to see how much they have stripped from our view - long tail results, actual results for product reviews and not ad spam, no preference for 20 page recipe sites. There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google. It makes me nostalgic for that brief window of time when the web was truly uncontrolled, when page rank had meaning and you didn't know if your search would return 0 results or 4,000 pages, which you could actually browse.

rkagerer 23d ago
surprising to see how much they have stripped from our view Don't forget cached pages.
- 4gotunameagain 23d ago
  Which being public wealth, should be publicly available. Data that were sourced from the public, should be public. We need this legislated, or wealth inequality and fiefdoms will keep growing with the historically known outcomes.
  - brainwad 23d ago
    Didn't Google take it down because of copyright? With the rise of paywalls it became a backdoor around them, but Google doesn't actually have a license to republish the copyrighted text they scraped.
    - embedding-shape 23d ago
      It's not a backdoor is the sites intentionally served Google a version without the paywall so the indexing got "better" than what real users got. The sites kind of dug their own hole here, Google wouldn't even sit on the content unless it was offered up to them in that way.

sehw 23d ago

[dead]

MiroslavPokorny 23d ago

Dont worry in a year or two, Google wont even redirect you to the actual true url, instead everything will be a page with all links rewritten so all http is tunnelled thru them.

HappMacDonald 23d ago
Doesn't this just describe AMP?
- karel-3d 23d ago
  what happened to that anyway? I remember seeing it everywhere, and it just... stopped?
  - seba_dos1 23d ago
    False start.
  - collinmanderson 21d ago
    They used to give preferential treatment to AMP search results, but November 2020 they announced switching to "Core Web Vitals" signal instead. https://developers.google.com/search/blog/2020/11/timing-for-page-experience https://developers.google.com/search/blog/2020/11/timing-for...
- MiroslavPokorny 22d ago
  Not really because AMP is a cutdown of a fully interactive HTML app.

userbinator 23d ago

I was suspicious when they started obfuscating URLs in their own browser, then on their SERPs, and now this... For many years, I had my filtering proxy rewrite the URLs in the way mentioned in the article. Almost exactly a year ago, Google stopped working without JS. I stopped using Google. Now they're upping the game, and as the article (which is a bit of marketing itself) admits, those who have the resources can still blast through these obstacles while those who don't are locked out. Since the article brings up "AI scrapers", I'll just point it out as being the latest scare-tactic for coercing people to give up the privacy, anonymity, and (browser) freedom of an open interoperable Internet.

ma2kx 23d ago

At least they seem still provide results for my searxng instance. I mean sure, they are horrible but duckduckgo just blocks most queries (and I'm the only person using the ip / seraxng instance)... Next i'll do is to implement tavilly, exa, tinyfish etc. as search engines for searxng. No agents, no mcp, just their search api endpoint.

deleted 23d ago

[deleted]

deleted 23d ago

[deleted]

JsonDemWitOster 23d ago

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling. While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.

Jakob 23d ago
Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash https://kapeli.com/dash It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.
- skinfaxi 23d ago
  What do you use now?
  - ormax3 23d ago
    https://devdocs.io/ https://devdocs.io/
CobrastanJorji 23d ago
Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.
orbital-decay 23d ago
https://yacy.net/ https://yacy.net/

angry_octet 23d ago

This is such an incredibly annoying and deliberate defect. I want an extension that will let me resolve the true url without visiting the site, or even rewrites the entire page to show the url.

aucisson_masque 23d ago

So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots. Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.

orbital-decay 23d ago
Yeah. I might be wrong but I think they only served direct links for a relatively short time in their history, early on in the 90's and in recent years with ping, which they used to track clicks anyway. At least half of their history they used either the 302 redirects or onmousedown link rewriting (which was terrible). And I'm not even starting on AMP.
deleted 23d ago
[deleted]
jolmg 23d ago
Privacy-wise, it enables them to track the result you click on. Also, though more niche, it would make archived search result pages (e.g. on the Wayback Machine) less useful.
- maxchehab 23d ago
  I think they always knew what link you clicked on (when most of the web has a ga4 tag on the other side of the link)
  - jolmg 23d ago
    You can block Google Analytics with your browser. You can't block needing to ask Google for the URL of the result you want to visit.
    - nprateem 23d ago
      If you're that concerned, use another search engine

skarlso 23d ago

Who the hell uses Google still? Kagi is the way to go! Even though it’s paid it’s worth it.

parisiansam 23d ago

Been using Brave Search now for a while including its Brave AI and aside of sporadic times I never needed Google (albeit Brave Search is slower to Google, you get used to it)

JohnTHaller 23d ago

Note: Article published by Autom.dev which, from a quick read of their homepage, seems like it scrapes Google search results in violation of Google's terms of service and sells those results to customers via an API. That's just my quick read of it, though, so this could be wrong.

googapologist 23d ago
Note: Irrelevant. The reported behavior exists, does it not? Anecdata: I've observed this behavior for several weeks already as a regular user without a Google account and there are countless comments from regular users reporting the same behavior.
- vachina 23d ago
  It’s very relevant because the author has a conflict in interest. They can wax poetic about “open internet” but really their business depend on it.
  - mitxela 23d ago
    Kagi also uses this
- JohnTHaller 23d ago
  It's relevant to know when the source of information is biased due to a conflict of interest. The source's business is apparently scraping Google (and Bing, etc).

Founderarcstone 23d ago

wow thanks for sharing

ChrisArchitect 23d ago

Append (for logged out users) to the title.

rkagerer 23d ago

I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course. I failed to gain traction on the problem, because to me the very idea of doing such a thing was too reprehensible to seriously consider. It broke an unwritten contract between the company and the user's expectation of how websites worked. You expect to be able to do things like right-click a link and copy the authentic URL, or hover to see where it wants to take you. The notion of obfuscating the link beyond easy recognition and polluting it with tracking markers felt misleading and, well, evil. A move that would mainly only benefit Google, and not it's users. I (quite mistakenly) presumed this opinion would be obvious and self-evident to anyone who spent enough time around the early web to understand its norms. I explored other ways of achieving the goal, but it clearly wasn't the answer the interviewer sought. I'm more seasoned now, and experienced enough to say with confidence the approach was wrong. This may seem like a small thing, but a series of misteps and chronic failure to adequately advocate for users is what has led us to the toxic waste dump that so much of the Internet has become today. I'm really glad to have fresh alternatives (like Kagi), and can't wait for the cultural zeitgeist among developers to swing back around to valuing users as human beings and living up to the trust they place in us.

fc417fc802 23d ago
The right click copy thing annoys me to no end.
- seba_dos1 23d ago
  I switched to DuckDuckGo around 2018 or so, which was when I realized that Google's results quality has deteriorated so much that I won't lose anything of value by doing so. This thread is how I learn about the atrocities that Google Search is committing these days. Which is an elaborate way to say: you don't have to live like this.
  - 23d ago

deleted 23d ago

[deleted]

on_the_train 23d ago

I'm confused. Google has done this for several years already. I have a Firefox extension installed that reverts it - it's several years old.

josephcsible 23d ago
They changed how they do it, such that it's no longer possible for extensions to revert it.

dalton74 23d ago

The udm=web parameter has been doing the job for a while now. It's a shame you have to know about an undocumented flag to get the old behavior.

brainwad 23d ago
That flag is the result of clicking on the "Web" tab of the results page, so it kinda is a documented feature. All the tabs have their own udm value (e.g. image search is 2, AI mode is 50, etc.).

1317 23d ago

> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. what? the Location header is the redirect, no?

stncls 23d ago

When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off. I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API. What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?

telotortium 23d ago
I’m guessing that the knowledge of the techniques to support large-scale web search have diffused out of Google - it has been a few decades after all. Not to mention that the distributed system knowledge that used to live only in Google was either published by Google or cloned in other projects, usually made by ex-Googlers. And you can now rent capacity at scales that 20 years ago required Google to build lots of their own data centers. Also, there is now a use case for paid search APIs - LLMs and agents - that essentially didn’t exist a few years ago. Not sure why Google hasn’t leaned more into this, but probably a combination of Gemini and Ads interests have combined to view their search index as an increasingly valuable asset, when if anything it might be corrupted already by these interests and therefore be less valuable.

karel-3d 23d ago

Why can't agents just click on the link and follow it? I don't get it

Almondsetat 23d ago

"google.com/goto considered harmful"

millicentricism 23d ago

Hopefully this also helps against the rampant CTR manipulation that’s made some search results purely a measure of spend.

bomewish 23d ago

I really like the idea of turning the search index into a public utility. It is one of those natural monopoly coordination problem things. Just quasi nationalise it for economic efficiency. Ofc google can still sell adds against their own ui (like everyone else). Hopefully this move will move that idea closer to reality.

dgellow 23d ago
With the current administration? They would manipulate the results to ridiculous extent
- kursus 23d ago
  So would the previous and the next one, this is not situational.

ArcHound 23d ago

It's funny. Recently I looked into what Google is doing (see at https://blog.miloslavhomer.cz/how-google-sees-your-site/ https://blog.miloslavhomer.cz/how-google-sees-your-site/). It's a lot of work to get the data to build an index. Why would they give it to everyone for free?

kolinko 23d ago
Becuse they are a monopolist and we can require certain things from monopolists. Similarly, Bell Labs was kind of required to release transistor for anyone to license - it was a part of social contract that they were allowed to maintain their monopoly in exchange for releasing certain parts of technology. Alternatively, they could be split up and their indexing division made an independent company selling to anyone on a free market.
- ArcHound 23d ago
  I tend to agree - businesses should give back. But they can also keep some secrets. R&D has steep cost. Publishing it all gives your competitors an advantage.
  - kolinko 23d ago
    It has steep costs, but with monopolists they have other ways of extracting value from the inventions. Awesome book about the history if Bell Labs - virtually all semiconductor tech we use today was created there (transistors, ics, solar, lasers, fiber optics, telecom satellites…), and they had to license it to be allowed to maintain their monopoly status. https://www.amazon.pl/Idea-Factory-Great-American-Innovation/dp/1594203288 https://www.amazon.pl/Idea-Factory-Great-American-Innovation...
    - ArcHound 23d ago
      Thanks for the recommendation, I do see your points. We should be treating monopolies differently.

osquar 23d ago

Is it a move to sell more of the paid Google Search API calls? If so, that's a sign of distress. Surely, the cost of serving search results to bots isn't that high

tobinfricke 23d ago

This seems good. I'm not sure why I should be upset that Google is preventing abuse of its service.

kolinko 23d ago
I think it’s the other way around - why should you care about a monopolist’s interests?

grey-area 23d ago

If you don’t like this, just don’t use Google folks. There are alternatives, use them.

Pxtl 23d ago

"There's nothing to explain. You're trying to kidnap what I've rightfully stolen."

nottorp 23d ago

Of course, this move is user hostile because you don't see what you're being sent to.

tantalor 23d ago
You can see the domain printed on the page, right?
- nottorp 23d ago
  I had to check :) I use duckduckgo mostly. They still display the domain, at least, yes. Wonder for how long.

avadodin 23d ago

What are you going to steal from Google? The Internet has been dead for years and, after the Scrapocalypse, the small living remnants are behind a login wall. Google can not provide you anything you couldn't find on either your local ZIM archive or the Media you consume.

shevy-java 23d ago

Google wants to build up a private web here. We already know this from AMP before. Also, the search results are total garbage now. Just give it a try and you see how useless the UI is, with AI results first, then tons of commercial go-to entries and only then a few links that are often also totally useless and irrelevant. Google optimised towards crap. We really need to get rid of Google. Qwant results are now a bit better than before, but still not that great, and I hate that its default UI is a clone of Google. We need more alternatives. Let's get rid of Google once and for all - it has disappointed too many people now.

deleted 23d ago

[deleted]

Hendrikto 23d ago

> The url parameter uses a custom, Google-specific encoding. So that will be figured out, making the whole exercise void. Google knows this. What is the real goal here?

Uzazo 23d ago
Imagine it being the ID of the database row holding the actual URL. There's no way to figure it out without access to that database.
tgsovlerkhgsel 23d ago
"Figuring it out" doesn't help if it's encrypted with a key only Google holds or a reference to some database record that you don't have. In either case, the only way to resolve this is to ask Google, which means they can track it as a click (and rate limit etc.).

mrkramer 23d ago

They are allowed to scrape everybody else but get their feelings hurt when they get scraped....oh yea they respect robots.txt. Guess what, scraping everything that is public is legal.

jasonjmcghee 23d ago

I've never heard of this particular SERP provider, but some marketer is very excited they wrote this blog post right now (500+ upvotes on an seo blog post). And their fix here - they just resolve all the urls- which I suppose could make the service slightly more expensive? But otherwise isn't that what every similar provider/ crawler etc will do and this change will only hurt users?

1vuio0pswjnm7 23d ago

1789154298 | Google's Emissions Climbed 48% Since 2019 due to AI | https://www.gadgetreview.com/googles-emissions-climbed-48-since-2019-ai-is-why https://www.gadgetreview.com/googles-emissions-climbed-48-si... | https://news.ycombinator.com/item?id=49663855 https://news.ycombinator.com/item?id=49663855 | 0 comments

gvieri 23d ago

I had tried now but I don't see the goto. Is it possible that has been deployed only for USA users ? (I work and live in UE). If so: probably a vpn can help you for a while. In the future: I think I'll really go for payed search engine.

pmarreck 23d ago

Is there any market for an open-source internet search engine that is paid for by honest ads?

chews 23d ago

now is time to start the ai to spam/reverse the - google.com/goto for url resolution, it can't be that hard to reverse engineer how the hash works. It was doable for twitter and YouTube, it will be doable here too...

herpdyderp 23d ago

That's it. I'm done with Google search. I just found you can add Kagi to Safari: https://apps.apple.com/us/app/kagi-for-safari/id1622835804 https://apps.apple.com/us/app/kagi-for-safari/id1622835804

lofaszvanitt 23d ago

Divide et impera. People need to learn. Get together, do things together, create alternatives, use that, maintain position, do not let outsider saboteurs neuter the project. Problem solved.

txheilmann 23d ago

[dead]

nijave 23d ago

Hmm that's actually pretty clever. They can serve each result page slightly different encrypted links and it should be obvious right away if it's a SERP bot (trying to grab a page of links) or a human that just picks a few here or there. I wonder if this would also work on other sites getting hammered with bots. Allow each anonymous user 1 "real" page load then turn the rest into encrypted links that the web server can decrypt. If a session cookie with reputation exists, stop screwing with the links. Kind of annoying but it'd allow tracking if the same agent/bot is churning through IPs/User Agents.

hivixop 23d ago

[dead]

pvillano 23d ago

If you were scraping only Google with all the IPs you can get, then this change really slows you down. If you're trying to fight scrapers on a small site, delay links can only flatten bursts. If bots can only scrape at human speed per IP, they can just scrape 100x as many sites at the same time. Once every bot operator does that, total traffic will return to the original level.

thwarted 23d ago

What does this mean? This text appears on this page https://www.autom.dev/blog/google-search-goto-links https://www.autom.dev/blog/google-search-goto-links > The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. And this text appears on this page https://www.autom.dev/blog/google-goto-url-fix https://www.autom.dev/blog/google-goto-url-fix > Do not follow the redirect. Read Location. That's what a redirect is, reading the value of the location header and then requesting it. How do you not follow the redirect by reading the location header? Once you've made the request to the /goto url, with GET or HEAD, to get the location header, google knows you're interested in whatever it is putting in the location header and can assume you're going to go there, if you're letting the User Agent (curl or the browser) go there for you or not.

echoangle 23d ago

> The real URL is in the Location header on /goto. Request that URL. Do not follow the redirect. What does this mean? Isn’t the location header the redirect? Am I not following the redirect by requesting the location header url?

rat_on_the_run 23d ago

Can confirm that when using google not logged in. Now when sharing a link from google search, I won't get the actual link. This will certainly help google's tracking. Google has gone so bad over the past few years. You only get like 8 results per page. I remember there was a time that I wonder how a site get reached if it ranked on the second page, when I can set the number of results to be 50. The censorship is also really bad, and google doesn't even tell you the results are censored, returning totally nonsense results while other search engines work normally. I've been supporting Brave search which returns 20 results per page and has other features. It used to be not good a few years ago, but now the results are often better than google's.

1vuio0pswjnm7 23d ago

"Combined with earlier moves like removing &num=100 and tightening BotGuard/SearchGuard, Google is steadily raising the cost of naive SERP scraping." Another "move" is suing companies like Autom, e.g., SerpApi Google's Amended Complaint from their suit against SerpApi https://ia801008.us.archive.org/25/items/gov.uscourts.cand.461513/gov.uscourts.cand.461513.45.0.pdf https://ia801008.us.archive.org/25/items/gov.uscourts.cand.4... "30. Copyright holders have authorized Google to implement access controls like SearchGuard for the content they license to Google, and in some cases insisted that Google do so. Googles authorization takes many forms. For example, Google has an agreement with a prominent licensing partner that holds copyrights to millions of works that it licenses Google to use in its Search results. Under the parties agreement, versions of which date back to 2017, Google is not only authorized, it is obligated to use commercially reasonable efforts to safeguard the licensed content against unauthorized third-party access. Other license agreements contain similar obligations. For example, another major content provider requires that Google ensure the content it licenses will not be available for download by third parties, thereby authorizing the implementation of technical access controls." "31. In other cases, Googles authorization to implement access control measures like SearchGuard is part and parcel of the grant of licenses themselves, as Google and its licensors recognize that the value of the licensed rights would be undermined if others were free to access, take and resell the licensed content without restriction. For example, Google has a licensing agreement with Reddit, under which Reddit licenses Google to use the copyrighted content of both Reddit and its users in Search Services." "32. Googles licensing partners have also expressly requested that Google prevent unauthorized access to licensed content. For example, when Reddit suspected that scrapers like SerpApi were accessing, taking, and reselling the content that Reddit had licensed to Google, it specifically asked Google to employ technical measures to prevent such unauthorized appropriation." But this does not account for material that is not covered by the "license with a prominent licensing partner", its license with "another major content provider" or its agreement with Reddit Google not only uses SearchGuard on SERPs containing links to the content covered by these licenses, it uses SearchGuard on _all_ SERPs Google needs more than a "goto" update. It needs to update its terms to require _all_ copyright holders for the materials it has indexed and cached to give Google authorisation to use "technological protection measures" to deny access to certain members of the public, e.g., Google's perceived competitors including any Google user who "searches too fast"

ValentineC 22d ago

One really annoying thing that I just found in practice is that this change breaks copying and pasting links directly from the Google search results. Now, I need to click on the link and access the page to “unfurl” the URL so that I can paste a nice, human-parseable URL into a chat. Google may have just made me actively want to switch away from it.

daft_pink 22d ago

so glad i use kagi

NishanStepak 22d ago

I am curious what the difference between scraping and linking. I have a project that links to many different sites, but does not scrape the content. I have been accused of scraping because of the number of links. Some sites have rate limits which limit the number of times you can go to a site from another site. Gutenberg does this. Internet Archive does not have rate limits. My site is not a link farm, however, this can be a pattern of unwanted backlinks in some cases. I am curious how this fits in with Googles Anti-scraping updated. On August 18-20, Google did a massive automated anti-spam update with the algorithm. There was no way to contest what your site was. There is a change being done in how Google is deciding what sites are spam or not. From what I am noticing many sites that with low traffic that are academic or personal were affected by this update.

latcom007 21d ago

[flagged]

bitbasher 21d ago

I've found duckduckgo to be a pretty good replacement these days. A few years ago I found myself using !g often to hop over to google, but these days I've been getting by without any !g ... I'm not sure if it's because duckduckgo has gotten better or google has gotten worse, but either way it's welcome.

javier2 20d ago

Oh, Google does not like others stealing and indexing their content?! How dare they!

nerdyadventurer 16d ago

While they can scrape all whole web, ignoring copyright content. Sometimes Gemini even output images with watermarks from a popular licensed media seller. Mean while Aron Swartz was prosecuted for scraping[1] and finally he end up taking his life. 1: https://blog.curiousquail.com/im-upset-again-about-a-co-creator-of-rss-being-prosecuted-for-something-meta-is-doing-with-little-consequence/ https://blog.curiousquail.com/im-upset-again-about-a-co-crea...