4 ms·
I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just
by edoceo 3mo ago
I've been working on a site. It's new, domain is only a few weeks old. It's got SSL, so all the bots know it exists. It's never had any sub-pages exposed, just the placeholder lander, no links.
Somehow in Google search one of the unguessable pages is indexed. We have used Claude and Gemini to assist with some design aspects.
I'm thinking some aggressive data ingestion/indexing is happening by all the bots in the quest for frontier models.
- phoghed 3mo agoYou ISP also collects and sells data to companies like Moz, and possibly to Google too.
- soblemprolver 3mo agoURL paths over https wouldn't be transparent to the ISP though, would they?
- throw10920 3mo agoThey would not - GP was probably bringing up something not directly relevant, but still related. (they should have clarified though)
- edoceo 3mo agoWhat can I add? I know the domain could be found. But how did Google index a page that is example.com/$MD5.html - guessing 128bit numbers is hard. I have visited that page from a signed-in Chrome profile.
- throw10920 3mo agoErm sorry when I said GP I meant of my comment - that is, phoghed's irrelevant comment about ISPs collecting data. The information you mentioned is relevant. Unfortunately, it could be either Google/Chrome, or the LLM service you're using for development is misusing your data.
- phoghed 3mo agoThat assumes it was using it from the get go and every time, which might not be a safe assumption.
- resonious 3mo agoI've also seen Google indexing pages with random values in the path that don't get linked to statically (server asks for the URL then redirects to it immediately). I'm pretty sure they index straight out of the Chrome address bar.
- foobarbecue 3mo agoHoly crap I hope that's not true. I've also had unguessable pages indexed, though, and don't have an explanation.
- nicce 3mo agoSomething worth inspecting further. We know that Chrome stores and sends the browsing history but this is an interesting vector.
- DANmode 3mo agoI’d be more surprised if they weren’t capturing this information. Especially if you have autocomplete-while-searching type of features on.
- EGreg 3mo agoWhy don’t they also capture information you enter into forms on Chrome? They control the entire browser surface, technically they can know everything, even TLS and E2E encrypted data, that they silently phone home… If you think this is silly, consider that Microsoft Recall had been observing everything on people’s entire SCREENS and phoning home much of it. That is how a guy was caught recently: https://x.com/t3chfalcon/status/2074134314145489195 https://x.com/t3chfalcon/status/2074134314145489195 And it is actually much worse than even that: https://community.qbix.com/t/increasing-state-of-surveillance-from-all-sides-around-the-world/804 https://community.qbix.com/t/increasing-state-of-surveillanc...
- pixl97 3mo ago
- dreambigwrkhard 3mo agoDepending on the CMS, if it's wordpress (15% chance, ha) there is a sitemap function built-in out of the box. The bots don't need to guess.
- edoceo 3mo agoNo CMS, no sitemap, no robots.txt either.
- cynerx 3mo agoAre you using Cloudflare by any chance? I think the Crawler Hints setting [1] exposed some of my "secret" pages in the past. [1] https://developers.cloudflare.com/cache/advanced-configuration/crawler-hints/ https://developers.cloudflare.com/cache/advanced-configurati...
- edoceo 3mo agoI'm not using cloudflare.
- pohuing 3mo agoThere's a couple avenues besides just stealing what's in your URL bar. If you don't use wildcard certs all of your subdomains can be scraped from the certificate transparency logs. Additionally, any domain+cert using HSTS with preload enabled end up in a big list at Google to speed up the initial connection from browser to site.
- fragmede 3mo agoCT logs just explain how they found the domain. T doesn't explain how they could have found unlinked content on the domain itself. If I put up secret-example.com/asdf-1234567.html, how does that page get found if there are no public links to it?
- Lomlioto 3mo agoDon't underestimate people not knowing were they share stuff by accident. Creating Sitemaps, sharing it somewere public, putting the url in some 3th party service, server logs, some indirect path in javascript. But if you never mention that url, it will not be found if not leaked by your server.
- saghm 3mo ago> But if you never mention that url, it will not be found if not leaked by your server That sounds like a claim that security through obscurity is infallible, which is dubious. Don't get me wrong, it can be a reasonable part of defense-in-depth strategy, but like, brute force attacks are kinda a well known thing, especially if your URLs aren't truly random...
- pohuing 3mo agoTrue. I just assumed imprecise phrasing. Google misusing chrome browser history as a hitlist for indexing sounds wild to me, so I tried to see if there's another way. It also felt unlikely because there's multiple subdomains of mine that aren't indexed, and wildcards+no preload are the only precautions I've made myself for my private sites. This might also be an EU vs rest of World thing, or my stuff isn't interesting enough to index(in retrospect the most likely reason I suppose)
- deleted 3mo ago[deleted]
- f311a 3mo agoGoogle Chrome used to report visited pages back to Google, not sure if this still the case. Also, Google Analytics can see visited pages and Google uses it. Finding domains is easy, everybody uses CTL to find them.
- VladVladikoff 3mo agoGoogle uses data from chrome. If you visited it with chrome, google knows it exists.
- mirekrusin 3mo agoIsn't leaking browser extension used by one of people on the team (doesn't need to be developer, could be qa or anybody with whom the access was shared) more plausible?
- htek 3mo agoNothing you enter into an LLM not hosted by you, or put onto the web is safe from being collected and exploited by these "AI" companies and their LLM's voracious appetite.
- brador 3mo agoChat programs catch links you send. Also that browser setting to check urls are safe sends them out “sometimes“.
- malwrar 3mo agoThey log all DNS requests made to their public resolver in a searchable internal database, at least when I worked there a decade or so ago. I wonder if they seed their crawler with it?
- Analemma_ 3mo agoDNS servers never see subpaths you request, only the domain itself, so that wouldn’t help with a hidden path. But there are lots of other ways to get it: caches/CDNs can leak paths, Chrome presumably sends Google a bunch of request details, and so on. It’s a different story if it’s a subdomain though, OP wasn’t clear.
- dataviz1000 3mo agoThe tin hat guess. Did you include Google analytics embedded in the pages? Do you navigate to pages and Google analytics sends that data home? 10 years ago I discovered that Google analytics would send the equivalent amount as organic users; meaning if we sent an email newsletter with links to articles, Google would send almost 1:1 ratio the same number of people from search results. They are tracking everything and using it for more than just reporting. Do you use a CMS or other tools that auto generate sitemap.xml? Perhaps you unknowingly told Google about those sub-pages.
- edoceo 3mo agoNo GA, no external assets, no Google fonts. I really thought I was being careful.
- stavros 3mo agoIt's indexed some unlisted draft blog posts of mine that were never touched by AI or published anywhere. I use a static site generator so there's no earthly way they ever found the pages by scraping, at most I visited the pages once or twice from my browser.
- animex 3mo agoI've always wondered if Chrome leaks these URLs too.
- NDlurker 3mo agoHmm. Seems like this could be used for an ARG
- DonHopkins 3mo agoI've gotten unguessable hits in the logs because somebody who was authorized to use them had a virus that exfilterated their browser history. Might have been an evil chrome extension, but ever since Google went IOK2BE ("It's OK to be Evil"), maybe it's just Chrome itself.
- yread 3mo agoI have about 50 subdomains. One was used by a colleague who cant do his shoelaces without claude. That subdomain gets 10 times more spam and hacking attempts.