11 ms·

Ask HN: How did the internet discover my subdomain?

• codingdave 2y ago
  If it is on DNS, it is discoverable. Even if it were not, the message you pasted says outright that they scan the entire IP space, so they could be hitting your server's IP without having a clue there is a subdomain serving your stuff from it.
  • govideo 2y ago
    Ahh yeah, my internet network knowledge was never super strong, and now is rusty to boot. Thanks for your note.
    • r3db34rd 2y ago
      [dead]
  • EQYV 2y ago
    Question: How does a subdomain get discovered by a member of the public if there are no references to it anywhere online? The only thing I can think of that would let you do that would be a DNS zone transfer request, but those are almost always disallowed from most origin IPs. https://en.m.wikipedia.org/wiki/DNS_zone_transfer https://en.m.wikipedia.org/wiki/DNS_zone_transfer
    • dnsfax 2y ago
      Certificate transparency logs.
    • arccy 2y ago
      you also have zone walking with DNS NSEC https://www.domaintools.com/resources/blog/zone-walking-zone-enumeration-via-dnssec-nsec-records/ https://www.domaintools.com/resources/blog/zone-walking-zone...
    • yatralalala 2y ago
      See my comment above https://news.ycombinator.com/item?id=43289743 https://news.ycombinator.com/item?id=43289743 there are many techniques!
  • paulnpace 2y ago
    Shouldn't the web server only respond to a configred domain, else 404?
    • precommunicator 2y ago
      Depends if it's configured like that, by default usually no
  • dnsfax 2y ago
    If you know what to query, sure. You can't just say "give me all subdomains"; it doesn't work that way. The subdomain was discovered via certificate transparency logs.
  • alexjplant 2y ago
    > If it is on DNS, it is discoverable. In the context of what OP is asking this is not true. DNS zones aren't enumerable - the only way to reliably get the complete contents of the zone is to have the SOA server approve a zone transfer and send the zone file to you. You can ask if a record in that zone exists but as a random user you can't say "hand over all records in this zone". I'd imagine that tools like Cloudflare that need this kind of functionality perform a dictionary search since they get 90% of records when importing a domain but always seem to miss inconspicuously-named ones. > Even if it were not, the message you pasted says outright that they scan the entire IP space, so they could be hitting your server's IP without having a clue there is a subdomain serving your stuff from it. This is likely what's happening. If the bot isn't using SNI or sending a host header then they probably found the server by IP. The fact that there's a heretofore unknown DNS record pointing to it is of no consequence. *EDIT: Or the Cert Transparency log as others have mentioned, though this isn't DNS per se. I learn something new every day :o)
    • fulafel 2y ago
      In practice it's not so far fetched: A zone transfer is just another dns query at the protocol level, i suppose you can conceptually view it as sending a file if you consider the dns response a file. Something like "host -t axfr my.domain ns1.my.domain" will show the zone depending on how a domain's name server is configured (eg in bind, allow-transfer directive can be used to make it public, require ip acl to match the query source, etc).
      • elric 2y ago
        No sensible DNS provider has zone transfers enabled by default. OP mentioned using CloudFlare, and they certainly don't.
      • alexjplant 2y ago
        > in bind, allow-transfer directive Configuring BIND as an authoritative server for a corporate domain when I was a wee lad is how I learned DNS. It was and still is bad practice to allow zone transfers without auth. If memory serves I locked it down between servers via key pairs.
• Kikawala 2y ago
  Is it available under HTTPS? Then it's probably in a Certificate Transparency log.
  • govideo 2y ago
    Yes, https via cloudflare's automatic https. Thanks for the info.
    • thisisgvrt 2y ago
      Automated agents can tail the certificate log to discover new domains as the certs are issued. But if you want to explore subdomains manually, https://crt.sh/ https://crt.sh/ is a nice tool.
    • snailmailman 2y ago
      Yeah this is a surprisingly little known fact- all certs being logged means all subdomain names get logged. Wildcard certs can hide the subdomains, but then your cert works on all subdomains. This could be an issue if the certs get compromised. Usually there isn’t sensitive information in subdomain names, but i suspect it often accidentally leaks information about infrastructure setups. "vaultwarden.example.com" existing tells you someone is probably running a vaultwarden instance, even if it’s not publicly accessible. The same kind of info can leak via dns records too, I think?
      • tialaramex 2y ago
        > The same kind of info can leak via dns records too, I think? That's correct "passive DNS" is sold by many large public DNS providers. They tell you (for a fee) what questions were asked and answered which meet your chosen criteria. So e.g. maybe you're interested, what questions and answers matched A? something.internal.bigcorp.example in February 2025. They won't tell you who asked (IP address, etc.) but they're great for discovering that even though it says 404 for you, bigcorp.famous-brand-hr.example is checked regularly by somebody, probably BigCorp employees who aren't on their VPN - suggesting very strongly that although BigCorp told Famous Brand HR not to list them as a client that is in fact the HR system used by BigCorp.
• daggersandscars 2y ago
  DNS query type AXFR allows for subdomain querying. There are security restrictions around who can do it on what DNS servers. Given the number of places online one can run a subdomain query, I suspect it's mostly a matter of paying the right fees to the right DNS provider.
• artursapek 2y ago
  presumably it has a DNS record
• vince14 2y ago
  I'm having the same issue. https://securitytrails.com/ https://securitytrails.com/ also had my "secret" staging subdomain. I made a catch-all certificate, so the subdomain didn't show up in CT logs. It's still a secret to me how my subdomain ended up in their database.
  • selcuka 2y ago
    They could be purchasing DNS query logs from ISPs.
  • arccy 2y ago
    maybe your server responded to a plain ip addressed request with the real name...
    • fc417fc802 2y ago
      He said he used a wildcard cert though. So what part of the response would contain the subdomain in that case?
    • averageRoyalty 2y ago
      Host header is a request header, not a response one, isn't it?
  • johnklos 2y ago
    Serious question: Do you really think that Cloudflare is trying to keep these kinds of thing private? If so, I'd suggest that's not a reasonable expectation.
    • fc417fc802 2y ago
      Related question (not rhetorical). If you do DNS for subdomains yourself (and just use Cloudflare to point dns.example.com at your box) will the subdomain queries leak and show up in aggregate datasets? What I'm asking is if query recursion is always handled locally or if any of the reasonably common software stacks resolve it remotely.
• parliament32 2y ago
  Certificate Transparency logs, or they don't actually know the domain name: just port-scanning[1] then making requests to open web ports. [1] Turns out you can port-scan the entire internet in under 5 minutes: https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan
  • andix 2y ago
    Port scanning usually can't discover subdomains. Most servers don't expose the of the domains they server content for. In case of HTTP they usually only serve the subdomain content if the Host: request-header includes it.
    • hombre_fatal 2y ago
      Most servers just listen on :80 and respond to all requests. Almost nobody checks the host header intentionally, it's just a happy mistake if they use a reverse proxy. You can often decloak servers behind Cloudflare because of this. But OP's post already answered their question: someone scanned ipv4 space. And what they mean is that a server they point to via DNS is receiving requests, but DNS is a red herring.
      • andix 2y ago
        This really depends on the setup. Most web servers host multiple virtual hosts. IP addresses are expensive. If you're deploying a service behind a reverse proxy, it either must be only accessible from the reverse proxy via an internal network, or check the IP address of the reverse proxy. It absolutely must not trust X-Forwarded-For: headers from random IPs.
        • hombre_fatal 2y ago
          I just don't see how any of this matters. OP's server is reachable via ipv4 and someone sent an http request to it. Their post even says that this is the case.
• andix 2y ago
  I'm surprised nobody mentioned subfinder yet: https://github.com/projectdiscovery/subfinder https://github.com/projectdiscovery/subfinder Subfinder uses different public and private sources to discover subdomains. Certificate Transparency logs are a great source, but it also has some other options.
• fsckboy 2y ago
  LPT, this is an object lesson in the weakness of security through obscurity
  • bangaladore 2y ago
    I mean you could argue that this is more of a multi-factor authentication lesson. Just knowing 1 "secret"— a subdomain in this case —shouldn't get you somewhere you shouldn't. In general you should always assume that any password has been (or could be) compromised. So in this case, more factors should be involved such as IP restricting for access, an additional login page, certificate validation, something...
  • andix 2y ago
    Security by obscurity can be a great additional measure for an already secure system. It can reduce attack surface, make it less likely to get attacked in the first place. In some cases (like this one) it can also be much easier to break than expected.
• OuterVale 2y ago
  https://www.merklemap.com https://www.merklemap.com pops to mind.
  • govideo 2y ago
    Interesting! Just checked them out. "MerkleMap gathers its information by continuously monitoring and live tailing Certificate Transparency (CT) logs, which are operated by organizations like Google, Cloudflare, and Let's Encrypt. "
    • Eikon 2y ago
      I made this, thank you!
• 8bitchemistry 2y ago
  Did you ever email the URL to somebody? We had the same issue years ago where google seemed to be crawling/indexing new subdomains it finds in emails.
  • govideo 2y ago
    Nope, never emailed or posted to anyone. Just me (it's my solo project at the moment).
• deleted 2y ago
  [deleted]
• spl757 2y ago
  Does the IP address for that subdomain have a DNS PTR record set? If it does, someone can discover the subdomain by querying the PTR record for the IP.
  • govideo 2y ago
    If it does, I did not set it up; it would have been automatically done by CloudFlare when I told it to use my custom subdomain for the upload urls.
• deleted 2y ago
  [deleted]
• andix 2y ago
  If a HTTPS service should be hard to discover, an easy way is to hide it behind a subdirectory. Something like https://subdomain.domain.example/hard_to_find_secret_string https://subdomain.domain.example/hard_to_find_secret_string. Another option are wildcard certificates. This obviously can't be the only protection. But if an attacker doesn't know about a service, or misses it during discovery, they can't attack it.
• LinuxBender 2y ago
  As others have said, likely cert transparency logs. Use a wildcard cert to avoid this. They are free using LetsEncrypt and possibly a couple other ACME providers. I have loads of wildcard certs. Bots will try guessing names but like you I do not use easily guessable names and the bots never find them. I log all DNS answers. I assume cloudflare supports strict-SNI but no idea if they have their own automation around wildcard certs. Sometimes I renew wildcard certs I am not even using just to give the bots something to do.
  • govideo 2y ago
    I have been just relying on CloudFlare's automatic https. But I will look into my own certs, though will likely just use CloudFlare's. I don't mind the internet knowing the subdomain I posted about; was curious how the bots found it!
• pabs3 2y ago
  ArchiveTeam has some docs about this: https://wiki.archiveteam.org/index.php/Finding_subdomains https://wiki.archiveteam.org/index.php/Finding_subdomains
  • govideo 2y ago
    I'm so often amazed (but no longer surprised) at the depth of niche (relatively) info and tools out there.
• ciaovietnam 2y ago
  There is a chance that your subdomain is the first/default virtual host in your web server setup (or the subdomain's access log is the default log file) so any requests to the server's IP address get logged to this virtual host. That means they didn't access your subdomain, they accessed via your server IP address but got logged in your subdomain's access log.
  • BrandoElFollito 2y ago
    And this is the correct answer, thank you. Transparency logs are fine except if you have a wildcard cert (or no https, obviously). IP scans are just this: scans for live ports. If you do not provide a host header in your call you get whatever the default response was set up. This can be a default site, a 404 or anything else.
• alberth 2y ago
  This site will find any subdomain, for any domain, so long as it previously had a certificate (ssl/tls) https://crt.sh/ https://crt.sh/
  • averageRoyalty 2y ago
    This is incorrect (or at least only technically correct). This is only true for subdomains with public, trusted CA signed certificates since certificate transparency has existed and only for subdomains with a specific, non wildcard certificate.
  • govideo 2y ago
    Thanks for mentioning. I checked it out, and am learning lots of new stuff (ie, realize how much I do not know).
  • nvarsj 2y ago
    Doesn’t find any of my semi secret subdomains.
  • socrateslee 2y ago
    https://crt.sh https://crt.sh can find your subdomain only when it doesn't have a wildcard certificate(*.somedomain.com)
• thedougd 2y ago
  Some CAs (Amazon) allow not publishing to the Certificate Transparency Log. But if you do this, browsers will block the connection by default. Chromium browsers have a policy option to skip this check for selected URLs. See: CertificateTransparencyEnforcementDisabledForURLs. Some may find this more desirable than wildcard certificates and their drawbacks.
  • klntsky 2y ago
    > Some may find this more desirable Why?
    • thedougd 2y ago
      A CISA article on wildcard security risks. Some of this is in part from common misimplementations (e.g.reusing private keys across servers), but not all of it. https://www.cisa.gov/news-events/alerts/2021/10/08/nsa-releases-guidance-avoiding-dangers-wildcard-tls-certificates-and https://www.cisa.gov/news-events/alerts/2021/10/08/nsa-relea... Direct: https://media.defense.gov/2021/Oct/07/2002869955/-1/-1/0/CSI_AVOID%20DANGERS%20OF%20WILDCARD%20TLS%20CERTIFICATES%20AND%20THE%20ALPACA%20TECHNIQUE_211007.PDF https://media.defense.gov/2021/Oct/07/2002869955/-1/-1/0/CSI...
  • navigate8310 2y ago
    To avoid subdomain discovery, I usually acquire certificate domain level and add a wildcard SAN.
    • luv2code 2y ago
      Will you send me an invite to tildes?
  • snailmailman 2y ago
    Firefox is currently rolling out the same thing. They will treat any non-publicly-logged certificate as insecure. I’m surprised amazon offers the option to not log certificates. The whole idea is that every issued cert should get logged. That way, fraudulently-issued certs are either well documented in public logs- or at least not trusted by the browser.
• rempargo 2y ago
  I assume you host this with a https certificate, so you can look your subdomains at: https://crt.sh/?q=sampledomain.com https://crt.sh/?q=sampledomain.com
• melson 2y ago
  Someone might used open-source tool like sublist3r
  • rawbytes 2y ago
    oh yes that for sure
  • ackbar03 2y ago
    yea was gonna mention this as well lol
• deleted 2y ago
  [deleted]
• paxys 2y ago
  Not sure why everyone is going on about certificate transparency logs when the answer is right there in the user agent. The company is scanning the ipv4 space and came upon your IP and port.
  • deleted 2y ago
    [deleted]
  • pkulak 2y ago
    Okay. But how did they get the proper host header?
    • peeters 2y ago
      There are a couple easy possibilities depending on server config. 1. Not using SNI, and all https requests just respond with the same cert. (Example, go to https://209.216.230.207/ https://209.216.230.207/ and you'll get a certificate error. Go to the cert details and you'll see the common name is news.ycombinator.com). 2. http upgrades to https with a redirect to the hostname, not IP address. (Example, go to http://209.216.230.207/ http://209.216.230.207/ and you get a 301 redirect to https://news.ycombinator.com https://news.ycombinator.com)
    • jimnotgym 2y ago
      I don't think op said that they had the correct host header?
    • INTPenis 2y ago
      Could be a number of ways for example a default TLS cert, or a default vhost redirect. I actually had a job once a few years ago where I was asked to hide a web service from crawlers and so I did some of these things to ensure no info leaked about the real vhost.
    • paxys 2y ago
      Who says they did?
  • peeters 2y ago
• deleted 2y ago
  [deleted]
• DeborahMatthews 2y ago
  [dead]
• arkfil 2y ago
  paloAlto (network devices like firewalls etc) is able to scan the sites that users want to visit behind their devices. these are very popular devices in many companies. users can also have agents installed on their computers that also have access to the sites they visit.
  • opello 2y ago
    This is what I was thinking it must be, along the lines of Cisco NAC. Could monitor via browser plugin for full URLs or DNS server for domains. I imagine the certificate transparency log is the avenue, but local monitoring and reporting up as a new URL or domain to scan for malware seems similarly plausible.
• govideo 2y ago
  Thanks for everyone's perspectives. Very educational and admittedly lots outside the boundaries of my current knowledge. I have thus far relied on CloudFlare's automatic https and simple instant subdomain setup for their worker microservice I'm using. There are evidently technical/footprint implications of that convenience. Fortunately, I'm not really concerned with the subdomain being publicly known; was more curious how it become publicly known.
  • groestl 2y ago
    I had to scroll pretty far down to see the first comment refering to the second most likely leak (after certificate transparency lists): Some ISP sold their DNS query log, and your's was in it. People buying such records do so for various reasons, for example to seed some crawler they've built.
• bashwizard 2y ago
  Like people have said already; Certificate Transparency logs. There are countless of tools to use for subdomain enumeration. I personally use subfinder or amass when doing recon on bug bounty targets.
• 3oil3 2y ago
  What happens if you google your subdomain? Maybe the bots have some sort of dictionary files and they just run them, and when there is a match, then they append it with some .html extension, or maybe they prepend it to the match as a subdomain of it?
• f4c39012 2y ago
  CSP headers can leak urls, but I assume that isn't the cause here if the subdomain is an entirely separate project
• ThePowerOfFuet 2y ago
  Others are saying CT logs but my own subdomains are on wildcard certificates, in which case I suspect they are discovered by DPI analysis of DNS traffic and resold, such as by Team Cymru.
• BLKNSLVR 2y ago
  There are a number of companies, not just Palo Alto Networks, that perform various different scales of scans of the entire IPv4 space, some of them perform these scans multiple times per day. I setup a set of scripts to log all "uninvited activity" to a couple of my systems, from which I discovered a whole bunch of these scanner "security" companies. Personally, I treat them all as malicious. There are also services that track Newly Registered Domains (NRDs). Tangentially: NRD lists are useful for DNS block lists since a large number of NRDs are used for short term scam sites. My little, very amateur, project to block them can be found here: https://github.com/UninvitedActivity/UninvitedActivity https://github.com/UninvitedActivity/UninvitedActivity Edited to add: Direct link to the list of scanner IP addresses (although hasn't been updated in 8 months - crikey, I've been busy longer than I thought): https://github.com/UninvitedActivity/UninvitedActivity/blob/main/02_InternetScanners/02_ZZInternetScannersSingleList.txt https://github.com/UninvitedActivity/UninvitedActivity/blob/...
  • mr_mitm 2y ago
    Getting the domain name from the IP address is not trivial, though. In fact, it should be impossible, if the name really hasn't been published (barring guessing attempts), so OP's question stands.
    • venj 2y ago
      I had this issue with internal domains indexed by Google. The domains where not published anywhere by my company. They were dcanned by leakix.net which apparently scans the whole web for vulnerabilities and publishes web pages containing the domain names associated with each IP address. I guess they read them from the certificates
      • jhart99 2y ago
        There is another source, SNI certs showing up on a server or load balancer during the TLS handshake. When the client tries to connect to a server using SNI without indicating the server, some will reply with a default or give a list of valid server names.
• lockhead 2y ago
  Most likely passive DNS data, if you use your subdomain you do DNS queries for it. If you use a DNS server to resolve your domains that shares this data, it can be picked up by others.
• nusl 2y ago
  It's pretty common to bruteforce subdomains of a domain you might be interested in, specially by attackers.
• xg15 2y ago
  TIL (from this thread) : You can abuse TLS handshakes to effectively reverse-DNS an IP address without ever talking to a DNS server! Is this built into dig yet? :) (Alright, some IP addresses, not all of them) I also wonder if this is a potential footgun for eSNI deployments: If you add eSNI support to a server, you must remember to also make regular SNI mandatory - otherwise, an eavesdropper can just ask your server nicely for the domain that the eSNI encryption was trying to hide from it.
  • yatralalala 2y ago
    Lifehack - it's especially awesome in cases where server operator is using self-signed certs / private cert authorities. Because you will not find these in public cert logs.
• _trampeltier 2y ago
  Did you send a link over Email, Whatsapp or something like?
• ralferoo 2y ago
  If you're using HTTPS, then you're probably using letsencrypt and so your subdomain will appear on the CT logs that are publicly accessible. One thing you could do is use a wildcard certificate, and then use a non-obvious subdomain from that. I actually have something similar - in my set up, all my web-traffic goes to haproxy frontends which forward traffic to the appropriate backend, and I was sick of setting up multiple new certificates for each new subdomain, so I just replaced them all with a single wildcard cert instead. This means that I'm not advertising each new subdomain on the CT list, and even though they all look nominally the same when visiting - same holding page on index and same /api handling, just one of the subdomains decodes an additional URL path that provides access to status monitoring. Separately, that Palo Alto Networks company is a real pain. They connect to absolutely everything in their attempts to spam the internet. Frankly, I'm sick of even my mail servers being bombarded with HTTP requests on port 25 and the resultant log spam.
• deleted 2y ago
  [deleted]
• keysshop 2y ago
  [dead]
• keysshop 2y ago
  [dead]
• clvx 2y ago
  Put it behind ipv6 and it won’t likely happen again. The address space is massive
• deleted 2y ago
  [deleted]
• supermatt 2y ago
  1) Are you sure that they are using the subdomain? They could be connecting via IP or an alternate host address. 2) Are you using TLS? Unless you are using a wildcard cert, then the FQDN will have been published as part of the certificate transparency logs.
• mightybyte 2y ago
  If you've made any kind of DNS entries involving this subdomain, then congratulations, you've notified the world of its existence. There are tools out there that leverage this information and let you get all the subdomains for a domain. Here's the first one I found in a quick search: https://pentest-tools.com/information-gathering/find-subdomains-of-domain https://pentest-tools.com/information-gathering/find-subdoma...
• yatralalala 2y ago
  Hi, our company does this basically "as-a-service". The options how to find it are basically limitless. Best source is probably Certificate Transparency project as others suggested. But it does not end there, some other things that we do are things like internet crawl, domain bruteforcing on wildcard dns, dangling vhosts identification, default certs on servers (connect to IP on 443 and get default cert) and many others. Security by obscurity does not work. You can not rely on "people won't find it". Once it's online, everyone can find it. No matter how you hide it.
  • TZubiri 2y ago
    "Security by obscurity does not work" This is one of those false voyeur OS internet tennets designed to get people to publish their stuff. Obscurity is a fine strategy, if you don't post your source that's good. If you post your source, that's a risk. The fact that you can't rely on that security measure is just a basic security tennet that applies to everything: don't rely on a single security measure, use redundant barriers. Truth is we don't know how the subdomain got leaked. Subdomains can be passwords and a well crafted subdomain should not leak, if it leaks there is a reason.
    • zevlag 2y ago
      > Subdomains can be passwords and a well crafted subdomain should not leak, I disagree. A subdomain is not secret in any way. There are many ways in which it is transmitted unencrypted. A couple: - DNS resolution, multiple resolvers and authoritative servers - TLS SNI - HTTP Host Header There are many middle boxes that could perform safety checks on behalf of the client, and drop it into a list to be rescanned. - Virus Scanners - Firewalls - Proxies
      • dharmab 2y ago
        I once worked for a company which was using a subdomain of an internal development domain to do some completely internal security research on our own products. The entire domain got flagged in Safe Browsing despite never being exposed to the outside world. We think Chrome's telemetry flagged it, and since it was technically routable as a public IP (all public traffic on that IP was blackholed), Chrome thought it was a public website.
• AtNightWeCode 2y ago
  Assuming this is not direct traffic to your IP people will say it is because of TLS logs. Maybe it is in your case. But if you spin up a CF worker on a subdomain to it you will also get hit by traffic immediately. And those certificates are wildcards. I think CF leaks subdomains in some cases. Never seen this behavior when using CF just as a DNS server though.
• jcalx 2y ago
  Some bots scan using giant lists of subdomains, e.g. https://github.com/danielmiessler/SecLists/tree/master/Discovery/DNS https://github.com/danielmiessler/SecLists/tree/master/Disco.... Your subdomain may be on that giant combined_subdomains list, or perhaps some other lists that other tools use.
• TZubiri 2y ago
  Maybe it's a cloudflare controlled scanner? Maybe you published the subdomain in a cert? Snooped traffic is unlikely. This is a good question, if you don't publish a subdomain, scanners should not reach it. If they do, there's a leak in your infra.
• CGamesPlay 2y ago
  Be careful with these. I had a subdomain like this (completely unlisted) with a Google OAuth flow on it, using a development mode Google app. Somehow, the domain was discovered, and Google decided that using their OAuth flow was a phishing scam, and delisted my entire toplevel domain as a result!
  • yoavm 2y ago
    What do you mean "careful with these"? With subdomains?
    • CGamesPlay 2y ago
      Yes, unlisted subdomains. I updated my post to be clearer.
      • joshstrange 2y ago
        I must be missing something. What does “unlisted” mean in this context? I have plenty of subdomains I don’t “advertise” (tell people about online) but “unlisted” is a weird thing to call those. Also I don’t see how it would matter at all when it comes to Google auth. My guess is they blocked it based on the subdomain name itself. I made a “steamgames” subdomain to list stream games I have extra copies of (from bundles) for friends to grab for free. Less than a day after I put it up I started getting chrome scare pages. I switched it to “games” and there have been no issues.
• fsflover 2y ago
  Could it be that Chrome shared the web page with advertisers? https://www.ghacks.net/2021/03/16/wonder-about-the-data-google-collects-in-chrome-and-links-to-you-now-we-know/ https://www.ghacks.net/2021/03/16/wonder-about-the-data-goog...
• perching_aix 2y ago
  Using the Certificate Transparency logs I'd imagine. Also note that your domains are live as they're allocated (they exist). Whether a web server or anything else actually backs them is a different question entirely. For "secret" subdomains, you'll want a wildcard certificate. That way only that will show on the CT logs. Note that if you serve over IPv4, the underlying host will be eventually discovered anyways by brute-force host enumeration, and the domain can still be discovered using dictionary attacks / enumeration. Never touched Cloudflare so this is as far as I can help you.
• curtisszmania 2y ago
  [dead]
• immibis 2y ago
  Additionally to what other people said, you can assume Cloudflare is selling lists of DNS names to someone.
• b112 2y ago
  If you ever email a link and it hits gmail, Google will index it.
• whalesalad 2y ago
  ICANN zone files - https://www.icann.org/resources/pages/czds-2014-03-03-en https://www.icann.org/resources/pages/czds-2014-03-03-en
• zeagle 2y ago
  Can I ask an adjacent question? I have a bunh of DNS A name entries for locallyaccessedservice.mydomain.tld point to my 10.0.0.x NAS's nginx reverse proxy so I can use HTTPS and DNS to access them locally and via Tailscale. My cert is for *.domain.tld. It's nothing critical and only accessible within my LAN, but is there any reason I shouldn't be doing this from a security point of view? I guess someone could phish that to another globally accessible server if DNS changed and I wouldn't notice but I don't see how that would be an issue. There are a couple nginx services exposed to public but not those specific domains so I guess that is an attack vector since.
  • yatralalala 2y ago
    As always, depends on your threat model. Generally having private IPs in public DNS is not great, because potential attacker gets "a general idea" how your private net looks like. But I'd say there's no issue if everything else is secured properly.
    • zeagle 2y ago
      Great thank you. I've mulled around running separate reverse proxies for public and internal services instead.
• Gabrys1 2y ago
  > Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple So my guess is reverse DNS
• itscrush 2y ago
  > I am using CloudFlare for my DNS. Based on this it sounds like you exposed your resource and advertised it for others. Reverse dns, get IP, scan IP. Probably simpler, you exposed resource on IPV4 publicly, if it exists, it'll be scanned. There's probably 100s of companies scanning entire 0.0.0.0/0 space at all times.
• eat 2y ago
  DNS enumeration (brute force) with a good wordlist, zone transfer, or leaking the name through a certificate served when accessing your host via IP address are all possibilities. The name "userfileupload" is far from not-obvious, so that would be my guess.
• aspbee555 2y ago
  cloudflare uses certificates with numerous other site names included on the certificate as alt names so your site name could have been discovered by any other site that happens to use that same cert
• 1vuio0pswjnm7 2y ago
  Why not experiment with multiple variations. For example, as part of the experiment, run own DNS, use non-standard DNS encryption like CurveDNS, or even no DNS at all, use non-standard port for HTTPS, self-signed CA, TLS with no SNI extension, or even TCPCurve instead of CAs and TLS. If non-discoverability is the goal, there are inifinite ways to deviate from web developer norms. If "the internet fails to find the subdomain" when using non-standard practices and conventions then perhaps "following the internet's recommendations", e.g., use Cloudflare, etc., might be partially at cause for discoverability. Would be surprised if Expanse scans more than a relatively small selection of common ports.
• codazoda 2y ago
  This discussion makes me wonder, how hard is it to find a Google Document that was shared with "Anyone with the link"?
• oliwarner 2y ago
  Certificate Transparency would also be my guess. These are logs published by big TLS certificate issuers to cross-check and make sure they're not issuing certificates for domains they have no standing on. The way around this is to issue a wildcard for your root domain and use that. Your main domain is discoverable but your subs aren't. There are other routes: leaky extensions, leaky DNS servers, bad internet security system utilities that phone home about traffic. Who knows? Unless your IP address redirects to your subdomain —not unheard of— it's not somebody IP/port scanning. Webservers don't typically leak anything about the domains they serve for.
• chenmu 2y ago
  [dead]
• zhongjiayu 2y ago
  [dead]
• MacGyver101 2y ago
  Let me list some of the ways that precious subdomain could have been leaked 1) CZDS/DNS record sharing program 2) CT Logs 3) Browser SCT audit 4) Browser telemetry 5) DNS logs 6) DPI 7) Antivirus/OS telemetry 8) Virus/Malware/Tracker 9) Brute forcing DNS records 10) DNSSEC 11) Server softwares with AutoTLS 12) Servers screaming their hostnames over any protocol/banner thing 13) Typing anything on the browser search bar 14) Posting it anywhere And many other novel ways I can't think of right now. I have successfully hidden some of my subdomains in the past but it definitely requires dedication. Simple silly mistakes can make all your efforts go waste. Ask any red/blue teamer. Want to hide something? Roll everything on your own.
• Saris 2y ago
  >I am using CloudFlare for my DNS. Could have been discovered from the SSL cert request for the subdomain.
• lr3783285 2y ago
  [dead]
• pagealert 2y ago
  The discovery of your unpublished subdomain by bots likely stems from a combination of technical factors related to DNS, server configuration, and bot behavior. Here's a breakdown of the possible reasons and solutions: 1. DNS Leaks or Wildcard Records Wildcard DNS Entries: If your main domain (sampledomain.com) has a wildcard DNS record (e.g., .sampledomain.com), any subdomain (including userfileupload.sampledomain.com) could be automatically resolved to your server’s IP. Even if the main domain is inactive, the wildcard might expose the subdomain. Exposed Subdomain DNS Records: If the subdomain’s DNS records (e.g., A/CNAME records) are explicitly configured but not removed, bots could reverse-engineer them via DNS queries or IP scans. Fix: Remove or restrict wildcard DNS entries and delete unused subdomain records from your DNS provider (e.g., Cloudflare). 2. Server IP Scanning IP-Based Discovery: Bots like Expanse systematically scan IP addresses to identify active services. If your subdomain’s server is listening on ports 80/443 (HTTP/HTTPS), bots may: Perform a port scan to detect open ports. Attempt common subdomains (e.g., userfileupload, upload, media) on the detected IP to guess valid domains. Fix: Block unnecessary ports (e.g., close port 80/443 if unused). Use a firewall (e.g., ufw or Cloudflare Firewall Rules) to reject requests from suspicious IPs. 3. Cloudflare’s Default Behavior Page Rules or Workers: If the subdomain is configured with Cloudflare Workers, default error pages, or caching rules, it might generate responses that bots can crawl. For example: A 404 Not Found page with a custom message could be indexed by search engines. Worker scripts might inadvertently expose endpoints (e.g., /_worker.js). Fix: Delete unused subdomains from Cloudflare’s DNS settings. Ensure Workers/routes are only enabled for intended domains. 4. Reverse DNS Lookup IP-to-Domain Mapping: If your server’s IP address is shared or part of a broader range, bots might reverse-resolve the IP to discover associated domains (e.g., via dig -x <IP>). Fix: Use a dedicated IP address for sensitive subdomains. Contact your ISP to request removal from public IP databases. 5. Authentication Flaws Presigned URLs in Error Messages: If the subdomain’s server returns detailed error messages (e.g., 403 Forbidden) when accessed without authentication, bots might parse these messages to infer valid endpoints or credentials. Fix: Customize error pages to show generic messages (e.g., "Access Denied"). Log and block IPs attempting brute-force access. How to Prevent Future Discoveries Remove Unused DNS Records: Delete the subdomain from Cloudflare’s DNS settings entirely. Disable Wildcards: Avoid .sampledomain.com wildcards to limit exposure. Firewall Rules: Block IPs from scanners (e.g., Palo Alto Networks, Expanse) using Cloudflare’s DDoS Protection or a firewall. Monitor Logs: Use tools like grep or Cloudflare logs to track access patterns and block suspicious IPs. Use Authentication: Require API keys, tokens, or OAuth for all subdomain requests. Example Workflow for Debugging bash # Check Cloudflare DNS records for the subdomain: dig userfileupload.sampledomain.com +trace # Inspect server logs for recent requests: grep -E "^ERROR|DENY" /var/log/nginx/access.log # Block Expanse IPs via Cloudflare Firewall: # 1. Go to Cloudflare > Firewall > Tools. # 2. Add a custom rule to block IPs (e.g., from scaninfo@paloaltonetworks.com). By tightening DNS, server, and firewall configurations, you can minimize exposure of your internal subdomains to bots.
• pagealert 2y ago
  By tightening DNS, server, and firewall configurations, you can minimize exposure of your internal subdomains to bots.
• Kipa1234 2y ago
  [dead]
• webpagealert 2y ago
  DNS Leaks or Public Records DNS Propagation: When you create a subdomain (e.g., blog.yoursite.com), your DNS provider (e.g., Cloudflare, GoDaddy) updates global DNS servers. These records are public and visible to anyone who queries the DNS (e.g., via dig blog.yoursite.com). WHOIS Data: If your domain registration details are public (not privacy-protected), your subdomain’s ownership info may be exposed.
• fmxsh 2y ago
  One way is query your ip, retrieve certificates, read the domain(s) in it. I made my server return self signed certificate without domain name in it, in case of access to wevserver's port 433 by ip instead of by domain name.
• summizer 2y ago
  [flagged]