11 ms·
When internal hostnames are leaked to the clown
- ranger_danger 9mo agoPennywise found my hostname? We're cooked.
- TeapotNotKettle 9mo agoMisconfigured clown - bad news indeed.
- defrost 9mo agoYou're IT, I'm IT, We're all IT.
- dcrazy 9mo agoSlightly surprised that this blog seems to have succumbed to inbound traffic.
- that_lurker 9mo agoOpens fine for me
- urbandw311er 9mo ago“Works on my machine”
- unsnap_biceps 9mo agoIf you're on an apple device, disable private relay. It appears the blog has tar pitted private relay traffic.
- bhaney 9mo agoIt's tar pitting my normal unproxied residential traffic too
- computerfriend 9mo agoSame, plus my VPN connection.
- alyandon 9mo agoSame here too. Ironically, the blog is accessible over TOR for me.
- daveoc64 9mo agoRachel has blogged quite a bit about blocking badly behaved RSS Clients in recent years. I'd link you to one of the articles if I wasn't blocked too, and my VPN wasn't also blocked!
- lapcat 9mo ago> Rachel has blogged quite a bit about blocking badly behaved RSS Clients in recent years. Unfortunately that blocking is buggy and overzealous. I just gave up eventually and unsubscribed from the RSS feed.
- stingraycharles 9mo agoI don’t understand. How could a GCP server access the private NAS? I agree the web UI should never be monitored using sentry. I can see why they would want it, but at the very least should be opt in.
- throwaway290 9mo agoIt said knocking, not accessing also > you notice that you've started getting requests coming to your server on the "outside world" with that same hostname.
- minitech 9mo agoIt couldn’t, but it tried.
- copperx 9mo agoA for effort, F for firewall.
- fragmede 9mo agoThis highlights a huge problem with LetsEncrypt and CT logs. Which is that the Internet is a bad place, with bad people looking to take advantage of you. If you use LetsEncrypt for ssl certs (which you should), that hostname gets published to the world, and that server immediately gets pummeled by requests for all sorts of fresh install pages, like wp-admin or phpmyadmin, from attackers.
- Spivak 9mo agoI like only getting *.domain for this reason. No expectation of hiding the domain but if they want to figure out where other things are hosted they'll have to guess.
- ttoinou 9mo agoSo how do you get this ?
- rossy 9mo agoLet's Encrypt can issue wildcard certs too
- hsbauauvhabzb 9mo agoThat’s really not a great fix. If those hostnames leak, they leak forever. I’d be surprised if AV solutions and/or windows aren’t logging these things.
- thakoppno 9mo ago> the Internet is a bad place FWIW - it’s made of people
- TZubiri 9mo agoNo, it's made by systems made by people, systems which might have grown and mutated so many times that the original purpose and ethics might be unrecognizable to the system designers. This can be decades in the case of tech like SMTP, HTTP, JS, but now it can be days in the era of Moltbots and vibecoding.
- NitpickLawyer 9mo agoNot sure why they made the connection to sentry.io and not with CT logs. My first thought was that "*.some-subdomain." got added to the CT logs and someone is scanning *. with well known hosts, of which "nas" would be one. Curious if they have more insights into sentry.io leaking and where does it leak to...
- jraph 9mo agoThat hypothesis seems less likely and more complicated than the sentry one. Scanning wildcards for well-known subdomains seems both quite specific and rather costly for unclear benefits.
- flexagoon 9mo agoBots regularly try to bruteforce domain paths to find things like /wp-admin, bruteforcing subdomains isn't any more complicated
- jraph 9mo ago> Bots regularly try to bruteforce domain paths to find things like /wp-admin Sure, when WordPress powers 45% of all websites, your odds to reach something by hitting /wp-admin are high. The space of all the possible unknown subdomains is way bigger than a few well known paths you can attack.
- rawling 9mo agoI feel like the author would have noticed and said so if she was getting logs for more than just the one host.
- A1kmm 9mo agoBut she mentioned: 1) it isn't in DNS only /etc/hosts and 2) they are making a connection to it. So they'd need to get the IP address to connect to from somewhere as well.
- 9mo ago
- notsylver 9mo agoI think people are misunderstanding. This isn't CT logs, its a wildcard certificate so it wouldn't leak the "nas" part. It's sentry catching client-side traces and calling home with them, and then picking out the hostname from the request that sent them (ie, "nas.nothing-special.whatever.example.com") and trying to poll it for whatever reason, which is going to a separate server that is catching the wildcard domain and being rejected.
- spondyl 9mo agoMy first thought was perhaps they're trying to fetch a favicon for rendering against the traces in the UI?
- hsbauauvhabzb 9mo agoSounds like a great way to get sentry to fire off arbitrary requests to IPs you don’t own. sure hope nobody does that targeting ips (like that blacklist in masscan) that will auto report you to your isp/ans/whatever for your abusive traffic. Repeatedly.
- b1temy 9mo agoIs "clown GCP Host" a technical term I am unaware of, or is the author just voicing their discontent? Seems to me that the problem is the NAS's web interface using sentry for logging/monitoring, and part of what was logged were internal hostnames (which might be named in a way that has sensitive info, e.g, the corp-and-other-corp-merger example they gave. So it wouldn't matter that it's inaccessible in a private network, the name itself is sensitive information.). In that case, I would personally replace the operating system of the NAS with one that is free/open source that I trust and does not phone home. I suppose some form of adblocking ala PiHole or some other DNS configuration that blocks sentry calls would work too, but I would just go with using an operating system I trust.
- jraph 9mo ago> Is "clown GCP Host" a technical term I am unaware of, or is the author just voicing their discontent? Clown is Rachel's word for (Big Tech's) cloud.
- TZubiri 9mo ago>Hope you didn't name it anything sensitive, like "mycorp-and-othercorp-planned-merger-storage", or something. So, no one competent is going to do this, domains are not encrypted by HTTPS, any sensitive info is pushed to the URL Path. I think being controlling of domain names is a sign of a good sysadmin, it's also a bit schizophrenic, but you gotta be a little schizophrenic to be the type of sysadmin that never gets hacked. That said, domains not leaking is one of those "clean sheet" features that you go for no reason at all, and it feels nice, but if you don't get it, it's not consequential at all. It's like driving at exactly 50mph, like having a green streak on github. You are never going to rely on that secrecy if only because some ISP might see that, but it's 100% achievable that no one will start pinging your internal host and start polluting your hosts (if you do domain name filtering). So what I'm saying is, I appreciate this type of effort, but it's a bit dramatic. Definitely uninstall whatever junk leaked your domain though, but it's really nothing.
- jraph 9mo ago> any sensitive info is pushed to the URL Path This too is not ideal. It gets saved in the browser history, and if the url is sent by message (email or IM), the provider may visit it. > Definitely uninstall whatever junk leaked your domain though, but it's really nothing. We are used to the tracking being everywhere but it is scandalous and should be considered as such. Not the subdomain leak part, that's just how Rachel noticed, but the non advertised tracking from an appliance chosen to be connected privately.
- TZubiri 9mo ago>This too is not ideal. It gets saved in the browser history, and if the url is sent by message (email or IM), the provider may visit it. Sure. POST for extra security. > Not the subdomain leak part, that's just how Rachel noticed, but the non advertised tracking from an appliance chosen to be connected privately. If this were a completely local product, like say a USB stick. Sure. but this is a Network Attached Storage product, and the user explicitly chose to use network functions (domains, http), it's not the same category of issue.
- teekert 9mo agoIs this a Chrome/Edge thing? Or do privacy respecting browsers also do this? If so, it's unexpected. If Firefox also leaks this, I wonder if this is something mass-surveillance related. (Judging from the down votes I misunderstood something)
- nomercy400 9mo agoFrom what I understand, sentry.io is like a tracing and logging service, used by many organizations. This helps you (=NAS developer) to centralize logs and trace a request through all your application layers (client->server->db and back), so you can identify performance bottlenecks and measure usage patterns. This is what you can find behind the 'anonymized diagnostics' and 'telemetry' settings you are asked to enable/consent. For a WebUI it is implemented via javascript, which runs on the client's machine and hooks into the clicks, API calls and page content. It then sends statistics and logs back to, in this case, sentry.io. Your browser just sees javascript, so don't blame them. Privacy Badger might block it. It is as nefarious as the developer of the application wants to use it. Normally you would use it to centralize logging, find performance issues, and get a basic idea on what features users actually use, so you can debug more easily. But you can also use it to track users. And don't forget, sentry.io is a cloud solution. If you post it on machines outside your control, expect it to be public. Sentry has a self-hosted solution, btw.
- jeroenhd 9mo agoMy employer uses Sentry for (backend) metrics collection so I had to unblock it to do my job. I wish Sentry would have separate infra for "operating on data collected by Sentry" and "submit every mouse click to Sentry" so I could block their mass surveillance and still do my job, but I suppose that would cut into their profit margins. My current solution is a massive hack that breaks down every now and then.
- wbobeirne 9mo agoMost organizations I've set Sentry up for tunnel the traffic through their own domain, since many blocking extensions block sentry requeats by default. Their own docs recommend it as well. All that to say, it's not trivial to fully block it and you were probably sending telemetry anyway even with the domain blocked.
- that_guy_iain 9mo agoThis is actually an really interesting way to attack a sensitive network. This is a way of allowing to map the internal network of a sensitive network. Getting access is obviously the main challenge but once you're in there you need to know where you go and what to look for. If you've already got that knowledge when planning the attack to gain entry then you've got the upper-hand. So while it kinda seems like "Ok, so they have a hostname they can't access why do I care?". If you're doing high-end security on your system admin level then this is the sort of small nitpicking that it takes to be the best.
- zaptheimpaler 9mo agoOh god this sucks, i've been setting up lots of services on my NAS pointing to my own domains recently. Can't even name the domains on my own damn server with an expectation of privacy now.
- jraph 9mo ago> Can't even name the domains on my own damn server with an expectation of privacy now. You never could. A host name or a domain is bound to leave your box, it's meant to. It takes sending an email with a local email client. (Not saying, the NAS leak still sucks)
- zaptheimpaler 9mo agoI don't know much about email, but how would some random service send an email from my domain if I've never given it any auth tokens?
- jraph 9mo agoIt should not, but it's usual to configure random services to send mails to users, for instance for password resets, or for random notifications. Another thing usually sending mails is cron, but that should only go to the admin(s). Some services might also display the host name somewhere in their UI.
- TheDong 9mo agoYou don't need any auth to send an email from your domain, or in fact from any domain. Just set whatever `From` you want. I've received many emails from `root@localhost` over the years. Admittedly, most residential ISPs block all SMTP traffic, and other email servers are likely to drop it or mark it as spam, but there's no strict requirement for auth.
- prmoustache 9mo ago> Admittedly, most residential ISPs block all SMTP traffic, and other email servers are likely to drop it or mark it as spam, but there's no strict requirement for auth. Source? I've never seen that. Nobody could use their email provider of choice if that was the case.
- ggm 9mo agoReverse address lookup servers routinely see escaped attempts to resolve ULA and rfc1918. If you can tie the resolver to other valid data, you know inside state. Public services see one way (no TCP return flow possible) from almost any source IP. If you can tie that from other corroborated data, the same: you see packets from "inside" all the time. Darknet collection during final /8 run-down captured audio in UDP. Firewalls? ACLs? Pah. Humbug.
- _gmax1 9mo ago"Darknet collection during final /8 run-down captured audio in UDP." Mind elaborating on this? SIP traffic from which year?
- LtdJorge 9mo agoRTP I’d say
- ggm 9mo ago2010/2011 time frame. Google and others helped sink the traffic, all written up at apnic labs. It's how 1.1.1.0/24 got held back from general release.
- advisedwang 9mo agoe.g. https://www.potaroo.net/studies/103-slash8/103-slash8.pdf https://www.potaroo.net/studies/103-slash8/103-slash8.pdf and https://conference.apnic.net/news-archives/2010/network-1/assets/Traffic-in-Network-1.pdf https://conference.apnic.net/news-archives/2010/network-1/as...
- atmosx 9mo agoI bought a SynologyNAS and I have regretted already 3-4 times. Apart from the software made available from the community, there is very little one can do with this thing. Using LE to apply SSL to services? Complicated. Non standard paths, custom distro, everything hidden (you can’t figure out where to place the ssl cert of how to restart the service, etc). Of course you will figure it out if you spent 50 hours… but why? Don’t get me started with the old rsync version, lack of midnight commander and/or other utils. I should have gone with something that runs proper Linux or BSD.
- tetris11 9mo ago(Copied from an earlier comment of mine) There are guides on how to mainline Synology NAS's to run up-to-date debian on them: https://forum.doozan.com/list.php https://forum.doozan.com/list.php
- PunchyHamster 9mo agoYou wanted a server and complain NAS is not just a server.
- Gud 9mo agoMore like, user wanted an open operating system but chose a proprietary one.
- atmosx 9mo agoNAS is the primary function. But yes, I want full linux server that I can decide what to install and which protocol to use to upload and/or download files.
- criddell 9mo agoWhy not just leave the NAS to be a NAS and get a separate server? You're probably better off not trying to overload the NAS to be everything.
- 9mo ago
- lsofzz 9mo ago[flagged]
- jeroenhd 9mo agoFrom the article: > Around this time, you realize that the web interface for this thing has some stuff that phones home, and part of what it does is to send stack traces back to sentry.io. Yep, your browser is calling back to them, and it's telling them the hostname you use for your internal storage box. Then for some reason, they're making a TLS connection back to it, but they don't ever request anything. Curious, right? Unless you actively block all potential trackers (good luck with that one lol), you're not going to prevent leaks if the web UI contains code that actively submits details like hostnames over an encrypted channel. I suppose it's a good thing you only wasted 30 seconds on this.
- lsofzz 9mo ago[flagged]
- bigibas123 9mo agoBlocking dns leaks from the local network will not prevent sentry from sending them to the cloud. Blocking sentry from reaching the cloud (like said in the post) will.
- jraph 9mo agoWow, just skip the "bad post", "took me 30 seconds", "Basic stuff" parts already, especially when you are completely missing the point and don't seem to realize it even after several people point it out. Show some humility. What's more, one doesn't really read Rachel for her potential technical solutions but because one likes her story telling.
- lsofzz 9mo ago[flagged]
- zaptheimpaler 9mo ago
- renewiltord 9mo agoHaha, this obtuse way of speech is such a classic FAANG move. I wonder if it’s because of internal corporate style comms. Patio11 also talks like this. Maybe because Stripe is pretty much a private FAANG.
- ashu1461 9mo agoIsn't the article over emphasising a little bit on leakage of internal urls ? Internal hostnames leaking is real, but in practice it’s just one tiny slice of a much larger problem: names and metadata leak everywhere - logs, traces, code, monitoring tools etc etc.
- reddalo 9mo agoIn other words: never put sensitive information in names and metadata.
- dmichulke 9mo agoOr name them after little bobby tables. Is there some sort of injection that's a legal host name?
- jerf 9mo agoDNS naming rules for non-Unicode are letters, numbers, and hyphens only, and the hyphens can't start or stop the domain. Unicode is implemented on top of that through punycode. It's possible a series of bugs would allow you to punycode some sort of injection character through into something but it would require a chain of faulty software. Not an impossibly long chain of faulty software by any means, but a chain rather than just a single vulnerability. Punycode encoders are supposed leave ASCII characters as ASCII characters, which means ASCII characters illegal in DNS can't be made legal by punycoding them legally. I checked the spec and I don't see anything for a decoder rejecting something that jams one in, but I also can't tell if it's even possible to encode a normal ASCII character; it's a very complicated spec. Things that receive that domain ought to reject it, if it is possible to encode it. And then it still has to end up somewhere vulnerable after that.
- m3047 9mo agoRules are just rules. You can put things in a domain name which don't work as hostnames. Really the only place this is enforced by policy is at the public registrar level. Only place I've run into it at the code level is in a SCADA platform blocking a CNAME record (which followed "legal" hostname rules) pointing to something which didn't. The platform uses jython / python2 as its scripting layer; it's java; it's a special real-time java: plenty of places to look for what goes wrong, I didn't bother. People should know that they should treat the contents of their logs as unsanitized data... right? A decade ago I actually looked at this in the context of a (commercial) passive DNS, and it appeared that most of the stuff which wasn't a "valid" hostname was filtered before it went to the customers.
- mixedbit 9mo agoI have investigated similar situation on Heroku. Heroku assigns a random subdomain suffix for each new app, so URLs of apps are hard to guess and look like this: test-app-28a8490db018.herokuapp.com. I have noticed that as soon as a new Heroku app is created, without making any requests to the app that could leak the URL via a DNS lookup, the app is hit by requests from automatic vulnerability scanning tools. Heroku confirmed that this is due the new app URL being published in certificate authority logs, which are actively monitored by vulnerability scanners.
- nightpool 9mo agoReally? Is that new? My apps use wildcard domains: https://i.postimg.cc/SQ82S0Dp/image.png https://i.postimg.cc/SQ82S0Dp/image.png
- mixedbit 9mo agoThis applies only to Heroku Fir and Cedar apps (apps that run in Heroku Private Spaces). Heroku Common Runtime apps still use shared wildcard certificate and their domains are not discoverable like this.
- adolph 9mo ago> certificate authority logs, which are actively monitored by vulnerability scanners That sounds like a large kick-me sign taped to every new service. Reading how certificate transparency (CT) works leads me to think that there was a missed opportunity to publish hashes to the logs instead of the actual certificate data. That way a browser performing a certificate check can verify in CT, but a spammer can't monitor CT for new domains. https://certificate.transparency.dev/howctworks/ https://certificate.transparency.dev/howctworks/
- wlonkly 9mo agoI think it was more of an intentional tradeoff, as one of the many goals of CT logs was to allow domain owners to discover certificates issued for their domains, or more generally for any interested party to audit the activity of a certificate authority. What you're describing there is certificate... translucency, I guess?
- notpushkin 9mo agohttps://archive.ph/siEdE https://archive.ph/siEdE
- mike-cardwell 9mo agoOnly way I can think of protecting against this is to put a reverse proxy in front of it, like Nginx, and inject CSP headers to prevent cross site requests. Wouldn't block the NAS server side from making external calls, but would prevent your browser doing it for them as is the case here. Also would prevent stuff like Google Analytics if they have it. If you set up a proxy, you could also give it a local hostname like nas.local or something with a cert signed by your private CA that Nginx knows about, and then point the real hostname at Nginx, which has the wildcard cert. Bit of a pain to set this all up though. I run a number of services on my home network and I always stick Nginx in front with a restrictive CSP policy, and then open that policy up as needed. For example, I'm running Home Assistant, and I have the Steam plugin, which I assume is responsible for requests from my browser like for: https://avatars.steamstatic.com/HASH_medium.jpg https://avatars.steamstatic.com/HASH_medium.jpg, which are being blocked by my injected CSP policy P.S. I might decide to let that steam request through so I can see avatars in the UI. I also inject "Referrer-Policy: no-referrer", so if I do decide to do that, at least they wont see my HA hostname in there logs by default.
- dd_xplore 9mo agoNPM is pretty painless
- RamRodification 9mo agoATM machine
- rini17 9mo agoFancy web interfaces are road to hell. Do simplest thing that works. Plain apache or nginx with webdav, basic auth(proven code, minimal attack surface). Maybe firewall with ip_hashlimit on new connections. I have it set to 2/minute and for browser it's actually fine, while moronic bots make new connection for every request. When they improve, there's always fail2ban. That the nas server incl. hostname is public does not bother me then.
- cwillu 9mo agoJust getting 404 not found
- draw_down 9mo ago[dead]
- yabones 9mo agoStuff like this is why I consider uBlock Origin to be the bare minimum security software for going on the web. The amount of 3rd party scripts running on most pages, constantly leaking data to everybody listening, is just mind boggling. It's treating a symptom rather than a disease, but what else can we do?
- behringer 9mo agoI also have taken to using adguard home on the router. It blocks 15 or 20 percent of all my traffic. It's quite scary how bad the tracking and other nasties has become.
- ck2 9mo agothat's actually a great spy trap idea, no? create an impossible internal hostname and watch for it to come back to you you don't even need a real TLD if I am not mistaken, use .ZZZ etc
- happyopossum 9mo ago> you don't even need a real TLD if I am not mistaken, use .ZZZ etc if it's not a real TLD, you won't ever see the dns requests coming to you...
- andix 9mo agoHostnames are not private information. There are too many ways how they get leaked to the outside world. It can be useful to hide a private service behind a URL that isn't easy to guess (less attack surfaces, because a lot of attackers can't find the service). But it needs to be inside the URL path, not the hostname. bad: my-hidden-fileservice-007-abc123.example.com/ good: fileservice.example.com/my-hidden-service-007-abc123/ In the first example the name is leaked with DNS queries, TLS certificates and many other possibilities. In the second example the secret path is only transmitted via HTTPS and doesn't leak as easy.
- amichal 9mo agoMarginally better for sure but in this case the path would also have been "leaked" to the sentry instance owned by developers of the the NAS device phoning home. This can happen in zillions of ways and is a good reason to use relatively opaque urls in generally and not "friendly ids" and generally being careful abou putting secrets in URLs.
- andix 9mo agoJust try it. The first example gets attacked by bots nearly immediately after issuing a TLS cert. The second one usually doesn't get detected at all.
- Kwpolska 9mo agoWhat if you have a wildcard cert for *.example.com?
- andix 9mo agoMuch better. But you still leave traces from dns queries. Subfinder has a lot of sources to find subdomains, not only certs: https://github.com/projectdiscovery/subfinder https://github.com/projectdiscovery/subfinder
- jamesfinlayson 9mo ago
- superkuh 9mo agoI love that this write-up is hosted both on HTTP and HTTPS. I cannot access the HTTPS version but the HTTP display just fine. Now that's reliability.
- DANmode 9mo ago> I cannot access the HTTPS version Curiosity begs: why not?
- superkuh 9mo agoI opened it on an old computer with an old linux distro with an old browser because old linux distros have reliable and working accessibility features like screen readers and good non-gpu text to speech and advanced keyboard/mouse sharing. Modern linux distros do not. Don't worry, I have javascript execution/etc turned off by default on that machine.
- trjordan 9mo agoHaving recently set up sentry, at least one of the ways they use this is to auto-configure uptime monitoring. Once they know what hosts you run, it'll ping that hostname periodically. If it stays up and stable for a couple days, you'll get an alert in product: "Set up uptime monitoring on <hostname>?" Whether you think this is valid, useful, acceptable, etc. is left as an exercise to the reader.
- Linkd 9mo agoExpansion opportunities
- rcakebread 9mo agoTIL Rachel uses a Mac.
- audience_mem 9mo agoHow do you know?
- JSR_FDED 9mo agoLittle Snitch?
- linhns 9mo agoWell somehow Rachel's website is not sending back any response now.
- m3047 9mo agoThis is exactly why I have a number of "appliances" which never get clown updates: have addresses in a subnet I block at the segment edge, have DNS which never answers, and there are a few entries in the "DNS firewall" [0] (RPZ) which mostly serve as canaries. This is the problem with the notion that "in the name of securitah IoT devices should phone home for updates": nobody said "...and map my network in the name of security" [0] Don't confuse this with Rachel's honeypot wildcarding *.nothing-special.whatever.example.com for external use.
- alimoeeny 9mo agoI personally have been blocking sentry and all relevant domains on my machines. I understand this is not a generally applicable advice. For me that’s the right choice
- HocusLocus 9mo agoThe Clown is my master I've been chosen! Eeeeeeeeeah!
- 1vuio0pswjnm7 9mo agoThese requests to the Sentry company used to be sent over HTTP with no encryption https://blog.sentry.io/sentry-ingestion-domains-updates/ https://blog.sentry.io/sentry-ingestion-domains-updates/ https://cloud.google.com/blog/topics/partners/using-sentry-to-monitor-web-and-mobile-apps https://cloud.google.com/blog/topics/partners/using-sentry-t... https://old.reddit.com/r/PleX/comments/1b12phf/plex_sending_requests_to_017675ingestsentryio/ https://old.reddit.com/r/PleX/comments/1b12phf/plex_sending_... There has never been any resource record for any sentry.io domain in the DNS that is used by computers I control. This DNS is local and I control it. I saw a request to an ingest.sentry.io domain once while experimenting with Firefox. It failed The DNS used by me only contains addresses for servers that I find useful But every user has their own preferences. It is possible that some end-users might see value in allowing their computers to automatically send requests to sentry.io while receiving nothing in return. I am not one of those users