16 ms·
MarkMonitor left 60k domains for the taking
- detaro 5y agoOops. People pay a lot of money to MarkMonitor because their promise is to not make such mistakes.
- toast0 5y agoWhen I was a customer, screwing up a parking page isn't the kind of mistake my employer was paying them not to make. My employer was paying them not to let outside groups change our domain's DNS settings. But that was for domains we were actively using. I don't even remember if we had a parking page on the domains we only had to sit on; if we did, while it wouldn't have been great, it also wouldn't have been a big deal for them to show something attacker controlled for a little while. Parked domains weren't in active use, didn't have links going to them, and probably shouldn't seem credible to others. I think we might have set up the parked domains to just redirect to our main website, but yeah.
- agwa 5y ago> it also wouldn't have been a big deal for them to show something attacker controlled for a little while This is understating the risk. If an attacker obtains control over a website for even a brief period of time, they can go to a certificate authority and obtain a domain validation for the domain (including all subdomains) that's valid for one year. At the end of that year, they can use the domain validation to obtain an SSL certificate that's valid for up to one year, even if they no longer control the website. This means that following a brief takeover, the domain continues to be vulnerable to attack for two years, which would be a problem if a parked domain is transitioned to active use during that time.
- detaro 5y agoNowadays a bit lessened by CT logs, so you (or MM) can at least notice and cause the cert to be revoked.
- agwa 5y agoYes, although the attacker can defer logging the certificate to CT until the moment before the certificate is used in an attack, so you would not have much time to respond.
- detaro 5y agoHow? The CA submits records (precertificates) to the logs before it signs the cert.
- agwa 5y agoPrecertificates are optional. Alternatively, the CA can skip creating a precertificate. The certificate would not contain any embedded SCTs. Instead, the certificate is logged using the add-chain endpoint[1], and the returned SCTs are delivered via an extension to the TLS handshake or an extension in a stapled OCSP response[2] [1] https://datatracker.ietf.org/doc/html/rfc6962#section-4.1 https://datatracker.ietf.org/doc/html/rfc6962#section-4.1 [2] https://datatracker.ietf.org/doc/html/rfc6962#section-3.3 https://datatracker.ietf.org/doc/html/rfc6962#section-3.3
- deleted 5y ago[deleted]
- detaro 5y agothanks, I didn't know both steps (pre-cert and actual cert submission into logs) were optional.
- tialaramex 5y agoLogging is not a policy requirement of the trust stores. It's a condition for your certificates to work in some popular browsers (notably Chrome and Safari) but not a policy requirement. This is on purpose. Google themselves for example will get a certificate issued, say, today, for shiny-new-google-product.example and then only at the last moment do they log that certificate when spinning up the web site https://shiny-new-google-product.example/ https://shiny-new-google-product.example/ to launch shiny-new-google-product -- you can't find out about it in the CT logs because it wasn't in there earlier. Now, the readily available and especially free certificates most people use are logged before issuance as you describe using the poisoned "pre-certificate" feature from RFC 6962 (in theory this will one day be replaced by a 6962bis actual pre-certificate document rather than poisoned X.509 certs) but that is not at all mandatory, it's just convenient because it requires no workflow changes. You get a certificate, it's a bit bigger (it has SCTs baked inside it) and it just works. If you do things the hard way, you must ensure your server application software understands what SCTs are and can send them to clients as necessary. You save some bytes talking to clients that don't want the SCTs, and you get to have this just-in-time logging behaviour if you want that. The other reason it isn't against policy to issue without logging is that some archaic systems that are subject the Web PKI rules but aren't actually talking to the public Internet, and especially to web browsers, do not have logging, it's not a thing for them. This will probably go away in the next few years, these systems age out, but they still existed when I last looked.
- jstanley 5y ago> At the end of that year, they can use the domain validation to obtain an SSL certificate that's valid for up to one year, even if they no longer control the website. Can you explain this further? I don't understand. Are you saying they could MITM the request that the CA makes to the website when it tries to do domain valudation? If that's the case, why is it limited to 2 years? Why couldn't they continue to do this indefinitely?
- agwa 5y agoNo, they don't have to MitM the CA's domain validation request. While they have brief control over the website, they use domain validation method 3.2.2.4.18 (Agreed-Upon Change to Website v2)[1] or 3.2.2.4.19 (Agreed-Upon Change to Website - ACME)[2] to legitimately complete domain validation by making a change to the website. Due to domain validation reuse[3], the certificate doesn't have to be issued right away. The attacker can wait and request the certificate up to 398 days later, without having to do domain validation again. [1] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd5d692685b97ebb6669c74fb7/docs/BR.md#322418-agreed-upon-change-to-website-v2 https://github.com/cabforum/servercert/blob/cda0f92ee70121fd... [2] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd5d692685b97ebb6669c74fb7/docs/BR.md#322419-agreed-upon-change-to-website---acme https://github.com/cabforum/servercert/blob/cda0f92ee70121fd... [3] https://github.com/cabforum/servercert/blob/cda0f92ee70121fd5d692685b97ebb6669c74fb7/docs/BR.md#421-performing-identification-and-authentication-functions https://github.com/cabforum/servercert/blob/cda0f92ee70121fd...
- codetrotter 5y agoSo if I understand the issue correctly, MarkMonitor pointed all of those domains to S3, without first creating S3 buckets for the domains in question?
- selcuka 5y agoYes, they first pointed them to S3, then created the parked domain page for each of them. There was a window where anyone could have claimed the bucket.
- fouc 5y agoAnd ultimately they still own the domains and can always point them to a new IP address at any time. The window for exploitation seems kind of small and temporary.
- IncRnd 5y ago> The window for exploitation seems kind of small and temporary. That depends on whether you are attacking the clients or the server. If an attacker obtained a domain cert or wildcard cert, while in control of the domain, then the attacks can continue.
- TedDoesntTalk 5y agoHow? They now have a cert for that domain but the domain owner can point the domain to any IP address he wants. The article mentions this, but I don’t see the vuln. Can you explain?
- manigandham 5y agoDNS is very easy to spoof and redirect. There are proposals to secure it (DNSSEC, DNS-over-TLS, DNS-over-HTTPS) but none are widely used and instead the server's certificates are used to both authenticate the correct destination and encrypt traffic to it. Redirecting traffic is much easier than generating certificates so a valid cert held by a bad actor can be a serious vulnerability.
- wheaties 5y agoThis should be something added to an AWS Security Hub check, e.g. do you have Route53 pointed to an unregistered domain. Otherwise, I can't find fault with AWS at all. Now I have to review our domains on Monday.
- toomuchtodo 5y agoPlease bug your TAM if you have one about this. I’ve bugged ours. S3 should not serve a bucket as a website without domain verification. In the interim, we’ve built middleware where a bucket serving content can’t be removed until the dns record has been.
- ctvo 5y agoOr use CloudFront with an S3 bucket as backing for this use case, like you'd expect? CloudFront has domain name verification.
- manigandham 5y agoThe point is that without domain verification, it won't stop someone else from registering that bucket (which is what happened with the domains in this article).
- ctvo 5y agoThe point is that S3 isn't a CDN. If you use it as a CDN, it's on you to ensure it'll work for your use case. CloudFront, however, is a CDN, and as expected, has domain verification.
- manigandham 5y agoWhether it's a CDN is irrelevant. This is already a supported use-case for S3 which is why it even has this functionality. It's one of many products that supports serving under custom hostnames and all such products should have domain verification.
- jtsiskin 5y agoI hear about subdomain takeovers with S3 all the time. It seems insane Amazon doesn't require any domain verification.
- tyingq 5y agoWhat do they require? You can just claim any S3 bucket you want that's been 'abandoned'?
- pyentropy 5y agoNo, S3 buckets named after hostnames are served when said hostname is pointing to S3 servers . Bob owns bob.com and sets CNAME bob.com.us-east-1.amazonaws.com. Eve creates the bucket and can now serve any static content she wants.
- dotBen 5y agoWhat does the OP's "S3 detection" mean in this context? I'm not clear how the list of vulnerable domains was collected in this instance - presumably he had to know which domains to create buckets for?
- kolp 5y agoPresumably he's scanning DNS records looking for subdomains (A or CNAME records) pointing to S3 buckets. Script then checks to see if these buckets exist and outputs a list of subdomains where the bucket doesn't exist. At least that's my understanding of his process.
- batch12 5y agoI found a similar issue in a huge, very popular site. Thousands of root domains ripe for the taking. I sent a heads up to the company in question, but never got a reply--
- deleted 5y ago[deleted]
- ttty2 5y agoI think it's the issue with mark monitor and not s3/aws. What if mark monitor would put all parked domains to bob.com.mys3.com service? Me, as mys3 provider I'm at fault? I doubt. What if mark monitor would point to an IP address they don't own? Still not their fault?
- TroisM 5y agoYou appear to be shadow-banned... which means that almost no one can see what your posts because of HN's algo and moderators (only people with show-dead enabled are able to see your posts).
- banana_giraffe 5y ago> While many domains began responding with an S3 404, others began switching from S3 to the parked page. What is interesting is that DNS was not involved — all domains pointed to 93.191.168.52 both before and after the issue. I don't get this. These domains weren't configured with CNAME DNS entries like one normally does. Does this mean MarkMonitor's parked domain name server was trying to load from S3, then falling back to something else if that failed?
- iancarroll 5y agoI suspect what happened (as the author of the post) is that given the IP address is from Akamai, they misconfigured Akamai to naively proxy the traffic towards S3. S3 will just pick a bucket based on the Host header in the request, and it seems like Akamai just proxied that from the client.
- banana_giraffe 5y agoAhh, sure, that'll do it. That's quite a big oops. I'd love to see the panicked messages on their internal slack as they figure out what they did.
- ectopod 5y agoI guess they have an automatic system to do the same thing as your exploit. After they misconfigured Akamai, their system and your system were racing to claim the exposed S3 buckets.
- carlsborg 5y agoThey should be using cloudfront to serve those pages over https.
- toredash 5y agoThis is satire, right?
- carlsborg 5y agoWhy so? Expecting to claim the bucket name for each of their domains is clearly not the way to go. With cloudfront they have can have a single bucket and map upto 100 multiple (or request a higher limit) domains per cloudfront distribution backed by that bucket.
- iJohnDoe 5y agoCan someone explain how a subdomain takeover is done in layman terms?
- tomklein 5y agoSubdomain points to a hosting provider. Hosting doesn’t know who the owner is (yet) and waits for someone to sign up/register. Attacker signs up before the real owner does, is lucky that the hosting provider does not verify ownership, and is able to serve whatever they want on the domain, for example a fake website or fake verification files.
- kuroguro 5y agoOr often the other way around - a subdomain for some legacy feature points to a host. After the feature gets axed the server gets shut down (as it costs money) but the domain is left dangling. Anyone can claim the target subdomain (for CNAME) or IP (for A records) on the same hosting provider if not in use. Apart from fake websites it can also be used to bypass SOP/CORS protection in some contexts.
- gregdoesit 5y ago“ MarkMonitor sells themselves as the domain registrar that does not make mistakes. (…) MarkMonitor is not a cheap solution to this problem, but it is widely used (apparently by "more than half of the Fortune 100", per the page)“ And then: “ MarkMonitor does not have a way of disclosing security issues, which inhibited reporting this to them in a timely manner. They have not responded to any of our communications.” Should anyone be surprised that a company that claims to not make mistakes, but has no way to report vulnerabilities gets their vulnerability on the front page of HN? (And this is a good case scenario, as opposed to their customers being hacked: which could have happened by an attacker claiming some of these domains) Let’s hope they make at least the change of putting a bug bounty program in place.
- CydeWeys 5y ago> This issue is not entirely the fault of MarkMonitor. While they need to be careful with handling parked domains, AWS is at fault for not being more stringent with claiming S3 buckets. Google Cloud, for example, has required domain verification for years, rendering this useless. This sounds like more of an Amazon problem than a MarkMonitor problem to me. And it makes a good case for using other cloud providers over AWS, as with GCP this attack wouldn't have been possible. Merely creating the DNS records pointing to the cloud provider's nameservers shouldn't be enough for anyone to then claim it and start hosting.
- alfiedotwtf 5y agoFrom Amazon's own documentation about this: "An Amazon S3 bucket name is globally unique, and the namespace is shared by all AWS accounts. This means that after a bucket is created, the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted. You should not depend on specific bucket naming conventions for availability or security verification purposes. For bucket naming guidelines, see Bucket naming rules." ... but if you want to host a static website via S3, they explicitly say you need to create a bucket with the domain name as the bucket :shrug:
- gekk05 5y agoSeems like a lot of people are blaming AWS. S3 buckets have a grace period of 24 hours after a bucket is to be deleted in which it cannot be registered by anyone else. We call it internally, "bucket sniping". AWS does not have control of your DNS records. This falls under the typical cloud shared security model. It's purely a DNS management issue. This happens with other technologies too, like mail servers. Domains (unique identifiers like S3 bucks) expire, someone registers it, spins up a mailserver, and begins recovering accounts via password resets. Who is to blame here, the application for allowing the password reset (provided the valid email/fetched the reset key), or the person who let the domain expire? I'd say the latter, and in the s3 situation, I believe it's the same.
- mxpxrocks10 5y agoAnother great post Ian! If any researcher here needs data or help to do investigations like this please reach out to me chris at securitytrails.com - we're trying to hone the tools to be as useful as possible with as little effort.