4 ms·
This is speculation, but I think reasonable: - Houseparty spins virtual servers up and down quickly to respond to changing demand. - They have a DNS TTL that'
by kstrauser 7y ago
This is speculation, but I think reasonable:
- Houseparty spins virtual servers up and down quickly to respond to changing demand.
- They have a DNS TTL that's longer than the draining time for those VMs. Like, maybe they it takes them five minutes to kill a VM but DNS records are cached for an hour.
- The attacker takes advantage of that window to spin up a bunch of tiny VMs and see if any of them are allocated one of the IPs that Houseparty recently abandoned. If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it.
- An end-user's app connects to "ephemeral-vm-837.houseparty.whatever" and gets the cached value that their ISP is still serving, because Epic configured their DNS to tell the ISPs to cache it for an hour, except now that IP is served by the attacker instead of by Epic.
Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation.
- eh78ssxv2f 7y ago> If so, then they fire up Nginx and serve poisoned PDFs to anyone who connects to it. Even if the attacker redirects the traffic, it still needs to serve the HTTPS/SSL certificates. How will the attacker do that?
- kstrauser 7y agoMaybe it does? It could be serving plaintext HTTP ("why would we encrypt our logo.png file?"), or the client library could be configured to accept any valid SSL cert because they don't want to deploy good certs to every ephemeral host they spin up. There are a lot of ways this could go wrong.
- eh78ssxv2f 7y agoAgreed. In my mind, if the server is using plain HTTP, and the attacker has control over parts of the network, then plain-old DNS is not the only attack model. The attacker does not need to hack DNS, they can just manipulate the content on the wire.
- munchbunny 7y agoI believe LetsEncrypt will issue you a certificate for that subdomain if you can control the server responding at that subdomain. EDIT: Specifically: https://letsencrypt.org/docs/challenge-types/ https://letsencrypt.org/docs/challenge-types/ It won't give you a wildcard certificate, but you don't need one for the type of attack we're talking about.
- sroussey 7y agoWell, hopefully they use certificate pinning. And if not, they should do certificate transparency like chromes does.
- toast0 7y ago> Mitigating this could be as blunt as setting DNS TTLs to 5 minutes, then putting "sleep 300" at the start of each server's shutdown script. When you decide to kill a server, first kill its DNS record so that nothing refers to it anymore, then start the waiting period before actually removing it from rotation. If I had a dollar for everytime traffic stopped to an IP after the DNS ttl expired, I wouldn't have any dollars. After taking IPs out of DNS for www.popular site, I would continue to see traffic for weeks, and the TTL was set for somewhere like 5 minutes, an hour tops. Edit: also, when we moved our authoritative DNS, the old provider kept seeing queries for at least 6 weeks. If the only thing preventing you from sending private data or receiving trusted data is that it came from an IP you got from DNS, it's not secure.
- mjw1007 7y agoI think what matters is whether the certificate authorities honour TTL correctly when they do domain-control validation. If they do, then the procedure above (and doing everything over https) should be enough, I think.
- kstrauser 7y agoOh, buddy, you know that's right. There are plenty of IPSs like, "5 minutes, lol! Two weeks it is!" What I proposed was a blunt force tool to try to help mitigate the problem, but you're 100% correct that it's not sufficient. Key pinning would go a lot further towards fixing it. What I suggest could be rolled out pretty quickly and without risk, though, so IMHO it's worth adding to the to-do list.
- thoraway1010 7y agoYou can bring your own IP range or pay for an elastic IP. Isn't this how folks do this (I do!). You get a bunch of elastic IPs, you point your domains to those, you point those to your instances. That's secure, no one can use those IPs until you release them. Is this rocket science? On AWS I don't think you really get charged for this even (or very little) from what I can tell if you have them all pointing to your instances.
- thanksforfish 7y agoYup. Used elastic IP addresses are mostly free[1]. Unused elastic IP addresses are $0.005/hr[2]. Although watch the remapping fee... $0.1 per remapping past the first 100. That could add up. [1] https://aws.amazon.com/premiumsupport/knowledge-center/elastic-ip-charges/ https://aws.amazon.com/premiumsupport/knowledge-center/elast... [2] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...
- kstrauser 7y agoThat would go a long way, but note that AWS limits you to 5 IPs per account per region[0] by default, and charge[1] $0.005 per Elastic IP per hour for EIPs that aren't attached to a running instance, and $0.10 to transfer an EIP from one instance to another. So, if you want to keep 1,000 EIPs around to map to ephemeral VMs, and you're not actively using them, you'll be paying $3,600 per month for those alone. One could argue that this is pocket change for a large customer to do the right them, but from a PM's perspective that $40K per year they could be spending towards another engineer's salary. [0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#using-instance-addressing-limit https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-... [1] https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Addresses https://aws.amazon.com/ec2/pricing/on-demand/#Elastic_IP_Add...
- jwatte 7y agoThis is what TLS certificates solve. They make sure the other end of the IP connection has the name you expect it to have.
- erpellan 7y agoIf they've hijacked a subdomain, they can submit it to LetsEncrypt and get a cert. LetsEncrypt checks that you control a domain by asking you to serve some data from it. The domain is already hijacked so from LetsEncrypt's perspective it's yours.