20 ms·

Hijack of Amazon’s domain service used to reroute web traffic for two hours

• rgbrenner 9y ago
  A website I own was affected by this. I got an alert from our monitoring saying the website was down for 1hr 2min 59sec. Luckily, they didn't redirect it to anything. I have other domains using Route53 (and hosted at AWS, just like this one).. that weren't affected AFAICT.
  • brianucf 9y ago
    Experienced the same behavior on only one of my hosted zones. Started around 11:28am UTC
  • isostatic 9y ago
    > I got an alert from our monitoring saying the website was down for 1hr 2min 59sec. That's a very accurate time. What system do you use to allow sampling at under 1 second intervals? My nagios boxes poll every minute, so an outage could be 2 seconds, or nearly 2 minutes, and nagios would report the downtime as 1 minute.
    • outworlder 9y ago
      They could be alerting based on metrics, instead of "pinging" the site.
      • isostatic 9y ago
        Yes, if the site serves up lots of traffic (multiple hits per second), then the hole in the logs would certainly give an indication. I get very suspicious when people quote things accurate to 1 part in 1000
    • breakingcups 9y ago
      Could be a process that checks every minute with slightly inaccurate measuring which leads to 1h3m minus 1 second reported.
    • syastrov 9y ago
      To be accurate, that’s a very precise time. We don’t know if it’s accurate or not.
• reaperducer 9y ago
  How do we know it was unnoticed? It could very well have been noticed and taken two hours for Amazon to mitigate it. I don't know the technical process behind making such a correction, but in any large organization there are steps that have to be followed. It's not like some kid typing out a shell command on his basement Linux box. Inflammatory/clickbait headlines do not make the internet a better place.
  • dang 9y ago
    Ok, we've taken that word out of the title above.
  • kolpa 9y ago
    If no one mentioned this important outage, it's likely that many/most/all people didn't notice.
    • ohiovr 9y ago
      How big was the attack? You can't fool all the people all the time. But fooling all their targets for even 30 minutes is a significant achievement considering what info they could have obtained. What do we know about the victims? What reason do we have for believing it was a random victim attack? Besides the insane difficulty part..
• dice 9y ago
  It was certainly noticed in under 2 hours. The Outages list thread about the issue started at 11:54 UTC: https://puck.nether.net/pipermail/outages/2018-April/011257.html https://puck.nether.net/pipermail/outages/2018-April/011257.... I'm sure that there were ops teams working on it before then. The people involved in actually fixing the problem wouldn't have been posting to public mailing lists.
  • kolpa 9y ago
    > The people involved in actually fixing the problem wouldn't have been posting to public mailing lists. Surely, but their coworkers should have been.
  • makomk 9y ago
    Looks like it was about an hour and a half before anyone in that thread figured out there'd been a BGP hijacking.
• djsumdog 9y ago
  So they obviously couldn't or didn't hijack certs, so they were depending on people who clicked through or APIs/systems that didn't do cert checks?
  • Diederich 9y ago
    Update: see other comment chains invalidating the security I'm claiming HSTS provides. APIs will probably go directly to https:// https://, and hopefully cert failures will cause the API to blow up with minimal harm. If you want to visit my web site, and you need to type it in, you'll probably just put realms.org into the URL bar of your browser. Most secure websites, such as paypal.com, have something called HSTS https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security to force the browser to always hit the site with https, even if requested to it it http: dana@araman:~$ curl -D - https://paypal.com https://paypal.com 2> /dev/null |grep -i strict-transport Strict-Transport-Security: max-age=63072000 dana@araman:~$ So even if going to paypal.com lands you on a blackhat server, the cert will still blow up, protecting you. Slightly related: much to my annoyance, it seems that neither AWS ELB nor ALB support sending HSTS, which is a real bummer.
    • philjohn 9y ago
      Except, it would have been trivial for them to get Lets Encrypt to issue them a valid cert.
      • Diederich 9y ago
        Yup, I saw that after I commented. I edited and put a warning at the top of my post.
      • Manozco 9y ago
        it depends, let's encrypt might be performing request from different sources (and then it would need a bigger BGP hijack to fake them)
        • 9y ago
• zegl 9y ago
  We experienced the same thing. DNS lookups via Googles public DNS servers where failing for one of our hosted zones. Interestingly this only seemed to be a problem with Google DNS, other large providers as well as smaller ones where not affected.
  • jbergstroem 9y ago
    As I mentioned in the previous thread; DNS lookups via Azure also seemed to be affected.
  • fernandotakai 9y ago
    we saw the same thing. dns stopped resolving while using google's dns for about 2h. there's a route53 status update on their page that says basically that -- only affected google dns.
    • eastdakota 9y ago
      That’s simply wrong. Best case, this is ignorance on the Route53 marketing team. Worst, it’s actively misleading. Disappointing either way.
• conistonwater 9y ago
  > This traffic was redirected to a server hosted in Russia, which served the website using a fake certificate — they also stole the cryptocoins of customers. What does this sentence mean? Sorry if I'm being slow. I thought the whole point of certificates is that you can't generate a legitimate one for a domain you don't control, so messing with name resolution wouldn't affect it.
  • deleted 9y ago
    [deleted]
  • polpo 9y ago
    It means just that. Since they had control of DNS, they could have easily gotten a DV certificate, but in this case, they didn't and people clicked through certificate warnings.
    • polpo 9y ago
      Here's a source on Twitter showing some documentation of the self-signed certificate that myetherwallet.com users ignored warnings about: https://twitter.com/GossiTheDog/status/988785871188045825 https://twitter.com/GossiTheDog/status/988785871188045825
    • ryanlol 9y ago
      Relatively easily, you'll need to be prepared with a mix of DV providers to make sure you don't get screwed by DNS caches.
    • discreditable 9y ago
      > people clicked through certificate warnings Which is a shame. They could have prevented users clicking through warnings by implementing HTTP Strict Transport Security.
  • kodablah 9y ago
    Maybe they didn't mean "fake certificate" as we think of it. Maybe they meant a valid certificate that they got from something like LE since they could show they hosted the domain (via the DNS check or the HTTP check).
• peterwwillis 9y ago
  This same attack can be used to generate valid TLS certs for any website, for example using Let's Encrypt. The best part? Your target doesn't need to use Let's Encrypt at all. Anyone can use them to forge certs for any domain. Of course, this is possible with most other cert providers, but Let's Encrypt automates it. So.... TLS certs mean jack squat if you can pull off a BGP+DNS attack. You might want to start pinning certs for your bank's website. (but not using HPKP, Google is removing it from Chrome)
  • flipp3r 9y ago
    What's also interesting is in the comments someone stated they'd used a Godaddy certificate issued on april 7. Maybe they had access to their account, or Godaddy's internal systems?
    • peterwwillis 9y ago
      Like I said: anyone can generate a valid cert for any domain. You don't need to hack Godaddy or anyone's account. Create a new account, hijack the target's DNS, generate a valid cert for the domain you want.
      • pbhjpbhj 9y ago
        It's surprising that GoDaddy allow the issuing of the cert. I'd expect them to require some proof of ownership.
        • ceejayoz 9y ago
          That proof of ownership is generally "publish a DNS record or a file at this specific URL", both of which hijacking DNS would permit you to do.
        • antonvs 9y ago
          They do require proof of ownership, but it's recently become common to support validation by proving that you control the domain or web site being validated so that the proof is purely electronic, e.g.: https://www.godaddy.com/help/verify-domain-ownership-html-or-dns-7452 https://www.godaddy.com/help/verify-domain-ownership-html-or... In a case like this, the attackers had sufficient control to pass those kinds of checks - although people are saying they didn't do that, and just relied on enough users clicking through the certificate warnings.
• solotronics 9y ago
  was this related to the myetherwallet.com hack? they hijacked BGP and DNS to redirect traffic to their own page https://altcoinreport.co/my-ether-wallet-hacked/ https://altcoinreport.co/my-ether-wallet-hacked/
  • AlphaWeaver 9y ago
    Yes, the article says the only compromised site they are aware of was MyEtherWallet.
  • nwsm 9y ago
    rtfa
• ed_balls 9y ago
  How does one prevent an attack like this? APIs and Browsers.
• lifeisstillgood 9y ago
  Can I check my understanding (Somehow) A BGP route was advertised and accepted for the IP range for amazons route53 - so that some / many DNS requests hit a fake server instead of the real amazon one. The only dns requests that changed seem to be for a bitcoin site, and they were redirected to a site that seems to have then grabbed their passwords and then emptied their wallets on the real site. ? Mitigation strategies that come to mind - certificate pinning (where I have my browser only accept certificates that I have approved, not just are signed by a CA (this has not really taken off) - Certificate warning - surely browsers should shout of a certificate that was valid for domain X changes today? (suffers similar problems to pinning) - BGP route acceptance. We are out of my comfort area now, but this is surely where the real solution lies - why did people accept a BGP chnage to redirect route53 to russia? Can routes be advertised with signing ?
  • jimktrains2 9y ago
    > surely browsers should shout of a certificate that was valid for domain X changes today Why? If the new cert is valid, why sound any alarms?
    • lifeisstillgood 9y ago
      I visit my bank site and my browser gets a certificate for barclaysbank Ltd, to expire in 12 months. Tomorrow i visit the same site and get a different certificate, or from a different CA. Should my browser warn me? I don't know. the chances are high that 99% of people will click OK and 99% of the 1% left will look at it and think "how do i verify this?" That is probably the reason browsers don't bother with pinning and the rest - there is simply no chain of trust i as an individual can possibly use to verify a fraction of the sites i visit. but still ... if I pay my landlord every week in cash and one day someone else turns up and says "hi i am your landlord, pay me" I have visual signals to warn me of a possible problem. The signals are there - my browser could fingerprint the servers, the headers, the response times, a lot could be done. But then most days my browser would act like Donald Sutherland and point at people and say "they have been replaced" and i would have no means to tell. we really have become reliant on one piece of technology haven't we.
• bklyn11201 9y ago
  Getting a basic certificate issued is incredibly easy these days. If you own the DNS resolution, you can get a cert. Here a few ways I can think of to make this harder for an attacker: * Add a Strict-Transport-Security to all HTTPS requests. This means the attacker will need to get a valid cert (still easy if you can hijack BGP). * Pick a preferred SSL issuer and stick with them. Add a CAA DNS record only allowing that one issuer. * Go through the EV SSL process with your preferred issuer. Now modify your CAA DNS record to add an EV policy. This should make it very difficult for an attacker to get a cert issued during a BGP hijack. * DNSSEC? Make sure to choose an SSL issuer that will correctly test DNSSEC?
  • tptacek 9y ago
    Choosing a particular CA doesn't help, since attackers can pick whichever CA is optimal for their attack.
  • EE84M3i 9y ago
    Forgive my ignorance but I have a few questions: * Why would HSTS help in this case? While HSTS is active, does it prevent clicking through the warning (which was done here)? * How would a CAA record help against cert issuance in this case? Is it only helping against compromise of the authoritative during the remaining TTL of the record in recursives AND if the CAA record points to something that doesn't have on-demand DV like Lets Encrypt?
    • bklyn11201 9y ago
      HSTS wouldn't help users clicking through warnings, but it's a good thing to have (myetherwallet doesn't use HSTS). CAA record would only help in remaining TTL. Once expired, then it doesn't matter. So yeah, these seem like decent steps to help protect but certainly not going to 100% prevent an attack like this one.
      • DividableMiddle 9y ago
        wait, myetherwallet doesn't use HSTS? I thought they had a vulnerability analysis done a few months ago. I feel like this should have been something that was caught.
• omarforgotpwd 9y ago
  Wow. Just wow. These BGP vulnerabilities are ridculous. Imagine, someone taking over DNS for even a small subset of people and being able to basically just rewrite the internet as they see fit, completely taking control of anything. Even without being able to get a valid SSL certificate you could do a lot of damage. For example, let's say I rewrote requests for SomeNationalBank.com to my proxy server. I make a request to the real SomeNationalBank.com with the weakest set of ciphers allowed, and then pass those bytes along to the client. Then I can try and decrypt it and log into your bank account as soon as I do. OR better yet I wait for you to log into an unsecured site like neopets.com or something and then pair that unencrypted password wait the fact that I saw you request the domain for a certain bank and if your password is the same (which it is for most people) then that can also get me in. We need a new more secure system to replace DNS.
  • pdimitar 9y ago
    Check out IPFS' DNS alternative -- IPNS. No blockchain needed, it's pretty solid.
    • d4l3k 9y ago
      The issue with IPNS is it's based off of public private key crypto which means the "domain" is a hash and has no name recognition. While still useful, it's not going to make its way into the mainstream anytime soon. For instance, I host my personal site on both IPFS and normal web traffic. https://fn.lc/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnknc2WGCGQ3HnDP6f/ https://fn.lc/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnknc2WGCGQ3H... https://gateway.ipfs.io/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnknc2WGCGQ3HnDP6f/ https://gateway.ipfs.io/ipns/Qmea45XwFtdwaCGAPLRMxFmoUP5YLnk...
      • alexdumitru 9y ago
        The first one took 61 seconds to load, while the second one took 121 seconds. It doesn't seem very practical.
    • KirinDave 9y ago
• adamb_ 9y ago
  Route 53 DNS queries getting black-holed at a small ISP in Ohio: https://pbs.twimg.com/media/Dbka0oAVMAAFEXx.jpg https://pbs.twimg.com/media/Dbka0oAVMAAFEXx.jpg
• notyourday 9y ago
  This is another one of the articles that says absolutely nothing. The only reason why such attack was possible is because some of the providers did not filter announcements from hobos claiming to be able to advertise AWS space. That could have only happened if: a) AWS did not register all their policy prefixes b) providers did not apply filters on the transit announcements from those that announced AWS prefixes based on the policies registered by AWS. This is a modern day "let residential IPs to send data directly to tcp/25 and not check".
  • lathiat 9y ago
    If you think the majority of BGP speakers world-wide are applying these announcement filters you are sadly mistaken.
    • notyourday 9y ago
      I dont remember when was the last time that I was able to just announce anything I wanted either as a customer or peer to any large network. So as long as the large networks filter their customers and peers it seems to minimize the exposure. [1] [1] That statement excludes Google whose idea of filtering is to let me announce anything I want and build filters based on what I announce.
• vasilipupkin 9y ago
  can someone ELI5 how this works? like, how do they physically do this ?
  • chatmasta 9y ago
    I’m not an expert, but I think it goes something like this — You register an AS and get a dedicated server [0] at at an internet exchange point (IXP) and install a BGP router on it. Your router advertises a route to the original route53 IP, with a shorter AS path to the target IP than the route advertised by the “real” Amazon router. You peer your router with at least one provider who peers with the real route53 servers (they accept your route advertisements). Then you redirect incoming traffic on port 53 to a mitm dns proxy, where you either do nothing or return an IP address you control, which could be anywhere. At the IP address you control, you run a transparent HTTP proxy that signs HTTPS responses with the valid DNS cert you obtained by spoofing MX replies in the dns proxy. You then either do nothing, or modify the page to return arbitrary content (like rewriting bitcoin addresses). [0] or you get a VPS at a provider willing to advertise BGP routes on your behalf without validating them
    • rocqua 9y ago
      Not an expert either, but I think it was slightly different. Firstly, I believe BGP hijacking works by announcing a more specific prefix rather than path-length. That is to get to 3.3.3.3 A 3.3.0.0/16 route takes precedence over a 3.0.0.0/8 route. Secondly, this attack did not obtain a valid cert. They probably could have using spoofing (either of MX or just of A records and using lets-encrypt), but they did not. This might be because the BGP hijack did not affect any CA, or it might just have been laziness.
      • chatmasta 9y ago
        Yeah you're correct. There was a great defcon presentation on this in 2015: https://www.blackhat.com/docs/us-15/materials/us-15-Gavrichenkov-Breaking-HTTPS-With-BGP-Hijacking.pdf https://www.blackhat.com/docs/us-15/materials/us-15-Gavriche... Really the hardest part of the attack, and what makes it so infrequent, is the social engineering aspect of registering the AS and getting an influential peer to accept your advertisements. To really cover your tracks you would need fake identities, fake businesses, fake bank accounts...
• technologyvault 9y ago
  Does anyone know if this happened to affect Amazon's Seller Central? Amazon sellers (including me) couldn't access their control panels for several hours last night.
• nodesocket 9y ago
  If BGP is open and anybody can modify routes, why don't these attacks happen constantly? What prevents these sort of attacks and how are they fixed?
  • notyourday 9y ago
    It is not open. Sane networks register prefixes. Other sane networks do not accept prefixes that are not registered to parties announcing them. If your network does not register prefixes, you need to fire them and move to someone else. If your network does not accept only registered prefixes, you need to fire them and move somewhere else. Not registering prefixes/filtering based on registrations is having sex without condoms with a toothless hooker behind a dumpster who charges $5 for a full service.
    • zero_intp 9y ago
      This comment is only partially true, the realities of managing customers on routers with thousands of routes precludes this careful management at the rock bottom tier-1 transit prices of the lowest common demoninators. The way HE & Cogent operate for large web-hosters at data centers is by trust and prefix-counts. The churn of various web-operators between hosting companies is massive.
      • notyourday 9y ago
        Did not Cogent pioneer massive use of multihop BGP to do prefix filtering outside the low powered edge in the nineties?
        • zero_intp 9y ago
          Starting in 2000, yes.
  • codetrotter 9y ago
    Just guessing here but it’s only open to the companies and countries that are peering with one another isn’t it? If so then companies that intentionally advertise routes that are not theirs to advertise would lose their peering agreements with others because other companies would no longer want to peer with them.
• kentt 9y ago
  Wow myetherwallet is trusted for transaction very non-trivial amounts. What would a good mitigation strategy be for these types of attacks.
  • artursapek 9y ago
    Making sure the cert is valid.
    • quickthrower2 9y ago
      Also do a dry run before each usage, generating a throwaway wallet, and inspect the network tab / use fiddler to check if there is any suspicious sending of data. Even this isn't 100% as they could seed keys if bad person gets control of the website. This has been done many times in the IOTA community for example, albeit those sites were dodgy from the get-go. So I'd probably get a snapshot of their client code when you trust it, and serve it to yourself locally thereafter. It's a sad state of affairs that people use services like this. I have done so myself :-( The reason is that the Ethereum desktop clients suck. More than suck, they are untenable. The official client never synced even after running for 2 weeks. I tried this twice. It is awful.
      • g09980 9y ago
        Also, you can save the MEW webpage locally and use it.
  • influx 9y ago
    Using a hardware wallet and verifying the address on your hardware wallet is the same you entered into the website.
• deleted 9y ago
  [deleted]
• sajal83 9y ago
  https://pulse.turbobytes.com/results/5adf2844ecbe40692e003ad2/ https://pulse.turbobytes.com/results/5adf2844ecbe40692e003ad... Some traceroutes captured during the incident. The results that show "Target unreachable" are the ones seeing the hijacked paths
• OJFord 9y ago
  Surely related (and warning!): During this period I received an email from: ship-confirm@amazon.com purporting to notify me of the despatch of a non-existent order; with an attached .zip. Nothing about the message, except the .zip and suggestion of an order's existence, was at all suspicious - all links point to amazon.com. I checked on amazon.com (without following a link) and on amazon.co.uk (where I would usually shop) and an order with the given number does not exist for my account.
  • ars 9y ago
    That's just regular spam. I bet if you checked you would see the email was not actually sent by Amazon, and did not have a DKIM signature.
    • OJFord 9y ago
      Maybe. I've deleted it now, but the headers looked OK to me. If I'd actually ordered something (e.g. it had come a day earlier) I'm sure I'd have taken it as being legit.
  • thetinguy 9y ago
    Why is an obvious phishing email related?
• tribune 9y ago
  How did they manage to steal coins if they didn't get a valid cert? Did people logging into MyEtherWallet just ignore the invalid cert warnings and log in anyway?
  • move-on-by 9y ago
    Could be, could also be API clients not doing cert validations
  • deft 9y ago
    Yes, multiple people on reddit have said that's exactly what they did.
  • Already__Taken 9y ago
    In addition they could be behind rubbish proxy's that aren't passing the warning forward. I know ours in work will happily resign that expired cert to mitm you for compliance and the user will see that green padlock.
    • rocqua 9y ago
      I get that companies need to control access to their internal networks. But surely when you do it this badly you have to realize you are only making your network less secure.
      • tialaramex 9y ago
        So far as we've been able to tell all the middleboxes on the market for this sort of purpose are worse than useless. If you remember that NCSC blog post that annoyed Adam Langley, its author Ian Levy insists that "there are some good products out there" I actually replied to that comment, requesting a list of these "good products" which presumably are warranted by the NCSC not to actually be worse than useless. Unsurprisingly Ian has elected not to in fact list any such products. There aren't any.
• 7ewis 9y ago
  How would you see this in ThousandEyes? I see he has a screenshot of it in the post, and doesn't appear to be logged in. Was troubleshooting this for the two hours, until I started reading reports of the BGP leak. Not in networking, so don't fully understand how it works. I'd just like to know how to troubleshoot this in the future, and check for leaks, if any. Are there any other online tools I could use? Obviously don't have an edge router or anything like that.
  • adamb_ 9y ago
    > How would you see this in ThousandEyes? Checkout this video analysis from the team on the hijack: https://youtu.be/YXm4GJMUlP0 https://youtu.be/YXm4GJMUlP0
    • 7ewis 9y ago
      Thanks! Do you know how I can access that page he is on? I'd like to have a click around myself.
      • ruddell 9y ago
        Here's the link he starts on in the video: https://webysvi.share.thousandeyes.com/view/tests/?roundId=1524576600&metric=availability&scenarioId=dnsServer&testId=619444&serverId=239927 https://webysvi.share.thousandeyes.com/view/tests/?roundId=1...
• sfsd11 9y ago
  Quite ugly, I can totally see where this is all headed in the near future... nod if you agree.
• textmode 9y ago
  "Between 11am until 1pm UTC today, DNS traffic-the phone book of the internet, routing you to your favourite websites-was hijacked by an unknown actor." Is it worth keeping a record of the IP addresses for "your favorite websites"? For example, with IP addresses saved, would this enable reaching the websites ven if DNS is not working? How often do these "favorite websites" change IP addresses? Are they all the same in that regard? (The frequency with which they chaange IP addresses.) Is it worth paying attention when one of them changes its IP address? Is it worth noting where these addresses are thought to be located (e.g. the countries)? What if several of "your favorite websites" each change their IP address on the same day, the same week or even the same month? Is this worth noting?
  • viraptor 9y ago
    Depends on how the website is hosted. If it's a tiny website on a single server/VPS, then it's going to be fairly stable with the IP. For anything else, you're out of luck. Shared hosting or PaaS will change ips when needed internally, or to rebalance the traffic, or when new hosts come up. Anyone with large enough traffic will use some edge network like CloudFlare which would also get a different address depending on your location. Some IPs may stay stable for a long time, but virtually every change to them is going to be intended. And you can't tell easily which one want.
    • textmode 9y ago
      "Anyone with large enough traffic will use some edge network like Cloudflare which would also get a different address depending on your location." What if the request is always sent from the same location? What if when the user moves location, e.g., from one country to another, she updates her stored IP addresses? "And you can't tell easily which one want." Personal experience as a user is that when it is an "intended" change, the previous IP address no longer hosts the content. Personal experience is that this is surprisingly rare. In the case of the more common load balancing, the user who may store several IP addresses for the website, all of which continue to serve the content. She can choose which of those she prefers. Personal experience is that this works very well. Further questions: If this was a BGP hijack, why did the hijackers target the IP addresses of DNS servers, rather than the target IP addresses of the websites? If they had targeted the IP addresses of the websites, then what would be the possible mitigations, if any?
• r1ch 9y ago
  It's sad that HPKP is being deprecated. It's one of the best ways to defend against an attacker with bgp hijack capabilities assuming you pin your own public key. Difficult to scale though and prone to disastrous misconfiguration.
  • tomc1985 9y ago
    Too many people have foot-gunned themselves with that one. As someone who has to readily fight technical fires for other people (when they should have googled their problems) I am glad to see HPKP go.
    • dcbadacd 9y ago
      Honestly there are a lot of ways for people to foot-gun themselves when dealing with providing a service. HPKP isn't perfect, but deprecating it just for the sake of keeping some incompetent people safe is just silly. Might as well remove HSTS and not roll out Expect-CT because everything allows someone to foot-gun themselves.
      • pfg 9y ago
        The difference between HSTS, Expect-CT and HPKP is that the former two offer a way out (support HTTPS, provide qualified SCTs) whereas HPKP can effectively brick your domain for a couple of months, and it's not even hard to pull off.
        • dcbadacd 9y ago
          HPKP can brick your domain if you use HSTS alongside with it, if you don't then one can fall back to http. Not an ideal solution, but it's a way out. HSTS is an extreme and possibly dangerous measure that should be carefully considered (maybe it shouldn't be TOFU but TOTU - trust on third use), but I still think it's not reasonable to totally deprecate it. There are cases where such measure would provide clear and strong security benefits.
      • tomc1985 9y ago
• dharma1 9y ago
  Also experienced this today - was doing user testing on an app, and suddenly our AWS hosted app that uses Route 53 stopped working, didn't resolve. When I checked http://status.aws.amazon.com/ http://status.aws.amazon.com/ it showed all green for Route 53 though. Luckily I noticed the app kept on working on my own devices where I had previously switched the DNS to 1.1.1.1 (Thanks HN for the heads up on that a few weeks ago!) - otherwise would have been a waste of half a day in an expensive user testing lab. Assumed it was Putin's blocking of Amazon and Google IP's that had been on the news, but guess it was just hackers after some ETH. Or was it?
• mercora 9y ago
  I wonder why they did not reroute traffic for their targets directly. I assume the amount of traffic the Route53 servers get is quite a lot to handle and failure to do so would make a lot of people aware of some issue. Also, how is it possible to reach the original servers in order to forward non targeted lookups to them? Is it possible to have your own routes not affected by your own BGP announcements? Maybe they were able to do this by using EC2 internal DNS resolvers? Maybe Route53 does use individual IP addresses for the nameservers of registered domains? Or at least are using them hihgly segmented? I guess this would help to handle the amount of traffic quite a bit if so. Otherwise it seems inplausible that the mentioned site is the only target or that there was just a few targets because that should be easier to pull off. On the other hand i am not sure if these targets might share IP addresses with a large pool of websites and handling "only" authoritative DNS requests for them (and any other that happens to have the same IP addresses setup as nameserver) would be alot easier. How does Route53 handle delegation exactly? Would i have to handle all NS requests that happens to resolve to Route53 or just some specific set of domains? Only handling these delegation related NS requests might reduce the amount of traffic alot in any case. But maybe they setup glue records for registered domains. In this case the amount of traffic they would have to handle would be even larger and i guess disrupt quite alot of DNS clients because they would not be able to sign the zone properly. I also wonder why they were not able to issue a new valid certificate. This appears to be very easy even without any interruption at this stage. Especially this feels like it was only done in order to direct the attention away. Hopefully a more detailed writeup appears because this seems rather interesting.
• mcqueenjordan 9y ago
  What's with the clickbait title? It is simply a falsehood. Here's the update from the horse's mouth: [5:19 AM PDT] We are investigating reports of problems resolving some DNS records hosted on Route53 using the third party DNS resolvers 8.8.8.8 and 8.8.4.4 . DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers are not affected at this time. [5:49 AM PDT] We have identified the cause for an elevation in DNS resolution errors using third party DNS resolvers 8.8.8.8 / 8.8.4.4 and are working towards resolution. DNS resolution using other third-party DNS resolvers or DNS resolution from within EC2 instances using the default EC2 resolvers continues to work normally. [6:10 AM PDT] Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced elevated errors resolving DNS records hosted on Route 53 using DNS resolvers 8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-party Internet provider. The issue has been resolved and the service is operating normally.
  • jsmthrowaway 9y ago
    The horse's mouth, in this case, speaks nonsense. Multiple /24s were announced more specifically to the DFZ, and that has absolutely nothing to do with Google's DNS resolvers. "Simply a falsehood," indeed. https://twitter.com/InternetIntel/status/988792927068610561 https://twitter.com/InternetIntel/status/988792927068610561 https://arstechnica.com/information-technology/2018/04/suspicious-event-hijacks-amazon-traffic-for-2-hours-steals-cryptocurrency/ https://arstechnica.com/information-technology/2018/04/suspi...
    • lathiat 9y ago
      The horses mouth actually does say it, you'll note they said "This issue was caused by a problem with a third-party Internet provider." As soon as I read this and saw the myetherwallet announcement I called it last night that's exactly what they did, and today I find out that was the case.
• chatmasta 9y ago
  What separates BGP hijacking from BGP anycast? Amazon is already advertising a bunch of routes to its route53 IP's for BGP anycast. How can an upstream peer tell if a new advertisement is another anycast route vs. a hijacked prefix? Is the only way to tell by validating the advertisement came from Amazon?
  • xraystyle 9y ago
    The issue here is that the attackers were advertising more specific prefixes, and when doing route selection, routers pick routes to these over less specific prefixes that cover the same ranges. e.g. if Amazon was announcing 205.51.192.0/21, that would cover all the /24s that were announced in the hijack. Not sure what Amazon actually announces, this is all just examples. Say I'm trying to get to 205.51.192.12, for example. If I have two routes, one advertising 205.51.192.0/21 and another advertising 205.51.192.0/24, the router will forward the traffic to the next-hop advertising the /24, all else being equal. So if you have the ability to advertise routes over BGP to the internet, you advertise a more specific route than what someone else is already advertising, and your peer has no filtering in place with regard to what they're accepting from you, you can potentially hijack IP space like this. ETA: Anycast DNS generally advertises the exact same prefix to multiple peers. That way, the best path to that IP will be dependent on where your traffic is coming from, and will hopefully come in over the closest peering to you.
    • lathiat 9y ago
      As much as it's bad for routing table inflation, given how much content they host Amazon should probably consider de-aggregating their DNS resolver prefixes to /24 announcements which would limit the effect to be much more local for the majority of the Internet.
• maltalex 9y ago
  Something similar happened once in 2014 [0] > That BGP hijack allowed the hacker to redirect the miners’ computers to a malicious server controlled by the hijacker. Also, the entire blockchain ecosystem is very susceptible to BGP hijacks [1] [0]:https://www.wired.com/2014/08/isp-bitcoin-theft/ https://www.wired.com/2014/08/isp-bitcoin-theft/ [1]:https://btc-hijack.ethz.ch/ https://btc-hijack.ethz.ch/
• unabridged 9y ago
  With all the talk of blockchain, dns should be one of the best uses.
• 925dk 9y ago
  Why do AWS http://status.aws.amazon.com http://status.aws.amazon.com say that only Google resolvers were affected? Between 4:05 AM PDT and 5:56 AM PDT, some customers may have experienced elevated errors resolving DNS records hosted on Route 53 using DNS resolvers 8.8.8.8 / 8.8.4.4. This issue was caused by a problem with a third-party Internet provider. The issue has been resolved and the service is operating normally.
  • fooblitzky 9y ago
    It seems like either the title is misleading, or the article does a poor job of explaining the situation. From my reading of the article text, it seems DNS traffic was rerouted to Route53, which the attackers then used to serve false DNS records. That does not sound like Route53 was hijacked at all, just that the attackers happened to use the service it provides.
    • 925dk 9y ago
      Traffic to Route 53 was rerouted to an alternative DNS server. But that has nothing to do with my question re. AWS calling out Google in particular in their status update.
      • dgemm 9y ago
        Some Route53 traffic was redirected somewhere else. Traffic can only be hijacked from neighbors that actually accept the routes. You would think Google of all networks would have effective ingress filtering to prevent this, but it seems like they did accept it in this case.
        • dward 9y ago
          I don't think ingress filtering on Google's edge would have helped if the rerouting happened in any of the transit ASs between AWS and Google.
• mlosapio 9y ago
  Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.
  • erulabs 9y ago
    I would assume they go far out of their way to resolve the address from a large number of distributed DNS resolvers - the question is if they're just round robin instead of multiple simultaneous queries, but I would hope that's part of why the DNS test usualy takes >60 seconds. If they're not treating DNS as a consensus protocol instead of as an immutable database, they'll always be open to poisoning attacks. If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation"
    • btown 9y ago
      Per https://github.com/letsencrypt/boulder/blob/master/bdns/dns.go#L260 https://github.com/letsencrypt/boulder/blob/master/bdns/dns.... it seems they round robin. But they are aware of the issue in the spec: https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html#rfc.section.10.2 https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.htm... - “Querying the DNS from multiple vantage points to address local attackers” is a mentioned mitigation that a server could implement. Seems like a reasonable basis for a pull request.
    • dexterdog 9y ago
      But on a big, distributed site the DNS is going to vary greatly depending upon the location of the query.
    • mnordhoff 9y ago
      > why the DNS test usualy takes >60 seconds The server-side part of DNS validation takes about a second. The delay is all about clients waiting for their authoritative DNS servers to update. If you use a fast DNS provider, there's no reason to wait longer than necessary. > If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation" It wouldn't be a compliance failure, though. CAs are not required to be invulnerable to BGP hijacking attacks.
• pierd 9y ago
  Is there any official post mortem available?
• pierd 9y ago
  The attackers are getting richer: https://etherscan.io/address/0xb3aaaae47070264f3595c5032ee94b620a583a39 https://etherscan.io/address/0xb3aaaae47070264f3595c5032ee94...
• quentusrex 9y ago
  Another write up of the event: https://blog.cloudflare.com/bgp-leaks-and-crypto-currencies/ https://blog.cloudflare.com/bgp-leaks-and-crypto-currencies/
• profalseidol 9y ago
  Time to use the Ethereum Name Service (ENS). As long as you keep your private key safe, your domain name is safe.
• jwbensley 9y ago
  A lot of people in here are talking about securing their DNS setups but not their BGP setups. For any BGP operators out there, you could automate your inbound/outbound BGP prefix filters using IRR/LIR databases, peeringdb.com, RADB etc. Tools like https://github.com/snar/bgpq3 https://github.com/snar/bgpq3 can help with auto-generation. Even manual prefix lists will do, anything is better than nothing! It would be good if operators could also use AS-SETs so that other operators can see what downstream ASNs should be seen via their peers (and which ones should not). Also BGP RPKI (Resource Public Key Infrastructure) based ROA (Route Origin Authorisations) can also help to ensure that prefixes are announced by their true originator. Most networks follow the Pareto rule - 20% of prefixes learnt via BGP (in some cases even less!) are the source/destination for 80% of their traffic flows. Some operators use an approach of writing explicit filters for those 20% of prefixes explicitly blocking them from all their BGP peers except the origin peer sessions. None of these techniques are fool proof and we all need to play along to improve their impact. They cost nothing, only man hours, and we can all benefit. In metaphor - no encryption is unbreakable but we can try and make it unappealingly difficult.
• donttrack 9y ago
  Isn’t dns hijacking a major problem? I could go to a Starbucks and setup my own dhcp server on my laptop. A certain fraction of the WiFi devices would probably pick up my dhcp given dns which would reroute requests to my own dns server. Nobody would ever know... or am I misunderstanding something about https? Or if I was a malware writer I would discretely target hosts files or the local dns configuration - exploits in certain routers to try to reconfigure the complete local network and such.
  • hyperpower 9y ago
    You are missing something about HTTPS: you can't produce a valid certificate for the site you're trying to impersonate, and therefore users would notice. HTTPS operates on the assumption that DNS is insecure.
  • hoschicz 9y ago
    That would only work for websites that do not use HSTS.
  • jmanderley 9y ago
    Let's hope your local Starbucks has device isolation enabled so that isn't possible.
    • donttrack 9y ago
      I will find out and report back.
• rodionos 9y ago
  > which served the website using a fake certificate This would cause an untrusted certificate error in browsers. > failed to obtain an SSL certificate while man-in-the-middle attacking The sentence doesn't make sense. MITM is by definition an attack where an actor was able to impersonate the target. In this case, they presented an certificate that was not signed by a CA in the root CA file on the clients. So this can be qualified as a failed MITM attempt.
• rodionos 9y ago
  If you use Route53, I encourage to you to check crt.sh if any SSL certificates were issued for your subdomain over the last few days. Example: https://crt.sh/?Identity=%.MyEtherWallet.com https://crt.sh/?Identity=%.MyEtherWallet.com Keep in mind that crt.sh has a certain time lag between the time a certificate is issued and a certificate entry is inserted into the crt.sh database.
  • YouKnowBetter 9y ago
    Your advice should get much much more attention. You got mine, thanks.