10 ms·

BGP leaks and cryptocurrencies

196 points by jgrahamc 9y ago

mlosapio 9y ago
Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.
- dannyw 9y ago
  BGP hijacks sadly happen on a weekly basis. It’s not new and there are hundreds more cases. It’s only noteworthy this time because high profile targets were affected instead of these being used for spear hijacking, etc.
thisisit 9y ago
> But during the hijack, it returned IPs associated with a Russian provider (AS48693 and AS41995). Curious what stopped cloudfare from naming these providers.
- coolspot 9y ago
  These are small providers with non recognized names. There is little point of naming them.
- d215 9y ago
  suit yourself: https://stat.ripe.net/AS48693#tabId=at-a-glance https://stat.ripe.net/AS48693#tabId=at-a-glance https://stat.ripe.net/AS41995#tabId=at-a-glance https://stat.ripe.net/AS41995#tabId=at-a-glance
  - lostmsu 9y ago
    One of them has just a name: Barbarich Viacheslav Yuryevich
- deleted 9y ago
  [deleted]
- jgrahamc 9y ago
  Nothing. It's just that you'd probably never have heard of them.
- cthalupa 9y ago
  They did. To have an AS number you've gotta register it with a provider (In this case RIPE, as noted by d215). You can always look up an AS number and get ownership details.
isaack 9y ago
If I were the hacker: 1. I would get a proper SSL cert signed 2. Only hijack one of the ranges, and do not respond to other domains (causing SERVFAIL), so other domains will resolve unaffected, instead of causing a scene (see: https://www.reddit.com/r/sysadmin/comments/8ejrkk/google_dns_issues/ https://www.reddit.com/r/sysadmin/comments/8ejrkk/google_dns...)
- cjbprime 9y ago
  > 1. I would get a proper SSL cert signed Uh, how? Are you assuming that a single BGP leak would be enough to cause e.g. a letsencrypt misissuance for the domain? It sounds like (from other comments) they have a global round-robin resolver setup for their DNS challenges. > 2. Only hijack one of the ranges, and do not respond to other domains (causing SERVFAIL), so other domains will resolve unaffected I think exactly that's what they did. I saw people posting SERVFAILs during the outage.
  - baby 9y ago
    > they have a global round-robin resolver setup for their DNS challenges no they don't
  - geoah 9y ago
    > I think exactly that's what they did. I saw people posting SERVFAILs during the outage. Author is suggesting they should be properly resolving the rest of the domains so they wouldn't cause SERVFAILs.
  - zyberzero 9y ago
    They do, and they don't. They have a round robin setup, for which resolver that validates the DNS challenge - however, they do not validate the DNS challenges from several resolvers[1]. So, if that particular resolver got caught in the BGP leak while doing a challenge verification you could get a valid cert. Lots of ifs and buts - but it is certainly possible. [1] https://news.ycombinator.com/item?id=16918382 https://news.ycombinator.com/item?id=16918382
vertex-four 9y ago
So what’s happening with the deployment of RPKI[0]? It would seem that owners of IP address space would have the incentive to use it - but perhaps routers don’t implement BGPSec widely enough? [0] https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastruc...
- vbernat 9y ago
  RPKI doesn't really solve the problem as it is a proof of origin. You can still inject routes, just ensure you add the original AS as origin of these routes. BGPsec would solve this issue, but is currently not deployed at all (and won't be before a long time).
edf13 9y ago
Another excellent write-up from cloudflare of a major issue... good job!
bluesign 9y ago
I find it funny how the article concludes basically almost everyone is responsible. And probably nobody will be hold responsible for this other than hacker. Thats why noone in this chain will have any incentive to fix this ever occuring again.
thesimon 9y ago
>A few things can be done on the upper levels of the network stack. On DNS, you can use DNSSEC to sign your records. The IPs returned by the fake DNS would not have been signed as they do not have the private keys. Would this have helped though? The attacker's DNS resolver could just return records without DNSSEC.
- boudin 9y ago
  With DNSSEC, the public key is not hosted on the zone iself, where your zone delegation is hosted. It's the DS records : https://uk.godaddy.com/help/what-is-a-ds-record-6148 https://uk.godaddy.com/help/what-is-a-ds-record-6148
  - cthalupa 9y ago
    I don't see how this really solve this attack. Like, it solves the specific way they did it - standing up their own DNS server on the IPs they've started advertising - but really, all they have to do is announce the IP(s) that are in the DNSSEC signed records. Then you're still getting the traffic, and nothing has thrown any red flags on the DNSSEC side because nothing has changed as far as DNS goes. The problem is that when you can hijack the routing for an IP there's nothing that can realistically be done with how the internet works today to protect again it. Basically every system of validation requires you have access to some outside authority that can verify authenticity, and if you can announce the route to those IPs to your own infrastructure, you can fool the validation.
    - boudin 9y ago
      If i'm not mistaken, in this scenario they only hijacked the dns traffic and were using it to resolve a domain to another IP, so they were changing the DNS zone. DNSSEC would have prevented this as the change they did in the zone needs to be signed, so it would require them to also get the private key used to sign the zone. It fails to solve the issue if the dns resolver doesn't check DNSSEC signatures though. If they managed to hijack the route to the server itself, there's no need to hijack DNS at all, so it's a different issue that can't be solved at DNS level itself as DNS is not involved in the attack
mehrdadn 9y ago
Taking a step back here... shouldn't proving ownership of a domain involve the domain registrar some way, rather than involving whoever happens to host your DNS? They know who you are, so maybe they can provide an interface to let you prove to others that you own the domain. Like I dunno, maybe just let the owner of the domain generate a unique one-time key valid for 30 minutes, and provide an API to allow anybody to check ownership if given that key, e.g. at a standard URL like yourregistrar.com/domainowner?domain=example.com&ownerkey=8719425131715293085
- kuschku 9y ago
  Much easier — allow people to put a public key into the WHOIS database.
  - cthalupa 9y ago
    How does either solve this if the bad actor is able to inject routes at will? Just redirect whois.iana.org to their servers. Then they can prevent you from accurately determining who the registrar is in the first place.
    - kuschku 9y ago
      Whois database results can be signed and are signed for some TLDs.
      - cthalupa 9y ago
        Where is that information on whether or not you should expect a TLD to be signed stored? Is it on the client side? If a client receives an unsigned whois response, what does it do? This isn't something I was aware of, and I'm not having much luck in finding out the implementation details.
        mehrdadn 9y ago
        > Where is that information on whether or not you should expect a TLD to be signed stored? Is it on the client side? If a client receives an unsigned whois response, what does it do? I can imagine a zillion different approaches... most obvious (not necessarily the best) one being to ask an IANA server over a normal TLS connection whether it should expect a TLD's records to be signed. And you can obviously cache that response for a while. Remember the point here is to validate domain ownership, which everybody already understands to be a big deal. If you can't get any trustable response from anybody, then I would expect it is your duty to refrain from issuing a certificate for that domain.
deleted 9y ago
[deleted]
return1 9y ago
i wonder why they dont mention a blockchain solution to dns problems.
- jgrahamc 9y ago
  Sure, let's use blockchain for everything. I mean, why not? It's clearly the perfect solution to every problem that has ever occurred in any distributed system.
  - return1 9y ago
    how is it not a better solution in this case?
    - jgrahamc 9y ago
      What DNS problem are you solving with blockchain and how?
      - return1 9y ago
        security, like DNSSEC does. but only 1% of domains use DNSSEC. there are other advantages, like uncensorability, direct access to dns records, disadvantage is speed I am not advocating "blockchain for every problem", but name resolution is a good fit - if only there were blockchains in the 80s. there are already implementations: https://medium.com/@jgm.orinoco/ethdns-an-ethereum-backend-for-the-domain-name-system-d52dabd904b3 https://medium.com/@jgm.orinoco/ethdns-an-ethereum-backend-f...
        jgrahamc 9y ago
        So, why not work for further DNSSEC deployment instead of coming up with a new solution? I don't see how blockchain makes things better.
        rlpb 9y ago
        > but only 1% of domains use DNSSEC. And what percentage of domains use "blockchain"?
SteveJS 9y ago
Not the meat of the issue, but when I see ‘what is BGP?’ I like the TLA expanded. https://en.m.wikipedia.org/wiki/Border_Gateway_Protocol https://en.m.wikipedia.org/wiki/Border_Gateway_Protocol
- jwilk 9y ago
  Non-mobile link: https://en.wikipedia.org/wiki/Border_Gateway_Protocol https://en.wikipedia.org/wiki/Border_Gateway_Protocol
- jgrahamc 9y ago
  We wrote in more detail about BGP in the past here: https://blog.cloudflare.com/why-google-went-offline-today-and-a-bit-about/ https://blog.cloudflare.com/why-google-went-offline-today-an...
lostctown 9y ago
An interesting aspect of this crypto boom over the last year has been the stress test on public internet infrastructure. Just the other day myetherwallet was hit with a nasty hack.
- mjmj 9y ago
  Which was?
  - deleted 9y ago
    [deleted]
  - deleted 9y ago
    [deleted]
piracy1 9y ago
It seems like they would have much better and more effective targets. They could have attacked bitcoin or crypto coin mining pools and just used the hashing power. It would have worked almost instantly and they wouldn't have to deal with SSL/TLS. Say a pool mines 20% of the blocks. After 2 hours they will have mined 2 blocks of 12.5 btc so 25 btc and $220K (although they would likly not get all the hashing power because not everyone is taking that route.)
- foepys 9y ago
  How could redirecting traffic of a Bitcoin mining pool make an attacker money? The transaction to which the block rewards goes is part of the block hash and thus still under full control of the miner that has the matching private key.
massaman_yams 9y ago
In a theoretical scenario like this where a hijacker used Let's Encrypt to receive a cert, rather than using a self-signed one, it's worth mentioning that there is a defense against that: HPKP - though much like HSTS, this is only effective for browsers that have previously visited the site. https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
- foepys 9y ago
  Chrome 67 will drop HPKP support.
- Diederich 9y ago
  Chrome, at least, is walking away from HPKP: https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/he9tr7p3rZ8/eNMwKPmUBAAJ https://groups.google.com/a/chromium.org/forum/#!msg/blink-d...

196 points by jgrahamc 9y ago

Wonder if services like Let’s Encrypt were affected. I imagine a scenario where a small hijack of DNS could allow for properly signed certificates for domains that are not owned. If I operated a CA service, I would carefully examine the requests received during this time frame. Maybe someone can audit the Transparency Logs during this period for anomalous activity.

dannyw 9y ago
BGP hijacks sadly happen on a weekly basis. It’s not new and there are hundreds more cases. It’s only noteworthy this time because high profile targets were affected instead of these being used for spear hijacking, etc.

thisisit 9y ago

> But during the hijack, it returned IPs associated with a Russian provider (AS48693 and AS41995). Curious what stopped cloudfare from naming these providers.

coolspot 9y ago
These are small providers with non recognized names. There is little point of naming them.
d215 9y ago
suit yourself: https://stat.ripe.net/AS48693#tabId=at-a-glance https://stat.ripe.net/AS48693#tabId=at-a-glance https://stat.ripe.net/AS41995#tabId=at-a-glance https://stat.ripe.net/AS41995#tabId=at-a-glance
- lostmsu 9y ago
  One of them has just a name: Barbarich Viacheslav Yuryevich
deleted 9y ago
[deleted]
jgrahamc 9y ago
Nothing. It's just that you'd probably never have heard of them.
cthalupa 9y ago
They did. To have an AS number you've gotta register it with a provider (In this case RIPE, as noted by d215). You can always look up an AS number and get ownership details.

isaack 9y ago

If I were the hacker: 1. I would get a proper SSL cert signed 2. Only hijack one of the ranges, and do not respond to other domains (causing SERVFAIL), so other domains will resolve unaffected, instead of causing a scene (see: https://www.reddit.com/r/sysadmin/comments/8ejrkk/google_dns_issues/ https://www.reddit.com/r/sysadmin/comments/8ejrkk/google_dns...)

cjbprime 9y ago
> 1. I would get a proper SSL cert signed Uh, how? Are you assuming that a single BGP leak would be enough to cause e.g. a letsencrypt misissuance for the domain? It sounds like (from other comments) they have a global round-robin resolver setup for their DNS challenges. > 2. Only hijack one of the ranges, and do not respond to other domains (causing SERVFAIL), so other domains will resolve unaffected I think exactly that's what they did. I saw people posting SERVFAILs during the outage.
- baby 9y ago
  > they have a global round-robin resolver setup for their DNS challenges no they don't
- geoah 9y ago
  > I think exactly that's what they did. I saw people posting SERVFAILs during the outage. Author is suggesting they should be properly resolving the rest of the domains so they wouldn't cause SERVFAILs.
- zyberzero 9y ago
  They do, and they don't. They have a round robin setup, for which resolver that validates the DNS challenge - however, they do not validate the DNS challenges from several resolvers[1]. So, if that particular resolver got caught in the BGP leak while doing a challenge verification you could get a valid cert. Lots of ifs and buts - but it is certainly possible. [1] https://news.ycombinator.com/item?id=16918382 https://news.ycombinator.com/item?id=16918382

vertex-four 9y ago

So what’s happening with the deployment of RPKI[0]? It would seem that owners of IP address space would have the incentive to use it - but perhaps routers don’t implement BGPSec widely enough? [0] https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastruc...

vbernat 9y ago
RPKI doesn't really solve the problem as it is a proof of origin. You can still inject routes, just ensure you add the original AS as origin of these routes. BGPsec would solve this issue, but is currently not deployed at all (and won't be before a long time).

edf13 9y ago

Another excellent write-up from cloudflare of a major issue... good job!

bluesign 9y ago

I find it funny how the article concludes basically almost everyone is responsible. And probably nobody will be hold responsible for this other than hacker. Thats why noone in this chain will have any incentive to fix this ever occuring again.

thesimon 9y ago

>A few things can be done on the upper levels of the network stack. On DNS, you can use DNSSEC to sign your records. The IPs returned by the fake DNS would not have been signed as they do not have the private keys. Would this have helped though? The attacker's DNS resolver could just return records without DNSSEC.

boudin 9y ago
With DNSSEC, the public key is not hosted on the zone iself, where your zone delegation is hosted. It's the DS records : https://uk.godaddy.com/help/what-is-a-ds-record-6148 https://uk.godaddy.com/help/what-is-a-ds-record-6148
- cthalupa 9y ago
  I don't see how this really solve this attack. Like, it solves the specific way they did it - standing up their own DNS server on the IPs they've started advertising - but really, all they have to do is announce the IP(s) that are in the DNSSEC signed records. Then you're still getting the traffic, and nothing has thrown any red flags on the DNSSEC side because nothing has changed as far as DNS goes. The problem is that when you can hijack the routing for an IP there's nothing that can realistically be done with how the internet works today to protect again it. Basically every system of validation requires you have access to some outside authority that can verify authenticity, and if you can announce the route to those IPs to your own infrastructure, you can fool the validation.
  - boudin 9y ago
    If i'm not mistaken, in this scenario they only hijacked the dns traffic and were using it to resolve a domain to another IP, so they were changing the DNS zone. DNSSEC would have prevented this as the change they did in the zone needs to be signed, so it would require them to also get the private key used to sign the zone. It fails to solve the issue if the dns resolver doesn't check DNSSEC signatures though. If they managed to hijack the route to the server itself, there's no need to hijack DNS at all, so it's a different issue that can't be solved at DNS level itself as DNS is not involved in the attack

mehrdadn 9y ago

Taking a step back here... shouldn't proving ownership of a domain involve the domain registrar some way, rather than involving whoever happens to host your DNS? They know who you are, so maybe they can provide an interface to let you prove to others that you own the domain. Like I dunno, maybe just let the owner of the domain generate a unique one-time key valid for 30 minutes, and provide an API to allow anybody to check ownership if given that key, e.g. at a standard URL like yourregistrar.com/domainowner?domain=example.com&ownerkey=8719425131715293085

kuschku 9y ago
Much easier — allow people to put a public key into the WHOIS database.
- cthalupa 9y ago
  How does either solve this if the bad actor is able to inject routes at will? Just redirect whois.iana.org to their servers. Then they can prevent you from accurately determining who the registrar is in the first place.
  - kuschku 9y ago
    Whois database results can be signed and are signed for some TLDs.
    - cthalupa 9y ago
      Where is that information on whether or not you should expect a TLD to be signed stored? Is it on the client side? If a client receives an unsigned whois response, what does it do? This isn't something I was aware of, and I'm not having much luck in finding out the implementation details.
      - mehrdadn 9y ago
        > Where is that information on whether or not you should expect a TLD to be signed stored? Is it on the client side? If a client receives an unsigned whois response, what does it do? I can imagine a zillion different approaches... most obvious (not necessarily the best) one being to ask an IANA server over a normal TLS connection whether it should expect a TLD's records to be signed. And you can obviously cache that response for a while. Remember the point here is to validate domain ownership, which everybody already understands to be a big deal. If you can't get any trustable response from anybody, then I would expect it is your duty to refrain from issuing a certificate for that domain.

deleted 9y ago

[deleted]

return1 9y ago

i wonder why they dont mention a blockchain solution to dns problems.

jgrahamc 9y ago
Sure, let's use blockchain for everything. I mean, why not? It's clearly the perfect solution to every problem that has ever occurred in any distributed system.
- return1 9y ago
  how is it not a better solution in this case?
  - jgrahamc 9y ago
    What DNS problem are you solving with blockchain and how?
    - return1 9y ago
      security, like DNSSEC does. but only 1% of domains use DNSSEC. there are other advantages, like uncensorability, direct access to dns records, disadvantage is speed I am not advocating "blockchain for every problem", but name resolution is a good fit - if only there were blockchains in the 80s. there are already implementations: https://medium.com/@jgm.orinoco/ethdns-an-ethereum-backend-for-the-domain-name-system-d52dabd904b3 https://medium.com/@jgm.orinoco/ethdns-an-ethereum-backend-f...
      - jgrahamc 9y ago
        So, why not work for further DNSSEC deployment instead of coming up with a new solution? I don't see how blockchain makes things better.
      - rlpb 9y ago
        > but only 1% of domains use DNSSEC. And what percentage of domains use "blockchain"?

SteveJS 9y ago

Not the meat of the issue, but when I see ‘what is BGP?’ I like the TLA expanded. https://en.m.wikipedia.org/wiki/Border_Gateway_Protocol https://en.m.wikipedia.org/wiki/Border_Gateway_Protocol

jwilk 9y ago
Non-mobile link: https://en.wikipedia.org/wiki/Border_Gateway_Protocol https://en.wikipedia.org/wiki/Border_Gateway_Protocol
jgrahamc 9y ago
We wrote in more detail about BGP in the past here: https://blog.cloudflare.com/why-google-went-offline-today-and-a-bit-about/ https://blog.cloudflare.com/why-google-went-offline-today-an...

lostctown 9y ago

An interesting aspect of this crypto boom over the last year has been the stress test on public internet infrastructure. Just the other day myetherwallet was hit with a nasty hack.

mjmj 9y ago
Which was?
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]

piracy1 9y ago

It seems like they would have much better and more effective targets. They could have attacked bitcoin or crypto coin mining pools and just used the hashing power. It would have worked almost instantly and they wouldn't have to deal with SSL/TLS. Say a pool mines 20% of the blocks. After 2 hours they will have mined 2 blocks of 12.5 btc so 25 btc and $220K (although they would likly not get all the hashing power because not everyone is taking that route.)

foepys 9y ago
How could redirecting traffic of a Bitcoin mining pool make an attacker money? The transaction to which the block rewards goes is part of the block hash and thus still under full control of the miner that has the matching private key.

massaman_yams 9y ago

In a theoretical scenario like this where a hijacker used Let's Encrypt to receive a cert, rather than using a self-signed one, it's worth mentioning that there is a defense against that: HPKP - though much like HSTS, this is only effective for browsers that have previously visited the site. https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning

foepys 9y ago
Chrome 67 will drop HPKP support.
Diederich 9y ago
Chrome, at least, is walking away from HPKP: https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/he9tr7p3rZ8/eNMwKPmUBAAJ https://groups.google.com/a/chromium.org/forum/#!msg/blink-d...