8 ms·

Large-Scale BGP Hijack

161 points by Ansil849 7y ago

ac29 7y ago
More here: https://www.manrs.org/2020/04/not-just-another-bgp-hijack/ https://www.manrs.org/2020/04/not-just-another-bgp-hijack/
booi 7y ago
It’s crazy to me that the fundamental protocol running the entire internet is essentially based on the honor system... it is not working out well
- Icathian 7y ago
  Border Gateway Protocol is just one routing protocol, calling it "the fundamental protocol" is stretching a point. That award probably goes to TCP. Maybe IPv4.
  - kortilla 7y ago
    Border gateway protocol is the only routing protocol to get prefixes reachable to the wider internet.
  - tlb 7y ago
    IPv4 and v6 run on the honor system too, in that the sender can supply any source address they like. TCP connection setup is the lowest layer that makes it nontrivial to spoof a sender.
    - Groxx 7y ago
      The ability to do this is effectively what enables DNS traffic amplification attacks (and others). Trust has been broadly effective, but also quite damaging at times. It's part of why I'm glad https is nigh universal - http's trust allowed stuff like Firesheep and all the hotel shit-injection that used to be so prevalent.
  - mox1 7y ago
    I mean no BGP no global internet...just local lans and wans.
    - iso1210 7y ago
      BGP didn't start major inroads into the internet until the mid 90s. RIP can't cope with the scale of the internet today, but to say BGP is essential to interconnecting networks is not right
      - jlgaddis 7y ago
        Huh? If you take away BGP, there is no Internet! No other protocol in existence/use today can scale the way BGP has.
- danharaj 7y ago
  All things considered it is working out pretty well.
- tialaramex 7y ago
  The world's financial systems depend heavily on trust too. Take credit cards. As an end user you perceive some fairly heavyweight albeit imperfect security, but in reality there are two parallel systems and only one of them is using those security features. Authorisation is the part that ties your good name to your transactions. The bank doesn't trust you, so considerable technological effort is used in Authorisation. Chip cards, PINs, extra passwords and codes for web transactions, that's all Authorization. Settlement moves money between your account and a merchant account. This isn't secured at all, it's just based on trust. Any merchant can assert that you used your card to pay them any amount and by default the bank takes it on trust. Authorisation is abuse resistant. If you try to replay the same authorization multiple times it doesn't work. Settlement doesn't care. If a worker messes up and puts a big merchant's daily settlement transactions through twice, every single one of those transactions just happens again. Paid $80 for groceries leaving $38 in the account until pay day? Sorry, now you're $42 overdrawn. This really happens. If you complain they'll fix it. If hundreds of customers complain they'll realize the error and fix it for everyone. But this is why you have to check your credit card bill. Sooner or later there will be errors, and if you don't check you'll never see them which can get expensive fast.
  - dx034 7y ago
    Financial institutions are much more heavily regulated by governments than ISPs. The honor system in finance only works because regulators ensure (today more so than a few decades ago) that all players fulfill minimum requirements.
  - buran77 7y ago
    On the other hand see how double spending works in less regulated financial markets like cryptocurrency. What keeps that honor system working in financial systems is that it's backed by strong regulation and strong supervisory authorities. There are so many controls built into this that mistakes happen less often than one might think, considering the volume of non-bank transactions (think billions++, yearly). ISPs have noting of the kind. One routing mistake (or "mistake") and all of your customers' data suddenly went through China, Russia, US (you name it), for better or worse. What's worse, in a manner of speaking that data is now devalued. There's value date equivalent in the ISP's world. As a customer you have no avenues, the thing isn't even between you and any ISP you may work with. The bank will never get away with telling you "well we accidentally made this extra payment, it got routed thorough this extra bank, so now you lost 25% of the money".
- q3k 7y ago
  Mostly in the US, where filters (if any) are set up by mailing eachother deadtree LOAs about party A allowing party B to announce their prefix. It's basically the checkbook way of doing things, but for BGP. Meanwhile in EMEA we've been using IRR (and recently RPKI) for automating filters for years now. I couldn't do a BGP hijack through my EU upstreams even if I wanted to.
  - dylz 7y ago
    RPKI in ARIN space is questionably annoying to do. I still haven't been able to add anything to my new allocations. As a bonus, my US upstream doesn't care if I have RPKI/ROA/IRR - even if I do, they ask for dead tree LOA anyway. When I sent a dead tree to one of my EU upstreams out of habit they were like "what do I need this for?"
    - kitteh 7y ago
      ARINs RPKI APIs could use some love, that's for sure.
- icedchai 7y ago
  It's actually working out amazing well, seeing it has barely changed in over 25 years!
- commandlinefan 7y ago
  When the internet really started to become a big "thing" in the mid-90's and I started learning how it all worked I thought, "well, as soon as hackers realize how insecure this whole thing is, that'll be the end of that..."
- kitteh 7y ago
  Cogent and Level3 weren't filtering their own customer. This is what happens when large companies say prefix list updates / validation is too much of a burden and allow customers to run bareback BGP. That said, Cogent is going to be doing RPKI on customers soon so that's good. As for Level 3 the world wonders.
  - jlgaddis 7y ago
    Rostelecom (AS12389) is directly connected to Level3 (AS3356) and announced the hijacked routes to them but Level3 filtered them out. But... AS12389 announced them to Rascom (AS20764), who accepted them and sent them on to Cogent (AS174), who accepted them and propogated them on to Level3 (AS3356), who, unfortunately, then accepted them from Cogent. The AS path would've been much longer than if Level3 had accepted them directly from AS12389 so, at the least, the actual effects of the hijacking weren't as bad as they could have been.
    - kitteh 7y ago
      Problem is they were more specific prefixes for quite a few people and that trumps everything.
      - jlgaddis 7y ago
        Ahhh, I missed that somehow. Looks like it was just a /24 here and there, though, out of much shorter prefixes, at least.
Icathian 7y ago
I'm curious as to the real impact. Facebook has the means to mitigate this quite easily. I wonder what's gained from this, if anything.
- betaby 7y ago
  Not really, see some explanation there https://dyn.com/blog/iran-leaks-censorship-via-bgp-hijacks/ https://dyn.com/blog/iran-leaks-censorship-via-bgp-hijacks/ and there https://www.youtube.com/watch?v=IzLPKuAOe50 https://www.youtube.com/watch?v=IzLPKuAOe50
- lathiat 7y ago
  Why do you say Facebook has the means to mitigate this quite easily? That's not really true as I understand it. The problem is that these announcements are made by third parties, and accepted by fourth parties, both of which Facebook has no control over. While it's possible for Facebook to implement some controls over what routes THEY accept and therefor what routes Facebook sends traffic TO - it cannot control where others send traffic. If a third party advertises Facebooks routes to a fourth party, and they accept them, Facebook cannot do anything directly about that. And then those fourth parties accepting the routes will send traffic TO Facebook via the third party.
  - toast0 7y ago
    I didn't look to see how much of Facebook's ranges got hijacked. Assuming some got hijacked and others didn't (which is usually what happened), they could direct more traffic to ranges that weren't hijacked (assuming their DNS ranges didn't get hijacked). For the ranges where the hijacking wasn't at the /24 level, Facebook also has plenty of staff who could adjust to advertise more specific routes. All the big internet companies with ASNs should have that though; but the smaller places that probably got hijacked too may not. Disclosure: I used to work at WhatsApp, including while it was part of Facebook. Everything in this message is armchair routing policy though.
    - wbl 7y ago
betaby 7y ago
More details https://radar.qrator.net/blog/how_you_deal_with_route_leaks https://radar.qrator.net/blog/how_you_deal_with_route_leaks and some analysis https://www.ripe.net/ripe/mail/archives/routing-wg/2020-April/004083.html https://www.ripe.net/ripe/mail/archives/routing-wg/2020-Apri... if RPKI ROA could help https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/
- gruez 7y ago
  >The route leak was distributed quite well through Rascom (AS20764) , then Cogent (AS174) and in a couple of minutes through Level3 (AS3356) to the world. Is there a reason why these can't be filtered at the ISP level? eg. Cogent knows that Rascom shouldn't be announcing those prefixes, and refusing to route traffic to them unless there's manual verification?
  - wmf 7y ago
    This kind of manual override can also go wrong and cause traffic to be incorrectly dropped.
    - betaby 7y ago
      https://www.manrs.org/ https://www.manrs.org/ would disagree with your reasoning.
  - betaby 7y ago
    There are no good reasons. Filtering implies opex, capex and man power. Also some customers are anti-filtering, basically accept everything or we wold leave to another uplink who won't filter us. I would say ~30% of organisations do not maintain their IRR data properly, notably this https://www.apnic.net/get-ip/faqs/route-management/ https://www.apnic.net/get-ip/faqs/route-management/ and https://www.apnic.net/manage-ip/using-whois/guide/as-set/ https://www.apnic.net/manage-ip/using-whois/guide/as-set/ (same fotr ARIN, RIPE, AfriNIC)
  - wbl 7y ago
    They can be filtered via RPKI.
rdxm 7y ago
lol....shocking.....NOT!!!! When are people going to wake the f* up....
peterwwillis 7y ago
Oh look, another submission of a tweet. Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet. It would literally be more useful if you just screenshot it and posted that, as at least it would last longer, not to mention add detail.
- 0az 7y ago
  I don't really get the hate for tweets. > Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet. I find the tweet helpful. I didn't know about BGPmon, for instance. Screenshots, on the other hand... They're not accessible, and they don't provide easy links back to the author, nor access to the tweet's reply context. I find all of those valuable.
annoyingnoob 7y ago
This is very similar to the caller-id problem that plagues telephones. Our networks were built for trust that no longer exists. Our protocols need to catch up with the times.
krebsonsecurity 7y ago
This appears to be related: https://twitter.com/InternetIntel/status/1247100092575813635 https://twitter.com/InternetIntel/status/1247100092575813635 https://www.ripe.net/support/service-announcements/accidental-roa-deletion https://www.ripe.net/support/service-announcements/accidenta... Last week a large BGP routing leak (>20k prefixes) occurred in Russia briefly impacted internet traffic around the world. See the interactive 3D visualization of the incident here: https://map.internetintel.oracle.com/leaks#/id/20764_12389_1585768500 https://map.internetintel.oracle.com/leaks#/id/20764_12389_1...
- kitteh 7y ago
  It's the same event. BGPmon was just delayed by a few days.

161 points by Ansil849 7y ago

ac29 7y ago

More here: https://www.manrs.org/2020/04/not-just-another-bgp-hijack/ https://www.manrs.org/2020/04/not-just-another-bgp-hijack/

booi 7y ago

It’s crazy to me that the fundamental protocol running the entire internet is essentially based on the honor system... it is not working out well

Icathian 7y ago
Border Gateway Protocol is just one routing protocol, calling it "the fundamental protocol" is stretching a point. That award probably goes to TCP. Maybe IPv4.
- kortilla 7y ago
  Border gateway protocol is the only routing protocol to get prefixes reachable to the wider internet.
- tlb 7y ago
  IPv4 and v6 run on the honor system too, in that the sender can supply any source address they like. TCP connection setup is the lowest layer that makes it nontrivial to spoof a sender.
  - Groxx 7y ago
    The ability to do this is effectively what enables DNS traffic amplification attacks (and others). Trust has been broadly effective, but also quite damaging at times. It's part of why I'm glad https is nigh universal - http's trust allowed stuff like Firesheep and all the hotel shit-injection that used to be so prevalent.
- mox1 7y ago
  I mean no BGP no global internet...just local lans and wans.
  - iso1210 7y ago
    BGP didn't start major inroads into the internet until the mid 90s. RIP can't cope with the scale of the internet today, but to say BGP is essential to interconnecting networks is not right
    - jlgaddis 7y ago
      Huh? If you take away BGP, there is no Internet! No other protocol in existence/use today can scale the way BGP has.
danharaj 7y ago
All things considered it is working out pretty well.
tialaramex 7y ago
The world's financial systems depend heavily on trust too. Take credit cards. As an end user you perceive some fairly heavyweight albeit imperfect security, but in reality there are two parallel systems and only one of them is using those security features. Authorisation is the part that ties your good name to your transactions. The bank doesn't trust you, so considerable technological effort is used in Authorisation. Chip cards, PINs, extra passwords and codes for web transactions, that's all Authorization. Settlement moves money between your account and a merchant account. This isn't secured at all, it's just based on trust. Any merchant can assert that you used your card to pay them any amount and by default the bank takes it on trust. Authorisation is abuse resistant. If you try to replay the same authorization multiple times it doesn't work. Settlement doesn't care. If a worker messes up and puts a big merchant's daily settlement transactions through twice, every single one of those transactions just happens again. Paid $80 for groceries leaving $38 in the account until pay day? Sorry, now you're $42 overdrawn. This really happens. If you complain they'll fix it. If hundreds of customers complain they'll realize the error and fix it for everyone. But this is why you have to check your credit card bill. Sooner or later there will be errors, and if you don't check you'll never see them which can get expensive fast.
- dx034 7y ago
  Financial institutions are much more heavily regulated by governments than ISPs. The honor system in finance only works because regulators ensure (today more so than a few decades ago) that all players fulfill minimum requirements.
- buran77 7y ago
  On the other hand see how double spending works in less regulated financial markets like cryptocurrency. What keeps that honor system working in financial systems is that it's backed by strong regulation and strong supervisory authorities. There are so many controls built into this that mistakes happen less often than one might think, considering the volume of non-bank transactions (think billions++, yearly). ISPs have noting of the kind. One routing mistake (or "mistake") and all of your customers' data suddenly went through China, Russia, US (you name it), for better or worse. What's worse, in a manner of speaking that data is now devalued. There's value date equivalent in the ISP's world. As a customer you have no avenues, the thing isn't even between you and any ISP you may work with. The bank will never get away with telling you "well we accidentally made this extra payment, it got routed thorough this extra bank, so now you lost 25% of the money".
q3k 7y ago
Mostly in the US, where filters (if any) are set up by mailing eachother deadtree LOAs about party A allowing party B to announce their prefix. It's basically the checkbook way of doing things, but for BGP. Meanwhile in EMEA we've been using IRR (and recently RPKI) for automating filters for years now. I couldn't do a BGP hijack through my EU upstreams even if I wanted to.
- dylz 7y ago
  RPKI in ARIN space is questionably annoying to do. I still haven't been able to add anything to my new allocations. As a bonus, my US upstream doesn't care if I have RPKI/ROA/IRR - even if I do, they ask for dead tree LOA anyway. When I sent a dead tree to one of my EU upstreams out of habit they were like "what do I need this for?"
  - kitteh 7y ago
    ARINs RPKI APIs could use some love, that's for sure.
icedchai 7y ago
It's actually working out amazing well, seeing it has barely changed in over 25 years!
commandlinefan 7y ago
When the internet really started to become a big "thing" in the mid-90's and I started learning how it all worked I thought, "well, as soon as hackers realize how insecure this whole thing is, that'll be the end of that..."
kitteh 7y ago
Cogent and Level3 weren't filtering their own customer. This is what happens when large companies say prefix list updates / validation is too much of a burden and allow customers to run bareback BGP. That said, Cogent is going to be doing RPKI on customers soon so that's good. As for Level 3 the world wonders.
- jlgaddis 7y ago
  Rostelecom (AS12389) is directly connected to Level3 (AS3356) and announced the hijacked routes to them but Level3 filtered them out. But... AS12389 announced them to Rascom (AS20764), who accepted them and sent them on to Cogent (AS174), who accepted them and propogated them on to Level3 (AS3356), who, unfortunately, then accepted them from Cogent. The AS path would've been much longer than if Level3 had accepted them directly from AS12389 so, at the least, the actual effects of the hijacking weren't as bad as they could have been.
  - kitteh 7y ago
    Problem is they were more specific prefixes for quite a few people and that trumps everything.
    - jlgaddis 7y ago
      Ahhh, I missed that somehow. Looks like it was just a /24 here and there, though, out of much shorter prefixes, at least.

Icathian 7y ago

I'm curious as to the real impact. Facebook has the means to mitigate this quite easily. I wonder what's gained from this, if anything.

betaby 7y ago
Not really, see some explanation there https://dyn.com/blog/iran-leaks-censorship-via-bgp-hijacks/ https://dyn.com/blog/iran-leaks-censorship-via-bgp-hijacks/ and there https://www.youtube.com/watch?v=IzLPKuAOe50 https://www.youtube.com/watch?v=IzLPKuAOe50
lathiat 7y ago
Why do you say Facebook has the means to mitigate this quite easily? That's not really true as I understand it. The problem is that these announcements are made by third parties, and accepted by fourth parties, both of which Facebook has no control over. While it's possible for Facebook to implement some controls over what routes THEY accept and therefor what routes Facebook sends traffic TO - it cannot control where others send traffic. If a third party advertises Facebooks routes to a fourth party, and they accept them, Facebook cannot do anything directly about that. And then those fourth parties accepting the routes will send traffic TO Facebook via the third party.
- toast0 7y ago
  I didn't look to see how much of Facebook's ranges got hijacked. Assuming some got hijacked and others didn't (which is usually what happened), they could direct more traffic to ranges that weren't hijacked (assuming their DNS ranges didn't get hijacked). For the ranges where the hijacking wasn't at the /24 level, Facebook also has plenty of staff who could adjust to advertise more specific routes. All the big internet companies with ASNs should have that though; but the smaller places that probably got hijacked too may not. Disclosure: I used to work at WhatsApp, including while it was part of Facebook. Everything in this message is armchair routing policy though.
  - wbl 7y ago

betaby 7y ago

More details https://radar.qrator.net/blog/how_you_deal_with_route_leaks https://radar.qrator.net/blog/how_you_deal_with_route_leaks and some analysis https://www.ripe.net/ripe/mail/archives/routing-wg/2020-April/004083.html https://www.ripe.net/ripe/mail/archives/routing-wg/2020-Apri... if RPKI ROA could help https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/

gruez 7y ago
>The route leak was distributed quite well through Rascom (AS20764) , then Cogent (AS174) and in a couple of minutes through Level3 (AS3356) to the world. Is there a reason why these can't be filtered at the ISP level? eg. Cogent knows that Rascom shouldn't be announcing those prefixes, and refusing to route traffic to them unless there's manual verification?
- wmf 7y ago
  This kind of manual override can also go wrong and cause traffic to be incorrectly dropped.
  - betaby 7y ago
    https://www.manrs.org/ https://www.manrs.org/ would disagree with your reasoning.
- betaby 7y ago
  There are no good reasons. Filtering implies opex, capex and man power. Also some customers are anti-filtering, basically accept everything or we wold leave to another uplink who won't filter us. I would say ~30% of organisations do not maintain their IRR data properly, notably this https://www.apnic.net/get-ip/faqs/route-management/ https://www.apnic.net/get-ip/faqs/route-management/ and https://www.apnic.net/manage-ip/using-whois/guide/as-set/ https://www.apnic.net/manage-ip/using-whois/guide/as-set/ (same fotr ARIN, RIPE, AfriNIC)
- wbl 7y ago
  They can be filtered via RPKI.

rdxm 7y ago

lol....shocking.....NOT!!!! When are people going to wake the f* up....

peterwwillis 7y ago

Oh look, another submission of a tweet. Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet. It would literally be more useful if you just screenshot it and posted that, as at least it would last longer, not to mention add detail.

0az 7y ago
I don't really get the hate for tweets. > Probably won't be able to read the tweet in the future when it gets deleted, no real context or other information in the tweet. I find the tweet helpful. I didn't know about BGPmon, for instance. Screenshots, on the other hand... They're not accessible, and they don't provide easy links back to the author, nor access to the tweet's reply context. I find all of those valuable.

annoyingnoob 7y ago

This is very similar to the caller-id problem that plagues telephones. Our networks were built for trust that no longer exists. Our protocols need to catch up with the times.

krebsonsecurity 7y ago

This appears to be related: https://twitter.com/InternetIntel/status/1247100092575813635 https://twitter.com/InternetIntel/status/1247100092575813635 https://www.ripe.net/support/service-announcements/accidental-roa-deletion https://www.ripe.net/support/service-announcements/accidenta... Last week a large BGP routing leak (>20k prefixes) occurred in Russia briefly impacted internet traffic around the world. See the interactive 3D visualization of the incident here: https://map.internetintel.oracle.com/leaks#/id/20764_12389_1585768500 https://map.internetintel.oracle.com/leaks#/id/20764_12389_1...

kitteh 7y ago
It's the same event. BGPmon was just delayed by a few days.