5 ms·

I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own o

I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks. Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.

netfortius 3y ago
I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.
tw04 3y ago
>I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks. I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it". Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
- gnfargbl 3y ago
  > I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately. > backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space. Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
  - tw04 3y ago
    >The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. So you've never worked in a corporate environment. Here's how that conversation would go: Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions. OK, so drop their feed and block them? Great, done. >Once a vulnerability is known then someone out there is going to start exploiting it almost immediately. And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false. >Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.
    - gnfargbl 3y ago
      I am aware that network operators have behaved this way for very many years. The point I am making is that all of the IT industry used to attempt to "work around" security vulnerabilities in the same way, until log4shell and all the others gradually beat that propensity out of them. I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.
      - devonkim 3y ago
        Software ecosystems like libraries and frameworks have completely different propagation and remediation mechanics compared to federated systems like core Internet backbone routers and switches is the thing. Try as we might conceptualize otherwise the modern Internet from a packet’s purview is more like a loose confederation of ultimately privatized or state-run fiefdoms than a cellular automata digraph explosion. So actors that try to act maliciously against the network will be basically shut out given the rule of an iron fist being the default.
hn8305823 3y ago
> I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor. The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.
ketralnis 3y ago
I practised picking locks on your house and broke the lock because of a poor lock design. Yeah okay the lock sucks. But I probably shouldn't have been doing it on your house in the first place and now you've got a broken lock. Who are you mad at? A researcher could have bought their own copy of the lock and picked it there. Or even borrowed yours with some notice, or let you know so you could have them use an interior door instead, or any number of other things that didn't break your actual house.
- timnetworks 3y ago
  "I can get into this lock" will not have the same effect or call to action as "I can get into your doors" but it admittedly has to be aimed at the landlords, not the doormen.
- zajio1am 3y ago
  That is not really a matching comparison. With BGP, there is kind of public space (global BGP tables) and participants are free to attach valid attributes to their routes. From a cursory look, these researchers used announcement that was compliant with BGP standards (in contrast to Ben Cartwright-Cox from this article, who used fuzzed invalid attributes), so they did something that was generally acceptable behavior for BGP participants.
patmorgan23 3y ago
Rule number 1 of the internet is don't break other people's shit. You can do whatever you want on your own equipment but don't break other people.

netfortius 3y ago

I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.

tw04 3y ago

>I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks. I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it". Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.

gnfargbl 3y ago
> I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately. > backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space. Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
- tw04 3y ago
  >The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. So you've never worked in a corporate environment. Here's how that conversation would go: Hey guys, some researchers found out if you use a test flag meant for the lab on the public internet, it breaks all our BGP sessions. OK, so drop their feed and block them? Great, done. >Once a vulnerability is known then someone out there is going to start exploiting it almost immediately. And yet the "vulnerability" in question was known, and was not being immediately exploited if you read through the mailing list or were participating in NANOG at that point in time. So your statement is provably false. >Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. And yet we're having this conversation in 2023, they have operated the same way for 40+ years, and somehow the internet is still working. Bad actors get blackholed, it worked in the past, it'll continue working in the future. The reality is that backbone routing is expensive, and expecting everyone to update their kit on YOUR timeline isn't reasonable.
  - gnfargbl 3y ago
    I am aware that network operators have behaved this way for very many years. The point I am making is that all of the IT industry used to attempt to "work around" security vulnerabilities in the same way, until log4shell and all the others gradually beat that propensity out of them. I am prophesying that a similar reckoning is likely to come upon the network backbone. You're arguing that the cost of entry to the game (BGP peering) is such that the old ways will continue to work. Let's hope you're right.
    - devonkim 3y ago
      Software ecosystems like libraries and frameworks have completely different propagation and remediation mechanics compared to federated systems like core Internet backbone routers and switches is the thing. Try as we might conceptualize otherwise the modern Internet from a packet’s purview is more like a loose confederation of ultimately privatized or state-run fiefdoms than a cellular automata digraph explosion. So actors that try to act maliciously against the network will be basically shut out given the rule of an iron fist being the default.

hn8305823 3y ago

> I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks We call out vendors when their quality suffers but you only have two or three practical vendor options for high end routers depending on your use case/feature requirements. All three have varying cycles of hardware and software quality, with hardware and software rarely being in phase for the same vendor. The fact that all three vendors make most of their money on non-network focused enterprises with less sophisticated engineers doesn't help get development attention where it needs to be for global infrastructure.

ketralnis 3y ago

I practised picking locks on your house and broke the lock because of a poor lock design. Yeah okay the lock sucks. But I probably shouldn't have been doing it on your house in the first place and now you've got a broken lock. Who are you mad at? A researcher could have bought their own copy of the lock and picked it there. Or even borrowed yours with some notice, or let you know so you could have them use an interior door instead, or any number of other things that didn't break your actual house.

timnetworks 3y ago
"I can get into this lock" will not have the same effect or call to action as "I can get into your doors" but it admittedly has to be aimed at the landlords, not the doormen.
zajio1am 3y ago
That is not really a matching comparison. With BGP, there is kind of public space (global BGP tables) and participants are free to attach valid attributes to their routes. From a cursory look, these researchers used announcement that was compliant with BGP standards (in contrast to Ben Cartwright-Cox from this article, who used fuzzed invalid attributes), so they did something that was generally acceptable behavior for BGP participants.

patmorgan23 3y ago

Rule number 1 of the internet is don't break other people's shit. You can do whatever you want on your own equipment but don't break other people.