12 ms·

Not surprised. A few years ago a team of researchers tried to use a feature attribute flag in BGP. Their experiment caused an important BGP software to crash.

• networkchad 3y ago
  [dead]
• deleted 3y ago
  [deleted]
• dogleash 3y ago
  > I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers. I suspect a Network Operators mailing list is going to be full of people trying to keep production up, with the tools they've got, rather than the tools they'd like. Thinking the software quality on the routers they're using are dogshit is probably just called "Monday" and thinking their peers' routers are even worse is called "Tuesday". I can understand they don't appreciate "research" done on production systems that might cause issues when they already know their equipment sucks.
  • themerone 3y ago
    Network operators are also the ones that pop into IPV6 discussions to voice their displeasure with having support another protocol.
    • booi 3y ago
      To be fair… supporting dual stack really does suck
      • yjftsjthsd-h 3y ago
        A lot about networking sucks, but that's what people are paying them for.
    • philg_jr 3y ago
      Also to be fair, IME there seems to be a lot more vocal pro-IPv6 people on the NANOG list than not.
    • kortilla 3y ago
      Not really, ISPs (who are on nanog) tend to like ipv6. The ones who complain are sysadmins and network operators of campus/corporate networks.
      • icedchai 3y ago
        They've only had over a decade to get their act together. I'm being generous, seeing IPv6 is older than the original IPv4 internet was when it went mainstream.
  • gnfargbl 3y ago
    I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks. Bitching at researchers (note: not "researchers") who are finding and calling out those flaws before someone really bad comes along and exploits them is somewhere between unhelpful and negligent. Internet security is always improved by more sunlight.
    • netfortius 3y ago
      I suppose you have never seen, ever, in your professional career, the "C" suits making business deals and thus decisions on what's to be acquired, with big network/server/storage/database/software vendors, over the head and against the advice of those then called to support crap in production.
    • tw04 3y ago
      >I can't understand it. The network operators should save their vitriol for the vendors who are selling them equipment that sucks, and for whoever in their own organisation is continuing to purchase equipment that sucks. I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. When "fixing" a router means you break half the internet, you just don't fix it. This isn't some walled garden where you can just say "tough shit if you broke, fix it". Most of the "issues" these researchers find are well known, but backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space.
      • gnfargbl 3y ago
        > I think you fail to realize that the internet is a bunch of disparate networks that no one person controls. The internet is a bunch of disparate networks that no one person controls, exactly. That means that when someone tells you there's a fault in the kit that you have installed in your network, then that is 100% your problem and it is 100% on you to get it fixed. Once a vulnerability is known then someone out there is going to start exploiting it almost immediately. > backbone operators are more militant than Linus when it comes to: rule #1 is don't break user space. Backbone operators can have all the hubris they want, but it won't change the reality that the only effective action they can take when a vulnerability is found is to get it fixed ASAP. This is a security lesson that has been learnt in recent years by many sectors of the IT industry and, judging by the woeful response that Ben Cartwright-Cox describes in his post, it's one that the network backbone is going to be learning soon.
  • loa_in_ 3y ago
    Except there's no "staging Internet" to test such equipment, and even if there was it would still sometimes not translate to the real open network. I'm sorry, were they inconvinienced with the world at large?
• throw0101c 3y ago
  > I felt back then that this community had a very unhealthy relationship with the quality of their software and the work of security researchers. Or they felt like some of us feel about Tesla/Musk unleashing cars with "Full" "Self-Driving" on public roads.
  • adra 3y ago
    As lousy as these systems are, they kill a hell of a lot fewer people (for now) than drunks, so net positive for the time being.
    • mschuster91 3y ago
      The solution to drunk drivers is public transport, not self-driving cars.
      • zaroth 3y ago
        It’s 2023 and driving kills like 40,000 people a year. We’ve been building trillions of dollars of public transit since about the same time as we started building the National highway system. So when exactly does public transit start saving all these lives? A new train from SF to LA has been in the works since 1996. They are expecting partial completion for $30 billion by 2030 which is projected to carry ~15 million passengers * ~400 miles = 6 billion passenger miles per year. The fatality rate for passenger cars is ~0.57 per 100 million miles, so eliminating 6 billion passenger miles could be reasonably expected to save about 60 * 0.57 = 34 lives a year out of the ~40,000 that are dying on the roads annually. This is just rough napkin math showing the scale of the problem would require a roughly $30 trillion investment in public transit to appreciably shift passenger-miles away from passenger cars. Public transit can be very useful. But it’s clearly not the solution to eliminating driving deaths.
        • autoexec 3y ago
          It seems unfair to say that public transport doesn't have a role in reducing deaths caused by drunk drivers just because the hugely flawed and half-assed public transport systems in our cities (which were designed to make public transport ineffective) haven't yet delivered on that promise. In places where cities and public transportation systems are well designed and useful people do use them and those places see far fewer deaths from drunk drivers than the US does as a result. You argue that it would be very expensive to fix the problems we've made for ourselves in order to get useful public transportation and that's certainly true. It's also very expensive to invent self-driving cars and part that cost has already been paid with lives lost and human suffering, but that hasn't stopped us from developing those systems. We could decide to invest instead in improving our cities and infrastructure. We just choose to let obscenely rich people kill innocent Americans while they play with their profit generating spying car tech instead.
• capableweb 3y ago
  Is there a reason why the researchers did this on live production BGP network rather than in a test environment, running their own BGP and FRR routers? Seems a bit haphazardly, as doing so first seems to have uncovered the issue. Overall I agree with your message though, and of course things shouldn't stop working just because you use attributes reserved for development.
  • LegionMammal978 3y ago
    The researchers claimed that they had first tested it with a few different vendors' routers without any issues, but they had not included FRR in those tests [0]. There was a bit of discussion on the mailing list regarding just how many BGP implementations (and configurations) a responsible researcher is obligated to test with before exposing something new on the public Internet. [0] https://mailman.nanog.org/pipermail/nanog/2019-January/098765.html https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
    • salawat 3y ago
      The correct answer is "All of them on your own network first". Thou shalt not blow up another's network knowingly. Blowing up another's network because you didn't test your theory in your own isolated lab space first counts as blowing up another's network knowingly. Because what the hell else did you think would happen? Cripes.
• ExoticPearTree 3y ago
  The sane thing to do is to drop that particular announcement with that invalid attribute or discard the invalid information. But some people tend to be more catholic than the pope. Dropping the whole session, especially in a DFZ scenario is really bad. The churn that you generate is immense for all your neighbors.
  • fanf2 3y ago
    As the fine article explains, that is what RFC 7606 specifies.
    • ExoticPearTree 3y ago
      Yeah, but I don't think the people who created that RFC thought about all the bad side effects their advice causes. BGP is one of the most robust protocols out there: there's no state (as in protocol state), you can treat each message independently, you can extend it super easy and so on. And because you can treat every message independently, you can have a really secure and stable mechanism to deal with any badness that might happen.
  • crest 3y ago
    Just have a bunch of fuzzers running 24/7 to keep all implementations honest.
    • clankyclanker 3y ago
      That’s what Google used to do (though not for BGP), back in their engineering days. That’s what gave us AFL.
      • tialaramex 3y ago
        And indeed GREASE the TLS feature where you just propose nonsense because every other party should go "Er, no I don't speak nonsense?" to prevent ossification also comes from Google engineers. GREASE means that when you build your "advanced" security device which "protects" customers by treating everything different as hostile, it won't work, so you'll need to tweak it to at least just ignore such differences as irrelevant, which is enough that we can come back later and intentionally improve things. Previously, without GREASE, we'd have to guess what oversights we could exploit to avoid this "protection" to deliver protocol improvements, if we guessed wrong nothing works, or everybody's security is broken, sometimes both. e.g. for TLS 1.2 the oversight we found is, if you're resuming an existing session the "security" products just wave that through because otherwise they break people's real workflows. So in TLS 1.3 protocol as actually spoken essentially an initial connection goes like this: Client: Hi some.dns.name.example I'm a TLS 1.2 Client, I'd like to resume our previous conversation #randomNonsense. Also, completely unrelated, I happen to speak FlyCasualThisIsReallyTLSv1.3 and so I have these TLSv1.3KeyAgreementParameters. then either: TLS 1.3 Server: Hi Client, of course, let's continue from there. [ whereupon everything further is encrypted because this is actually TLS 1.3, but to a dumb middlebox it makes sense that they're just resuming a prior encrypted conversation #randomNonsense which it doesn't remember ] OR TLS 1.2 Server: Er, no I don't remember any such conversation and I don't know FlyCasualThisIsReallyTLSv1.3 so let's start a fresh conversation as normal.
• hinkley 3y ago
  Recipe as I got it from grandma 1) Beleaguered people misbehave. 2) You are usually the architect of your own destruction. 3) It’s easier to see other people’s problems than your own. Stir into a death spiral and let it rest for six to twelve months. Bake at 375° for two years, then serve cold with a sprinkling of schadenfreude to offset the bitterness.
• braiamp 3y ago
  I don't think that is an accurate representation of the "comunity", it was a "singular complaint from a company advertising unallocated ASN and IPv4 resources", and got told off by literally most of the mailing list. I'm counting at least 20 other individual responses and all of them are in support of the experiment https://mailman.nanog.org/pipermail/nanog/2019-January/thread.html#99143 https://mailman.nanog.org/pipermail/nanog/2019-January/threa...
• tw04 3y ago
  If you read the links: they discovered that using this flag broke EVERY installation of FRR on the internet. They let the developers know. And then said they were going to test again in a week. Does any sane person think that 1 week is enough time to both notify every user of FRR in the WORLD, as well as ensure they have upgraded their installations for a "bug" that only is triggered by someone using experimental flags meant for testing purposes only? Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy.
  • LegionMammal978 3y ago
    > Mind you, these researchers didn't test ANYTHING in a lab environment first "because it's a lot of work to test all the open source BGP implementations in the wild". AKA: I'm lazy. That's a gross exaggeration: they said that they first tried it "in a controlled environment with different versions of Cisco, BIRD, and Quagga routers" [0] without issue. Presumably, the problem we know about would have been avoided if they had happened to include FRR in their test suite beforehand. But that raises the next question, exactly how many implementations does one need to test before one is no longer "lazy"? [0] https://mailman.nanog.org/pipermail/nanog/2019-January/098765.html https://mailman.nanog.org/pipermail/nanog/2019-January/09876...
• xyst 3y ago
  To be honest, if you are a responsible researcher you are NOT running “experiments” in live environments that have potential to impact multiple regions. The outrage here is understandable. I would be pissed if I was paged because halfway across the world, some asshole caused this intentionally. Also this mailing list is aimed at the operators, not the vendors. Different communities. The vendor response here, I do agree is irresponsible. Besides OpenBSD, every one here just sucked.
  • pixl97 3y ago
    Ya the researcher could just sell their findings to whatever blackhat groups are out there instead. You generally get less pushback and more cash from them.
• kjartann 3y ago
  Two thoughts: - I think it's OK not to want experiments like these to be conducted connected to the global internet and rather in a lab such as the one benjojo used. - NANOG is known to be a bit of a joke within the networking space, not too long ago someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something to that effect, so don't always take them too seriously. I think the first applies here.
  • njt 3y ago
    > someone complained about NTP issues with traffic passing through TATA and they were told to make their devices stratum 0 or something Not saying your summary of that thread is wrong, I just wanted to expand a bit to maybe clarify: - someone posted to NANOG about issues with their (European) servers reaching 0.freebsd.pool.ntp.org - they noticed they were being routed through Tata (AS6453) - they mentioned that their other servers (e.g. in Africa) had no issues - they posted to NANOG to ask about this - they thought it was a routing or FreeBSD issue (they later posted about the issue to a FreeBSD list as well) So AFAICT, the correct response was posted about a day and a half (~42 hours) later: - the NTP pool uses GeoDNS to map clients to servers - the OP's server IP address was incorrectly geolocated to Mauritius - the Mauritius zone in the pool has only one server - so the OP would only ever get that IP as an answer The short-term fix was to use a different pool address (e.g. europe.pool.ntp.org), and the responder also mentioned that the NTP pool folks were working to fix the problem. Full thread here: https://mailman.nanog.org/pipermail/nanog/2023-August/222706.html Now the interesting part of that thread was that shortly after (~4 hours) the OP posted about their problem, someone replied and recommended that they setup their own GPS-based NTP network. From there, that sub-thread was mostly about NTP attacks and GPS security and mounting receivers on top of datacenter buildings and such. Counting up the messages (if my mail client's threading capabilities are to be trusted), of the 94 or so total messages in the thread, 79 (84%) of the messages were in that GPS-related subthread, and only 15 messages responded to the OPs issue directly. I didn't really see this as a failing of NANOG per-se, it's just a public mailing list and some folks went on a nerd sniping tangent (cf. XKCD #356).