12 ms·
The internet is held together with spit and baling wire
- pjf 5y agoCome on, who in 2021 actively drops routes because of an entry in the Level3 IRR? I mean, IMHO, it's not close to impacting the whole Internet.
- FredPret 5y agoEvery complex system - including the human body, the global economy, the ecosystem, and large software systems - grew one functional hack at a time
- quickthrower2 5y agoCould the -MAIL-FROM requests somehow be tunneled through a VPN so that they are only accepted through that VPN and not the wider internet?
- solotronics 5y agoSome of this has to do with IPv4 and massive carrier NAT schemes. If technical people could get an ASN easier and route with each other on IPv6 it theoretically could provide backup paths for neighborhood level traffic. I guess the transit/ISP would have to enforce something like BGPsec with these little peers to prevent mistakes or malicious ASNs.
- a-dub 5y agoreminds me of the old email interface run by network solutions when they were the sole registrar for everything that wasn't government.
- HPsquared 5y agoHeld together with spit and baling wire as it is, the fact that it mostly works proves that the overall architecture is robust.
- IncRnd 5y ago> proves that the overall architecture is robust. Not really. What it shows is the stark difference between two ideologies. The first camp contains people who believe in Postel's Law, "be conservative in what you do, be liberal in what you accept from others". The second camp has people who recognize that the current world is not a cooperative network of researchers: "all input is untrusted". Krebs is absolutely in the second camp.
- shadowgovt 5y agoBut the two philosophies aren't really in contention. Proper adherence to Postel's law also includes accepting malicious traffic (and then doing something reasonable with it, like black-holing it). The "liberal in what you accept" part is mostly honest acceptance of the reality of the network: you cannot control the information sent to your service, only how you respond to it.
- deleted 5y ago[deleted]
- inopinatus 5y agoThese are not ideologies, but compatible principles from separate concerns. One relates to protocol errors, the other to information validation & verification.
- deleted 5y ago[deleted]
- azeirah 5y agoIf Alan Kay is to be believed (which I hope he is!) then the internet was originally inspired by multicellular lifeforms. I'd say the internet has some sort of a "biological" architecture. Robust in the sense that organisms are robust; extremely messy, sensical from a high-level view, chaotic from a low-level view.
- ragingrobot 5y agoO'Brien: I'm afraid to touch anything. It's all cross-circuited and patched together - I can't make head nor tails of it. Bashir: Sounds like one of your repair jobs. Seriously though, it seems like every form of infrastructure we rely on is held together in such a fragile manner. I hate hate to think of the chaos should there be a major Internet and physical infra failure in close proximity, time-wise.
- hypertele-Xii 5y agoWe have a pretty good track record keeping it running though; The Internet has never gone down! I'd be far more concerned with agricultural logistics, though we've never starved to extinction, either. Perhaps we've reached a point of positive no return: We can no longer cease to exist!
- adventured 5y agoIn this context it's a very important point that there isn't one Internet and there isn't one "we." Unless you strictly mean the human race has yet to go extinct due to starvation. A lot of starvation events have of course occurred across many different nations, peoples, civilizations. Europe, as one example, was numerous times ravaged by extreme starvation events that collectively killed millions of people across the 19th and 20th centuries. I think your agriculture concerns are well placed. The various Internets have gone down routinely for all sorts of reasons.
- deleted 5y ago[deleted]
- setpatchaddress 5y agoWe absolutely can. Imagine a US president outwardly Trump-like but who was also a True Believer instead of a cynical, mentally-ill real estate huckster. They would have the unilateral capability to start a global nuclear war. And they might actively desire to do this, on the basis of religious delusion (“it’s time for armageddon!”). We are frighteningly close to this scenario in 2025. It’s a bad idea to assume that the gop candidate will be a fascist-wannabe in on the joke (someone like desantis or cruz). We can survive run-of-the-mill fascism, although it won’t be fun. But it could instead be someone truly existentially dangerous.
- h2odragon 5y agoIt's Anti-Fragile. If it breaks all the time, everybody is highly experienced at patching together new workarounds, mechanisms for fail over are in place and regularly tested, and there's whole classes of corner case bugs that get flushed out to be stomped (or nurtured as cherished pets) instead of breeding in the dark and jumping out at you all at once. How can the "Internet routes around failure" be trusted without testing? Everything needs regular exercise or it atrophies.
- tata71 5y agoIPFS ENS
- fredophile 5y ago"The internet routes around failure" hasn't been true for a long time. It refers to the original topography which has been replaced with a hub and spoke model. Remove a few hubs and you have disabled a large portion of the internet.
- Iv 5y agoDisabled? Throttled down the speed by half? Yes. But to disconnect whole regions, you have to do conscious sabotage and even Mubarak did not manage to switch off Egypt when he wanted to and gave orders to.
- paxys 5y agoWhat original topology are you talking about? There was never a time when your end consumer device could be used as even a semi-reliable web server.
- Teever 5y ago> There was never a time when your end consumer device could be used as even a semi-reliable web server. I dunno man, My used T420 laptop is serving several sites over a residential symmetrical fiber connection just fine.
- TheMagicHorsey 5y agoWhat are some other/better proposals for how to organize a world-wide network? This is one area where I have not seen many articles. But admittedly, I'm probably not looking in the right places. Any suggestions?
- teddyh 5y agoYou could ask Cloudflare or Google and they would probably say that they should run it all.
- hunterb123 5y agoWhich funny enough, are the only parts that really go down all at once. When the internet has troubles it's a mistake of a giant centralized service. Most of the time only that service is affected by their own mistakes, but sometimes that service hosts a lot of others, or is so massive that they cause DDoS attacks like when Facebook went down and their clients spammed DNS servers. Seems the internet is fine. Centralized services not so much.
- cblconfederate 5y agoblockchains are one
- throwawaysea 5y agoI am not a security expert, but this article seems to just be saying that the Internet is held together by trust, convention, and an ever-evolving set of technologies. In the case of Level3 (now Lumen), it seems they did not deprecate an insecure method that others already deprecated. And it seems that better technologies are on the horizon (RPKI) but not yet fully in use. To me this doesn't feel as bad as "spit and baling wire". We could be more secure by holding everyone to a stricter standard on adopting newer, more secure technologies. But is it really as broken as the title suggests? I don't think so.
- oblib 5y agoI've been leaning on my app's users to do what they need to do on their end to implement offline/local first use of the app and they just do not get it. For them the only issues they've had were connection issue on their end with their service providers so they don't feel this is an issue of concern. But I read stuff like this, and in this case it's Krebs, so I have to expect these kinds of issues will pop up. The article mentions the FB outage and most everyone on my FB feed was freaking out over not being able to access it, and for the most part it's not a critical service. And when they came back online some of the conspiracies they were sharing about what/why it happened were way over the top. From my perspective it feels like everything on the internet is just one missed tap on a keyboard from breaking.
- imchillyb 5y agoHas no one here seen: https://www.reddit.com/r/cablefail/ https://www.reddit.com/r/cablefail/ There are 1000s of these pictures. The entire world's IT infrastructure has been held together with spaghetti-noodle-cabling and bodged patches for over three decades now. I'm even guilty of it. Most IT guys are guilty of it. Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware running with bandaids and bubblegum. Software that more than 5 different teams have had their fingers in, mucking about, and some codebases have orders of magnitude more than that. Anyone that didn't know this, doesn't pay attention, or just started internetting.
- jl6 5y ago> Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware running with bandaids and bubblegum. Software that more than 5 different teams have had their fingers in, mucking about, and some codebases have orders of magnitude more than that. And yet, perhaps counterintuitively, the main downside of such systems is that they are slow and expensive to change, not that they are unreliable.
- dragonwriter 5y ago> And yet, perhaps counterintuitively, the main downside of such systems is that they are slow and expensive to change, not that they are unreliable. They are slow and expensive to change because they are maintained because they have been, at great expense and cost in both failures and remediation efforts, made tolerably reliable (but still extremely fragile) so long as things are exactly within certain expectations (which often have been narrowed from the intended design based on observed bugs that have been deemed too expensive to fix), and it is inordinately difficult to modify them without causing them to revert to a state of intolerable unreliability. They are systems that generations have been spent reshaping business operations around their bugs to make them “reliable”.
- charcircuit 5y agoHow "pretty" your cable routing is doesn't effect performance / reliability.
- rufus_foreman 5y agoIs there a shortage of spit and/or baling wire?
- winternett 5y agoAnd Google requiring (costly and/or time consuming) SSL certs to be applied on all sites to "ensure security" was also a big industry money making nightmare for many independent (non-income-driven) sites that is still playing out badly, and not providing much more security. Two factor authentication and account verification is really an elaborate corporate sham to get people's phone numbers and PII for free. It doesn't do anything new or good for consumers in terms of security over time. There, I said it. I prefer the old Internet. All these new fangled "fixes" are only makin it worse, more expensive, and overly complicated. :/
- fulafel 5y agoDeprecating unencrypted HTTP is a big systemic improvement even though some individual sites may not benefit much. It's a network effect. (What's the money grab given free let's encrypt certs?)
- winternett 5y agoLets encrypt from what I understand require time consuming updates every few months. My host provider also does not allow me to install them manually, further complicating the process, and conveniently they sell certs for $125 a year... Per site. It's been a thorn in my side because we're too big to easily move now.
- fulafel 5y agoYou are absolutely not meant to do the updates manually.
- winternett 5y agoOn one ISP that I host sites on, they restrict cert installs and don't allow SSH access. It's done in order to sell their cert services. I have too many sites on there to move easily... It's complicated. Eventually I'll bite the bullet and move to a new host. :/
- krisrm 5y agoI'm a bit shocked that MAIL-FROM auth was ever accepted, let alone until 2012. Even the other auth methods via email seem somewhat dangerous, though I sincerely hope these registries follow extremely strict policies for key management.
- throwaway984393 5y agoThat's how validation of TLS certificates and domain registration still works.
- oneepic 5y ago...and routers
- SahAssar 5y ago> “LEVEL 3 is the last IRR operator which allows the use of this method, although they have discouraged its use since at least 2012,” Korab told KrebsOnSecurity. “Other IRR operators have fully deprecated MAIL-FROM.” I'd prefer if we kept deprecated and removed as two different terms. It sounds like level3 deprecated it, and everyone else removed it. To me (and most definitions I can find) deprecated basically means "don't start using it, if you are using it stop using it, we will remove it soon but have not done so yet for compatibility reasons"
- throwaway984393 5y agoWhenever people are predicting the end of the world because of some political or cultural upheaval, I think about the internet, or airport security. There are really simple ways that any idiot could totally fuck up either of them and cause catastrophic problems. But it doesn't happen. What that shows is that the potential for catastrophe has nothing to do with catastrophe actually happening. Even if the world could fall apart around you at any moment, it's probably not going to.
- 234023048230948 5y agoAh, Brian "dox the critics" Krebs. Reeaaaly solid guy.
- Ekaros 5y agoI have kinda wondered how robust Internet would truly be if the proverbial stuff really hit a fan. I mean proper nuclear wars or economies including large ISP going down with admins dying or being fired on left and right. Specially with all of the talk of global crypto and so on... As I understand it is continuously almost hand-tuned machine and enough misconfiguration or entities simply stopping to exist much of the transit capacity would be gone. Works well enough when it is not touched and there is someone on call to fix it, but specially if later was gone? How robust and self-correcting is it really? My guess is probably not at all...
- yazanobeidi 5y agoYou’d be surprised what goes on in Level 0… who works there, how they work, the hour-by-hour. Even at a big company that handles most of the world’s credit card transactions at their biggest data centre, for example. It would absolutely blow your mind. It’s amazing the internet works as reliably as it does!
- mmaunder 5y agoBGP is how the most critical infrastructure on the Net communicates and BGP4, the current version, is 15 years old.
- pmlnr 5y agoAge, alone, doesn't make something bad or unreliable.
- illwrks 5y agoValue less comment... but I've always had a thought in the back of my head that it will only take some catastrophe for us to be back in the dark ages. I hope I'm wrong.
- a9h74j 5y agoAs a less all-or-nothing scenario, one relevent concept comes from Joseph Tainter as a theorist of collapse. He proposes that eventually complexity reaches a point where it not sustainable (either in general or at the margin), and a society collapses back to a configuration of less complexity.
- thenthenthen 5y ago'spit'...nice way to put it I guess, Krebs still living with his mom init?
- rswail 5y agoThe term "spit and baling wire" is a well known (US/AU rural at least) expression to mean something that is working well enough but is fragile.
- thenthenthen 5y agoI never knew! Excuses for the terrible joke
- walrus01 5y agoKrebs is not wrong but he also doesn't get into how many things are "verified" through an even older mechanism, the LOA (letter of authorization/letter of authority), which is literally just an actual letter on one corporate entity's letterhead, specifying some CIDR-notation prefixes/ranges that are permitted to be announced by that entity's upstreams, signed by a corporate officer. Often in the form of a raster scanned PDF. BGP4 in general dates back to a time when everybody trusted each other and the global routing table/network engineering community was a much smaller thing. We've been trying to glue better things on top of it for 25+ years.
- chasd00 5y agoYou know what? I get the feeling that _everything_ is held together with only spit and baling wire.
- balaji1 5y agoI get the same feeling too. And maybe that's fine. We are led to believe that we are over-reliant on a particular things like the internet or the shipping supply chain but we probably are not. Most modern, massive systems seem that held with only spit and baling wire. This might not be true for infra like roads, bridges and dams. And there are things, buildings and structures that are 10s or 100s of years old that still work. Power plants and cars for example.
- jrochkind1 5y ago> However, the additional integrity RPKI brings also comes with a fair amount of added complexity and cost, the researchers found. I wish our government spent billions on funding open source software to reduce that complexity and cost, instead of on introducing/hoarding vulnerabilities. This is an appropriate task for government funding. (even better yet, multi-lateral/international funding). Nobody else is going to provide the resources to get it done. Clearly.
- alexshendi 5y agoYes, but why did he only realise this on November 26th, 2021 AD?
- alexshendi 5y agoOf course FidoNet is much better.
- infologics 5y agoI really like reading through a post that can make people think. Also, many thanks for permitting me to comment! http://infocampus.co.in/ui-development-training-in-bangalore.html http://infocampus.co.in/ui-development-training-in-bangalore... https://www.firstenquiry.com/ui-development-training-in-bangalore https://www.firstenquiry.com/ui-development-training-in-bang... http://infocampus.co.in/web-designing-training-in-bangalore.html http://infocampus.co.in/web-designing-training-in-bangalore.... http://infocampus.co.in/web-development-training-in-bangalore.html http://infocampus.co.in/web-development-training-in-bangalor... http://infocampus.co.in/courses/reactjs-training-in-marathahalli-bangalore/ http://infocampus.co.in/courses/reactjs-training-in-marathah... https://www.firstenquiry.com/reactjs-training-in-bangalore https://www.firstenquiry.com/reactjs-training-in-bangalore http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angularjs-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.html http://infocampus.co.in/angulartwo-training-in-bangalore.htm...
- cainxinth 5y agoSo is society. When you’re a kid, it seems like everything is incredibly well organized and orchestrated. As an adult, you realize the world is a patchwork of semi-functioning systems and it’s a miracle the whole thing works as well as it does.
- iancmceachern 5y agoI've been thinking along these lines a lot recently. When young I thought everything was all carefully architected, thought out and planned. I've come to realize it's all chaos, interactions between the disparate desires of various individuals and groups randomly acting constructively and destructively with eachother creating random results.