4 ms·

As someone who understands networking at the office / rack level (but not the datacenter or nation-wide level), what are the single points of failure that could

• pquerna 15y ago
  At a guess, things like nation wide authentication servers? (I don't know how devices are authenticated in LTE though)
  • aarghh 15y ago
    That is a good guess - the Home Subscriber Server is the authentication server in an LTE network. Short of this or a PCRF (Policy Server), I cannot think of any other single element that could bring the network down. Routing (as identified by another reply) could be an issue, but I would assume that 3G and 4G data use the same core network.
    • Maven911 15y ago
      I have a feeling it might be the routing to the hss/pcrf that could have been the isue. Almost every carrier has redundancy for these products and they are often in geographically distinct areas. Perhaps the routing from the eNodeBs/edge network to the core network was handled by another division within Verizon or even leased from a third party and could have caused all of this Or a disgruntled employee left a cron job on all the key systems to rm -rf :-)
• nknight 15y ago
  pquerna's guess is a reasonable one, though I don't know the authentication mechanisms, either. Here are some other wild guesses that may have little or no resemblance to what Verizon is experiencing, but kind of show what can bring down an Internet-like network: * Some sort of configuration fat-finger that got rolled out everywhere without proper testing. You'd like to think that's impossible on this scale, but after having an insider's look at the operations of a couple major US cable companies, I put nothing past telecoms. * Verizon probably runs some form of routing protocol internally. If a core router has gone insane, you could, conceivably, get route flapping and bring down pretty much everything, but that's not supposed to happen on carrier-grade equipment (cough see previous paragraph). * In a closely related scenario, large portions of the public Internet have been brought to their knees at least once by a nasty BGP bug in a major vendor's routers that got triggered when somebody broadcast just the right set of options in their announcement. Picture a guy in upstate New York trying to route around a bad line card, and suddenly the US goes dark because half the routers crash when they got the BGP table update. You can probably spot a common theme here -- a "decentralized" but homogenous network is still vulnerable to the triggering of problems shared by all points.
  • bgray 15y ago
    Very good points. It would be interesting to hear (technically) from Verizon about the issues and resolution.
  • Nrsolis 15y ago
    The thing you have to understand about "wireless" systems like CDMA/GSM/HSPDA/LTE is that there is little diversity in the vendors of these types of systems and there are a bunch of relatively closed protocols that govern how things like the base stations authenticate subs and do provisioning of stations. It's not like IP; it's more like Novell IPX. So even though LTE is ostensibly an IP-based system, there are a host of dependencies on single vendor closed management systems and protocols that could be the cause of any outage. A nationwide outage points to the failure of one of these systems to properly handle letting subscriber stations onto the network, not something simple like getting packets from here to there. Think failure of the RADIUS/DNS like systems, not the routing infrastructure.
    • Maven911 15y ago
      Your premise is more accurate, you just need to replace IPX with 3GPP standard protocols (vendors dont make their own protocols - everything is standardized), and instead of radius/dns (which are used in telcos nonetheless), its more likley HLRs,CSCFs, MMEs, or PDN gateways that can bring an entire network to its knees, or a poorly implemented mobile backhaul solution
      • deleted 15y ago
        [deleted]
      • Nrsolis 15y ago
        Correct. There are standards, but the standards are relatively proprietary (at least compared to the RFCs from the IETF) and the standards bodies that certify solutions as "compliant" are controlled by the vendors themselves. But your point stands. The last time I got a look at a major wireless data network (3G) I was floored by how damn complicated it was relative to how a similar solution would be implemented if you used "pure" IP technologies rather than the over-engineered versions them implement in order to constrain the number of competitive solutions they'll find in the marketplace. Its WiMax vs LTE.
• alanh 15y ago
  The connection between the network and Langely, through which all data passes.
• Maven911 15y ago
  Things are usually not properly tested, you have to deal with a lot of interoperability issues from different vendors which only get played out on carrier network, logic fails by employees/integrators such as assigning an ip addresss in one corner of the nation and re-using it elsewhere inadvertently due to poor IP address managment, errors flooding a system which cause it to self-reboot which then signals to other systems to also self-reboot or simply a core central node such as hlr/hss has failed and proper geo-redundancy was not implemented