21 ms·

5G: The outsourced elephant in the room

590 points by sam_lowry_ 6y ago

ttty 6y ago
In short, the discussion now is whether European telecommunication companies should source equipment from Huawei. The worry is that since Chinese companies are heavily influenced and often owned (indirectly) by their government (or ruling political party), picking Huawei equipment for 5G might be bad for us. In the 5G discussion, the assumption is that national, large scale telecommunication service providers are currently in good (or even full) control of their networks. The idea is that these providers (think Vodafone, Deutsche Telekom, Proximus, Orange, Telefónica, KPN etc) procure equipment. Most billing has been outsourced since the early 2000s at least. One large Dutch mobile provider has handed over most of their technical staff to Huawei. At one major mobile provider the chain is now that the company has outsourced IT to Tech Mahindra. In turn, the company in turn talks to Ericsson, who then finally operate the network. The idea that telecommunication service providers can guarantee the privacy of their subscribers is highly questionable. All service providers have a security department, and I know many of these people well, and feel their pain. Sadly in all providers I know, security departments struggle to get their recommendations implemented. Ericsson software components appear to be developed in China, which may bring worries of its own. European service providers have reasons beyond balance-sheet gymnastics to outsource. All large scale outsourcing companies have been thoroughly compromised. In the end, outsourcing is now almost the only possibility to survive. One European 15-million subscriber network now relies on a core team of 4 people (one of whom is their manager) Technical expertise is the first line of defense against malicious vendors attempting to spy and destabilise. Having strong local knowledge of telecommunications helps assure the future autonomy of vital capabilities. European service providers are by and large currently not in good control of their networks, writes Picking Huawei is not specifically a sea change but simply a continuation of existing policy for most providers. If we really care about our privacy and the stability of our communication networks, we should be able to build such networks autonomously. The Galmon GNSS Monitoring Project monitors the RNA levels of humans in the human genome.
- wellx 6y ago
  " European service providers are by and large currently not in good control of their networks, writes Picking Huawei is not specifically a sea change but simply a continuation of existing policy for most providers. If we really care about our privacy and the stability of our communication networks, we should be able to build such networks autonomously. The Galmon GNSS Monitoring Project monitors the RNA levels of humans in the human genome." Can you just get lost? Thank you!
- layoutIfNeeded 6y ago
  Are you GPT-3?
  - misnome 6y ago
    It does seem to be some sort of automated spam - a mix of the article, the same chunks of disconnected factlets and some random sprinkling of nonsense. Is it possible to flag a whole account? @mods?
    - wffurr 6y ago
      The “random nonsense” in this particular example are the next/previous article links from the bottom of the page.
    - mkl 6y ago
      I believe the whole account is flagged if enough comments are flagged by multiple people. I've already emailed dang.
      - misnome 6y ago
        Right it looks like 10 months ago it was an actual account, and either got hacked to karma-farm(?) or author decided to use as a throwaway test.
        ChrisMarshallNY 6y ago
krona 6y ago
This aligns with my experience working at IBM. I knew Vodafone was a big customer, but I found the extent of that relationship peculiar; in presentations given by senior engineers it came across that IBM seemed to running the core parts of their network. That can sometimes mean Vodafone engineers are barely allowed to touch anything without an IBM contractor on site. It got me thinking; who and what is really running these telecom networks and are they not much more than a branding exercise.
- g_p 6y ago
  This is more common than many think. Once a managed provider steps in, they want to "own" the configuration. You end up with the operator itself actually having to raise tickets with the MSP to change things on their own network. All this becomes a huge issue if there's a major outage, as the MSP might not have enough access to actually get in and do anything. Most telecoms networks are run (to some significant extent) by a managed service provider, in my experience. When O2 UK had a major core outage due to an Ericsson certificate expiry inside the core, it wasn't O2 engineers that found and fixed the issue; it was Ericsson engineers. The margins as an operator don't make it easy to keep around the deep technical skills to be an expert in the network you own.
  - taurath 6y ago
    Operator margins in the US are enough to pay billions for media companies, at least. But maybe more competition is allowed in Europe.
    - g_p 6y ago
      There's definitely a "side of the pond" aspect to this - US operators pay for media companies, but they likely do so to raise their margins, and I presume they're borrowing the funds from institutional investors to do this, rather than bootstrapping their acquisitions with profits. In Europe you'll likely see far lower per-user pricing due to competition. You'll typically have 3 or 4 operators with physical networks, and a number of virtual operators providing white labelled service over the underlying networks. A standard target ARPU (average revenue per user) would probably be around 15 GBP per user per month. You'll likely get to that via contract users who you try to get on 22 GBP per month or thereabouts, and pay as you go users whose ARPU is far lower (maybe 8 or 10?) Compared with the US, consumer prices paid are incredibly cheap - expect unlimited calls and SMS, and many gigabytes of data. If you shop around you'll get even cheaper still. In the UK you'd be able to get unlimited 4G or 5G data for 25 GBP per month. Clearly the US has a much larger geography to cover, but there's definitely more competition leading to downward price pressure in Europe, in my view.
g_p 6y ago
For background context around telecoms for anyone reading this, there is an underlying difference in how telecoms networks are designed and architected - in the Telco world, links between networks were predicated on trust. Originally, telecoms networks were run by national level quasi-government operators, one per country. You interconnected with other "known entities". Even now, you likely have 3, 4 or 5 national mobile operators in any one country. They negotiate their own roaming agreements in order for you to get roaming access. It's all driven by these kinds of relationships predicated on trusting other networks. In IT, we are rapidly moving towards zero trust (due to the internet), but circuit switched (legacy) voice is still all designed to be sent over private circuits between operators who trust each other. The legacy protocols (see SS7), used to route calls between operators are functional, but also lack access control and authentication, as it's assumed only trusted parties are on the network and able to use them. Those assumptions are no longer valid, and there's a huge challenge in dealing with this - hence SMS and call interception and rerouting attacks to steal 2FA tokens etc.
- est31 6y ago
  Isn't the internet in the same situation, with BGP being assumed to be done between trusted parties?
  - darkr 6y ago
    Yes, but increasingly traffic is strongly encrypted, with users able to exert some level of control over that encryption
  - pas 6y ago
    The first rule of BGP is to filter what you get. Don't just blindly accept whatever the peer advertises. The second rule is obvious, but for the third there's also a lot of knobs for traffic engineering with BGP. And on top of that there's RPKI [ https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/ ] I guess all of the big telcos have some homegrown ossified hacky "solution" that also serve as a minimal kind of "firewall" for SS7. (Basically I imagine that there's a lot of hardcoded rules for phone numbers, country codes and operators. Sure, they probably are an opposite of a problem for national intelligence services, after all it's easier to go by unnoticed in the noise, but they at least help with a total BGP-like hijack of a whole country code by an operator.)
kanisae 6y ago
In the past I worked at a mid level 4G provider in the US who had to deal with the larger providers on a regular basis. I was always astounded at how little they knew about their own networks. Regarding the articles statement of providers wanting an "all-in-one" solution, I have seen that in person, where management forced it, found it was horrible and then gave in and let us build the mixed vendor solution that worked well. I've personally mixed enode-b's from 2 different vendors to 3 different vendors SGW's and a different vendors PGW with no issues. The "One Throat To Choke" idea doesn't work if your business depends on that throat to operate so you end up with the vendor calling the shots instead of the business.
- g_p 6y ago
  The bigger they are, the harder they fall... On the whole, the technical standards should allow the kind of interoperability you described. That's the kind of fun real-world engineering that techies love. The bean-counters don't, because it's more devices needing support packages, it's more suppliers on the books, and ultimately it's probably (slightly) less profit than buying a single box. I've seen big household name operators in Europe stop even pretending they're doing the work, and straight up pass on contact details and a mobile number for the person at their tier-1 vendor partner, so you can liaise directly with them. It seems in these "5G" days even more than before, operators are retreating into the business of connectivity service, and leaving more and more for their vendor partners to do. When you're not even hiding the fact to a client that they may as well speak directly to the vendor, that says it all(!)
- sgt101 6y ago
  >>The "One Throat To Choke" idea but it sounds soooooo good in meetings !
baq 6y ago
> We recently asked a large European service provider why only part of their customers get IPv6 service, and how they pick which parts do or do not get such service. They could not tell us, and informed us they too would like to know woah. as a EU citizen, i'm terrified. i wanted to say surprised, but after a moment's thought, turns out it's only a moderate misalignment of expectations.
- JPLeRouzic 6y ago
  I retired in 2012, but at that time my employer had completely subcontracted the operation and maintenance of its mobile network to Ericsson, Huawei and Nokia. It was in France. Once I asked for a one day snapshot of all mobile data for a cooperative R&D project. The saga went on for months with repeated requests at various hierarchical levels, but to no avail. It's not that they refused, but I guess that the guys in charge simply were unable to get the requested information from the subcontractors.
- iagovar 6y ago
  I work for a large EU Telco and I can tell you the inside battles to get stuff done are absolutely ridiculous. Of course I work for a subcontractor too.
afarviral 6y ago
Wow ... I read this whole thing thinking it was a human. On a re-read I'm wondering what I was thinking. One thing that stands out on the account's previous posts is the paragraph lengths are quite consistent. I'm going to have to train my bot senses. I got conned. Any tips? Mindlessly reading me is not prepared...
- afarviral 6y ago
  Oh dang.. Was trying to reply to ttty comment that got flagged as likely not hijacked account. Off topic but interesting.
  - monkey_monkey 6y ago
    Ah ok, I got terribly confused, because the sam_lowry account seemed quite legit!
    - afarviral 6y ago
      Apologies... But also: question everything. He he.
- g_p 6y ago
  One tell-tale sign I've noticed is sentences that don't go anywhere - the last sentence of the first paragraph starts going somewhere, then suddenly stops. If we assume people communicate for the reason of conveying some meaning, always dig for the meaning. If it's not there, it's likely some kind of vapid content-less babble... If you read critically with a view of "what are they actually saying?", you tend to spot this fairly quickly. The ending with some irrelevant babble gave the game away a bit though.
  - afarviral 6y ago
    That's an excellent tip. Form an idea of what "they" are trying to convey... If it evaporates unexpectedly it might not be trying to convey anything...cause it's just an ML algo. The trouble is having clarity of thought myself, I barely know what point I'm trying to make let alone following the precise points of others half the time. These things really nail verisimilitude of chatter about an unfamiliar topic or of a smarter person, for instance, where you can't detect the BS because you assume lack of knowledge.
deleted 6y ago
[deleted]
one2three4 6y ago
This is gold. >> In reality, most service providers have not been operating on this model for decades. Driven by balance-sheet mechanics and consultants, service providers have been highly incentivised to outsource anything that could possibly be outsourced, and then some. >> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor. Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers. I think it's quite a safe bet that no operator in China went that way by buying and outsourcing from/to Western companies.
- dragonelite 6y ago
  Ericsson threaten to leave Sweden, if actions by Swedish government meant that they lose access to the Chinese markets and running/future contracts.
- tguvot 6y ago
  It's nearly impossible for telecom to deploy/configure/maintain their networks by themselves due to the scale. For example, I just googled, AT&T seems to have 67000 towers/macro cell sites. Let's say they want to update all of them to install modern 5G equipment. In many cases this equipment may come from different vendors and to deploy it might be multi-day job. Of the top of my head, about 20% of site visits fail due to various reasons (with good percent of them failing even before starting due to scheduling issues, sickness, not delivered at time equipment, etc) . How much time and people it will take to AT&T to do all the work on it sown ?
  - tgv 6y ago
    They did it before, even had to wire everything together, coast to coast.
    - tguvot 6y ago
      "Before" it was simple. Now it's very complex. It's very complex exercise in large scale planning, logistics and coordination. If company tries to do it by itself, it something that will take years of work of hundreds to thousands of dedicated to this task people to accomplish. Market and customers require faster pace.
commandlinefan 6y ago
> Driven by balance-sheet mechanics and consultants It’s not just telcos...
- ng55QPSK 6y ago
  No only by balance-sheet mechanics, there is some politics attached. In my work place we're currently discussing moving in-house data centers to a cloud-provider. At the first glance it looks like at a higher cost (in-house cheaper). Now some balance-sheet-makeup sets in.
Semaphor 6y ago
Cynical me certainly can believe all this. But on the other hand, I’m wary of just reinforcing what I believe anyway. How trustworthy is this? There seems to be a lot of inside information, where did they get it from? Does anyone have corroborating links? All article links are either general, or US specific.
- g_p 6y ago
  Much of this is fairly widely known in the telecoms sector, and is "open secrets". The sector is a pretty "closed shop" though, full of trade secrets and "proprietary" things. Underneath it all though, actually it's fairly simple once you get your head around it. If you work closely with an operator, even as a client, you'll see examples of this - the number of people brought to meetings from the vendor, versus from the operator. Who answers the questions. For a public example, see the Telefonica O2 outage in the UK (and Japan, I believe) due to an Ericsson certificate outage, and how much of a role Ericsson played in this. (https://www.theregister.com/2018/12/06/ericsson_o2_telefonica_uk_outage/ https://www.theregister.com/2018/12/06/ericsson_o2_telefonic...) Press releases also give bits and pieces away: https://www.ericsson.com/en/press-releases/2019/11/orange-opts-for-ericsson-operations-engine-in-managed-services-extension https://www.ericsson.com/en/press-releases/2019/11/orange-op... https://www.mobileeurope.co.uk/press-wire/9588-three-uk-joins-telefonica-by-outsourcing-core-management-to-huawei https://www.mobileeurope.co.uk/press-wire/9588-three-uk-join... Although they might not give the level of detail you're looking for, it should hopefully corroborate things.
  - Semaphor 6y ago
    No exactly what I was looking for, but a good start. Thank you!
- ahubert 6y ago
  Hi - author here. By all means ask around. I can only tell you that I've received may corroborating anecdotes over the past year. Many telcos even assumed I was writing about them specifically, when I wasn't! I also have a second post that has some more logos and names where I based this article on -> https://berthub.eu/articles/posts/how-tech-loses-out/ https://berthub.eu/articles/posts/how-tech-loses-out/
walrus01 6y ago
>> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor If you think this is bad in some place like the UK, you should see how ISPs and mobile network operators are set up in some countries in the developing world, where the vendor has fully captured the Telco as basically a hostage to its technical services. This is what happens when you have a mixture of institutional corruption, kickbacks and bribes, lack of local technical resources to develop a domestic network engineering talent pool, and a vendor that knows how weak the client entity's negotiating position is.
- kazen44 6y ago
  What also does not help is that becoming good at networking engineering at scale is something that is hard, if not impossible to learn on your own. Software Engineering is something one can teach themselves on a laptop, learning to design, build and operate networks at a large scale across geographies is simply not possible without being part of the industry. Network engineering talent is incredibly hard to come by in most regions of the world, especially if you consider that ISP networking deals with arcane technologies not really used in most "enterprise" networks. (BGP in various ways, MPLS is a big one, and arcane transports like SONET or DWDM solutions). Sure, one might be able to learn how to configure BGP, how ip works etc from their laptop using GNS3 or a couple of second hand routers/switches, but learning how to design networks at scale is completely different beast. Most people seem to enter the field by getting hired as tech support at a NOC and working their way up from there, which is kind of a grind compared to some more lucrative positions available to people who posses the technical talent.
  - walrus01 6y ago
    I agree with 100% of that - a much harder problem to solve when learning real network engineering is much more capital intensive and requires real financial resources that may be beyond the reach of many.
    - kazen44 6y ago
      Also, it is a really difficult problem to solve. Most curriculums of universities and polytechnical schools only focus on the technical, theoretical aspects of network engineering, but getting hand-ons experience in actually running an ISP network is very, very difficult. The market for people who want to get into this line of work is also very small compared to software engineers, programmers and system administrators. The work is usually high risk, technically very complex and it can be very stressful. (It being vital infrastructure, and breaking something can result in catastrophe for your customers in some cases) I sometimes wonder what will happen if we have no one left to maintain the systems so many layers of software and systems depend upon.
magwa101 6y ago
Interesting insights on who operates these networks. However author equates US eavesdropping to Chinese eavesdropping. That is simply not serious. The "ship has left the harbor" view of existing infrastructure is not sensible, ownership matters.
ng55QPSK 6y ago
Ehm. You all are aware that 5G was created with the expectation that in the future all networks (and core functions) will be cloud-based? The last remaining HW will be the physical antenna and some PA/LNA and some local signal processing. You connect fiber to that and everything else is a operator-as-a-service model - running on AWS/Azure/GCP.
- walrus01 6y ago
  You're aware that telcos have massive amounts of physical stuff needed to make that happen first? You can't "cloud" hand wave away things like massive metro scale dark fiber networks for backhaul. Or things like inter city long haul DWDM networks. I assure you there's a lot more going on hidden behind the scenes of a modern ISP or 4G/5G carrier than just some antennas and software in a VM somewhere.
  - ng55QPSK 6y ago
    This will not happen this year. But in the long run you don't need much own HW to be an operator. And fiber connectivity is (as the startpoint of this comments) outsourced in large scale.
    - walrus01 6y ago
      "in the long run you don't need that much HW to be an operator" - have you ever worked for a facilities based ISP or Telco and visited the interiors of dozens of different POPs? Please do so and then come tell us all about how telecom infrastructure on a national and global scale isn't composed of massive amounts of hardware all over the place.
      - ng55QPSK 6y ago
        I work in a team that created 5G. And around ~2014 it was clear: operators want to go away from own HW.
        6y ago
not1ofU 6y ago
Related: This white paper was published within the last 2 weeks relating to "5G Network Slicing" Quote from the author: “Currently, the impact on real-world applications of this network slicing attack is only limited by the number of slices live in 5G networks globally. The risks, if this fundamental vulnerability in the design of 5G standards had gone undiscovered, are significant. Having brought this to the industry’s attention through the appropriate forums and processes, we are glad to be working with the operator and standards communities to highlight this issue and promote best practice going forward.” PDF can be downloaded from here: https://info.adaptivemobile.com/5g-network-slicing-security https://info.adaptivemobile.com/5g-network-slicing-security
Clewza313 6y ago
> Since the early 2000s at least, most billing has been outsourced. This works by sending all Call Detail Records (CDRs) to a third party, often from Israel or China. This is quite misleadingly written: telcos are not shipping reams of CDRs to some cubicle farm in Haifa or Chongqing. Yes, almost every telco outsources its billing software to other companies, notably Amdocs (founded in Israel, now HQ's in the US). However, billing info is some of the most sensitive data a telco has for both privacy and commercial reasons, so that software always runs in a closed environment from where it cannot dial home. Historically that's been on-prem, it's slowly moving to the Cloud but even there it's going to be firewalled off very carefully.
- waheoo 6y ago
  Oh yea, super secure, their outsourced security team is all over it.
- sgt101 6y ago
  There's a big gap between what happens in the larger telcos that run infrastructure, and the white label operators.
  - tguvot 6y ago
    White label operators can't usually afford Amdocs solution from one side. From the other side, Amdocs doesn't have a server farm in Israel to do CDR processing :) Everything happens either on client premises (and Amdocs can deliver end-to-end solution, including buildout of complete datacenter if it required by client) or "in the cloud"
- Thr0wawayDocs 6y ago
  Let's not oversell the fact that Amdocs' official HQ is in the US: It's an Israeli company through and through. It did more development in the US back in 2001 than it does now: Today, their US footprint is mostly customer sites. And you will find people brought in from Israel everywhere. It's always been bad enough that managers that don't speak Hebrew knew they were always at a disadvantage. Your best bet for saying it's not an Israeli company is to say that a whole lot of R&D is being done in other countries with cheaper labor. While it's true that the installations are on-prem (having been to quite a few of those), Amdocs business model isn't about dropping code and going away: They are so embedded with your typical deployment that there's plenty of opportunity to exfiltrate data. Sending every CDR to Haifa? Probably not: The Sysadmins on your typical large telco are iffy, but not that iffy. And carefully firewalled? The talent was never great, and the security practices were never all that serious: I've been handed production shells that I had no business having, because it was convenient at the time. Once again, I'd say that the best argument to claim that there's no data exfiltration is that the people writing the code aren't good enough to do this under the customer's nose.
lifeisstillgood 6y ago
>>> what remains in the other half are IT Architects who do not get closer to actual operations than an Excel sheet or a Visio diagram. the only light point in an otherwise depressing read
carlosf 6y ago
> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor. Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers. > The host service provider often has no detailed insight in what is going on, and would have a hard time figuring this out through their remaining staff. Rampant outsourcing has meant that most local expertise has also left the company, willingly or unwillingly. 100% reflects my experience working in Huawei BR a few years ago. Carriers are mostly customer facing companies and very limited technically. Our customer (million + subscribers BR carrier) often hadn't the slightest idea how their own network was built and worked. Banning Huawei is absolutely impossible, at least in Brazil.
tguvot 6y ago
I worked for a bunch of years in biggest Israeli company that is selling OSS/BSS and related outsourcing services to telecoms (those who can afford it's solutions), and had some first hand experience with them >Since the early 2000s at least, most billing has been outsourced. This works by sending all Call Detail Records (CDRs) to a third party, often from Israel or China. A CDR stores who called whom and for how long. More data might be attached, for example the location of the customer, or where the customer was roaming abroad etc. Don't know about software from China, but the one that we sold doesn't send anything back to Israel. There are a lot of rules and restrictions upon CDRs and we had a bunch of training with regards to it. Everything is running on client site, usually on hardware deployed by us at their data centers and managed by dedicated team of people who relocate to live next to the client in order to provide 24/7 support of the systems on site >Typical service providers have hundreds of thousands of network elements. Surprisingly perhaps, many of these are actually maintained manually (!). Thousands of networking engineers labour to keep all this infrastructure operating well. This is a mix of half-truths and lies. None of the operators have thousands of people to manually configure day-to-day network stuff. Operators have rather sophisticated automation systems (aka OSS) that deal with provision and configuration of everything in their networks. Or almost everything. Any given operator whose life span is a decade or two today has a boatload of equipment (thousands of different types of hardware from same amount of vendors). In many cases this equipment was bought and deployed 10 or 20+ years ago. Companies that made it do not exist for many years. This hardware can't be replaced with anything, because nobody does this type of systems anyway. Those systems tend to have proprietary interfaces and in many cases can be managed only through Element Manager which can be managed only manually through some ancient windows or java application. >Meanwhile, modern large scale internet companies (like Google, Netflix, Facebook) have automated all such maintenance. Automation in this context means that no configuration states are edited manually but instead, entire networks get provisioned and configured from central templates. >With such automation, small teams of engineers can control and operate vast networks with relative ease - especially if good use is made of continuous integration and real life testing. I also worked for a while in one of FAANGS. They have it easy: all the hardware with modern with nice interfaces. You can actually automate it. Also their networks are much-much smaller compared to mid-sized telecom, much simpler and much more homogeneous. Automation that FAANG I worked for was a joke compared to automation systems that run telecom networks. My job was near network engineering team and during conversations they admitted that what they have is crap. I believe that at one point of time they considered to buy telecom level OSS system but bailed out because they couldn't get a source code .
rijoja 6y ago
"Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers". Which providers are using and which are not using Huawei?
erikerikson 6y ago
The author would find Pivotal Commware[0] interesting. Not only stateside but offering advances in hardware efficiency and durability. Full disclosure: employee, soon investor [edit: also, they are hiring] [0] https://pivotalcommware.com/ https://pivotalcommware.com/
BenoitP 6y ago
> One even went so far as to state during an all-hands meeting with technical staff that ‘running a communication network’ was by no means a core competence for them. This is an outraging but very widely spread phenomenon. No industry is spared from the MBA hawks. Everything now is rent-seeking and moat building. Innovation has been packaged away and can only happen when the market makers say it can. What can an engineer do about that?
- quadrifoliate 6y ago
  > What can an engineer do about that? Lots! - Name the companies in question - Stop working for them, and start working for companies that favor engineering expertise - As a consumer, advertise the good companies and call bullshit on the bad ones - Raise awareness about these practices among your elected political representatives and their constituencies If all of their skilled engineers leave, the bad company cannot run only on the basis of MBAs juggling balance sheets. Unfortunately most of my fellow engineers are far more likely to sit around blaming “the MBAs” over drinks than take any of the above actions.
  - aembleton 6y ago
    > If all of their skilled engineers leave, the bad company cannot run only on the basis of MBAs juggling balance sheets. The MBAs can just outsource the work.
quelsolaar 6y ago
My assumption is that all intelligence services in all countries would love to have access to fully compromised networks. They spend all their time thinking about how to access information, so they would be fairly incompetent not to consider this. Does that mean that all systems are compromised? No, because there are risks associated with tapping in to these systems. Partly it depends on if they have access to the systems, but mostly on the possible blow-back if they get caught. Example: Swedens FRA (NSA equivalent) could in theory ask Ericsson (a Swedish company), to install a backdoor. But, Sweden has a fairly free press, and there are good chances that someone would leak this information. If it got leaked it would be a major scandal that could go as far as toppling the government and destroy one of Swedens most important export companies. Its very risky, and its a risk no one wants to take, so the parts made in sweden are probably not compromised. China, on the other hand has almost no risks associated with adding backdoor. No free press, hard suppression of whistle blowers, and since most foreign intelligence services already assume the equipment is compromised, there is no real reputational damage either. I assume they are all compromised, why wouldn't they be? The US is somewhere in between. Sometimes companies are compromised by intelligence services, but much more often I think its employees. Why try to change Tim Cooks stance on privacy, when all you need to do is find one Apple employee, willing to take a sack of money to "do their country a great service"?
- marsven_422 6y ago
  "Sweden has a fairly free press" That's so wrong it hurts! All our press are dependent on government "presstöd" aka handouts.
  - Swenrekcah 6y ago
    The press can be free regardless. If the allocations are according to some objective metrics then I don’t see a problem.
    - whatshisface 6y ago
      Objective metrics can be as biased as subjective metrics.
ksec 6y ago
Probably Off Topic, but may be a chance if anyone within the industry might know. What are the current / purposed patent licensing terms of NR-U; finalised and related with 3GPP Rel 16 are going to be? Specific to standalone NR-U ( As in MultiFire in 4G. ) which could compete with WiFi 6e.
deleted 6y ago
[deleted]
rcarmo 6y ago
There is an amazing amount of FUD in this article. I have worked in the telco industry for the better part of 30 years, and am back on it now after a 5-year hiatus in cloud computing. Before I "left" there was certainly a trend towards outsourcing and large "swaps" of radio gear (Nortel-Ericsson in my case, and Motorola-Huwawei at a direct competitor, to quote only two examples), but there was no way in $UNDERWORLD that we would let a vendor have direct access to our gear unsupervised (be it Cisco, Ericsson, whatever). Remote troubleshooting was possible, but usually via jump boxes and VNC (only very seldom we would let anyone VPN in, and even then it was only to sub-sections of the network). Nothing left our O&M network. Nothing came in, either, because upgrades were rolled out from internal servers. And it is still very much the same thing today. Although there are outsourcers and vendors who work alongside core staff in my telco customers (like myself now), we don't have access to anything but lab or dev environments, and even then mostly with MFA and very stringent limitations. Outsourced staff _does_ do field service of various kinds, and they do have access to base stations, DSLAMs and various other physical infrastructure, but that's usually done with (usually much cheaper) local technicians and not vendor staff. There are certifications for those. The reality is that most telco services are being "automated out" and moved to virtualized stacks that are easier to manage. And yes, VoIP on the core (no more SS7 if anyone can help it) and Kubernetes everywhere... But what I found to be really weird was the notion of outsourcing billing. Besides being a GDPR nightmare (and I'm in Europe, like the author, so I find it doubly unsettling), that was only done "off-prem" when all companies involved were in the same group (which was customary when fixed and mobile operators were separate). These days billing is, comparatively, greatly simplified (thanks to flat fees, real-time billing systems for prepaid and streamlined bundles), so the only data that actually leaves the BSS core goes to the (smaller and smaller) printing facilities. So I would take it all with a massive dollop of salt.
- thu2111 6y ago
  Which telco is that? Is it UK based by any chance? Article says things are different in the UK.
varispeed 6y ago
Isn't outsourcing essentially a tax avoidance? Companies look for cheap labour overseas because they don't want to pay local rates, which typically include higher tax and cost of adhering to any regulations. If a person tried to do this - for example outsourced their bank account to tax haven and asked their salary to be paid there, the authorities would be all over it. So now the fact that companies are not only not hiding the fact they are trying to bypass the system, they also are lobbying governments to make it easier! I for a second don't believe that there is no money under the table involved. Why otherwise politicians would choose to funk up the local population to appease a big corporation? Bottom line is that outsourcing should only be possible if it was not possible to create a product locally or companies should pay any difference in tax locally, so that people who got put out of jobs because of this can at least get benefits. And finally I don't understand why even discussing doing any deals with China does not amount to farting in a room.
- cptskippy 6y ago
  Outsourcing is done for many reasons. Sometimes companies have more projects than staff and outsourcing offers them flexible bandwidth. The downside to this approach is that it puts stress on staff who have to train or maintain the outsourcer.
  - varispeed 6y ago
    I would understand outsourcing to countries that value human rights, have workers' protections in place and so on. But simply going for the cheapest possible option, where you get forced labour and children making your product simply wrong and indefensible. We at least should lobby online and offline shops to show country of origin on the labels. I wish if I could go on Amazon and have ability to filter out anything that comes from China. If you want to buy something responsibly it is very time consuming and some manufacturers go to great length to hide where their products are really made. Those people who exploited labour in Asia now got filthy rich and they are in position to shush any politician looking to put a stop to this or make sure such filters would not be implemented.
elzbardico 6y ago
Just another instance of how Harvard Business School completely f*ing up things. Future historians will have trouble understanding how we let MBAs destroy our civilization.
phreeza 6y ago
What I don't get in this entire conversation is how is 5G different from any other transport layer? Can't secure communication be achieved by encrypting the communications at a higher level?
- mensetmanusman 6y ago
  Metadata still exists in that framework.
mmaunder 6y ago
Steel production capability is considered strategically important in case we go to war, and it has been so since World War II. Steps have been taken to retain domestic production capacity for this reason. Until we have a planet of one people and one nation, we’re stuck thinking this way about things that are critically important, should we find ourselves at war with a former partner. Tech sovereignty has become such a thing. And the bad news is that we have lost. I’ll leave others to debate why, but we can’t manufacture our own chips, we cant make our own telco networks, and the cloud systems that provide back end services are almost lost. The state of play here is dire for the US and it’s strategic partners. I’d say that surveillance is less worrying than the simple fact that a potential future adversary has an off switch for these things that they can toggle at will: no more chips, no more telco products and no more cloud services - now, let’s have that South China Sea conversation one more time...
peter_d_sherman 6y ago
5G Vs. Starlink Which will win? ? Or, more likely, does the future hold a coming broadband internet connectivity price war? ? And if so... ...who will be the "last IP address standing"? ?
Foivos 6y ago
This is my experience interacting with mobile telcos as well. Even to get some simple logs from a base station you need to either ask an Ericsson engineer or, worse, wait for the Telco employee with the relevant knowledge to find time to do it. Telco employees with such knowledge are very few compared to the amount of workload they have to do, so it is hard to get them to dedicate time to help you.
philjohn 6y ago
My concern wouldn't be with data being exfiltrated, seeing as so much is E2E encrypted these days. But what if there is a remote kill switch - taking down a cellular network could cause a whole heap of problems in the 21st century.
dd_roger 6y ago
I work in OT security in an industry completely unrelated to Telecoms but which is also a matter of national security and everything in this blog post doesn't really sound believable from my experience dealing with sensitive infrastructures. Different industries, different countries, etc. I get that things can differ a bit, but going from "the infrastructure is airgapped" (in my industry) to "the infrastructure is managed remotely by a foreign entity" (as claimed by the author) seems too big to be true. Not gonna lie, I'm a bit sceptical about the veracity of some of these claims.
tim333 6y ago
I wonder if it really matters if networks are insecure. I generally connect to the internet via whatever hotspot is available or 4G if not with no expectation that it's fully secure. That's what https and other encryption is for. And I don't generally have anything to hide - there are a bunch of tools like tor and VPNs for people who do. That's probably the way forward - secure tech on your device rather than trusting to the kindness of strangers. Talking of phone network security one thing that does piss me off is my phone company just transferred my phone number of 10 years to fraudsters who presumably called customer support with some sob story. You'd think they could have some standards to stop that like at least sending an email to your usual address saying "There's been transfer request - you good?"
- spixy 6y ago
  Ok, but what about SMS, MMS, phone calls, getting your approximate position, ...? What if they decided to shut down the whole network because of some reason? Then https wont help you.
  - tim333 6y ago
    You could drop SMS, MMS and phone calls and use something like whatsapp instead, which a lot of people seem to be moving towards anyway.
WhyNotHugo 6y ago
I can’t help notice that we always keep relying on closed-source for these kind of things, and then distrust providers because we don’t know what that secret code does. Am I the only one noticing that the obvious solution is to stop using closed-source tools for such delicate infrastructure that’s meant to serve the general population? I honestly can even grasp how we got into this mess in the first place: publicly funded software for public infrastructure that has such delicate security implications should obviously be open source only.

590 points by sam_lowry_ 6y ago

ttty 6y ago

In short, the discussion now is whether European telecommunication companies should source equipment from Huawei. The worry is that since Chinese companies are heavily influenced and often owned (indirectly) by their government (or ruling political party), picking Huawei equipment for 5G might be bad for us. In the 5G discussion, the assumption is that national, large scale telecommunication service providers are currently in good (or even full) control of their networks. The idea is that these providers (think Vodafone, Deutsche Telekom, Proximus, Orange, Telefónica, KPN etc) procure equipment. Most billing has been outsourced since the early 2000s at least. One large Dutch mobile provider has handed over most of their technical staff to Huawei. At one major mobile provider the chain is now that the company has outsourced IT to Tech Mahindra. In turn, the company in turn talks to Ericsson, who then finally operate the network. The idea that telecommunication service providers can guarantee the privacy of their subscribers is highly questionable. All service providers have a security department, and I know many of these people well, and feel their pain. Sadly in all providers I know, security departments struggle to get their recommendations implemented. Ericsson software components appear to be developed in China, which may bring worries of its own. European service providers have reasons beyond balance-sheet gymnastics to outsource. All large scale outsourcing companies have been thoroughly compromised. In the end, outsourcing is now almost the only possibility to survive. One European 15-million subscriber network now relies on a core team of 4 people (one of whom is their manager) Technical expertise is the first line of defense against malicious vendors attempting to spy and destabilise. Having strong local knowledge of telecommunications helps assure the future autonomy of vital capabilities. European service providers are by and large currently not in good control of their networks, writes Picking Huawei is not specifically a sea change but simply a continuation of existing policy for most providers. If we really care about our privacy and the stability of our communication networks, we should be able to build such networks autonomously. The Galmon GNSS Monitoring Project monitors the RNA levels of humans in the human genome.

wellx 6y ago
" European service providers are by and large currently not in good control of their networks, writes Picking Huawei is not specifically a sea change but simply a continuation of existing policy for most providers. If we really care about our privacy and the stability of our communication networks, we should be able to build such networks autonomously. The Galmon GNSS Monitoring Project monitors the RNA levels of humans in the human genome." Can you just get lost? Thank you!
layoutIfNeeded 6y ago
Are you GPT-3?
- misnome 6y ago
  It does seem to be some sort of automated spam - a mix of the article, the same chunks of disconnected factlets and some random sprinkling of nonsense. Is it possible to flag a whole account? @mods?
  - wffurr 6y ago
    The “random nonsense” in this particular example are the next/previous article links from the bottom of the page.
  - mkl 6y ago
    I believe the whole account is flagged if enough comments are flagged by multiple people. I've already emailed dang.
    - misnome 6y ago
      Right it looks like 10 months ago it was an actual account, and either got hacked to karma-farm(?) or author decided to use as a throwaway test.
      - ChrisMarshallNY 6y ago

krona 6y ago

This aligns with my experience working at IBM. I knew Vodafone was a big customer, but I found the extent of that relationship peculiar; in presentations given by senior engineers it came across that IBM seemed to running the core parts of their network. That can sometimes mean Vodafone engineers are barely allowed to touch anything without an IBM contractor on site. It got me thinking; who and what is really running these telecom networks and are they not much more than a branding exercise.

g_p 6y ago
This is more common than many think. Once a managed provider steps in, they want to "own" the configuration. You end up with the operator itself actually having to raise tickets with the MSP to change things on their own network. All this becomes a huge issue if there's a major outage, as the MSP might not have enough access to actually get in and do anything. Most telecoms networks are run (to some significant extent) by a managed service provider, in my experience. When O2 UK had a major core outage due to an Ericsson certificate expiry inside the core, it wasn't O2 engineers that found and fixed the issue; it was Ericsson engineers. The margins as an operator don't make it easy to keep around the deep technical skills to be an expert in the network you own.
- taurath 6y ago
  Operator margins in the US are enough to pay billions for media companies, at least. But maybe more competition is allowed in Europe.
  - g_p 6y ago
    There's definitely a "side of the pond" aspect to this - US operators pay for media companies, but they likely do so to raise their margins, and I presume they're borrowing the funds from institutional investors to do this, rather than bootstrapping their acquisitions with profits. In Europe you'll likely see far lower per-user pricing due to competition. You'll typically have 3 or 4 operators with physical networks, and a number of virtual operators providing white labelled service over the underlying networks. A standard target ARPU (average revenue per user) would probably be around 15 GBP per user per month. You'll likely get to that via contract users who you try to get on 22 GBP per month or thereabouts, and pay as you go users whose ARPU is far lower (maybe 8 or 10?) Compared with the US, consumer prices paid are incredibly cheap - expect unlimited calls and SMS, and many gigabytes of data. If you shop around you'll get even cheaper still. In the UK you'd be able to get unlimited 4G or 5G data for 25 GBP per month. Clearly the US has a much larger geography to cover, but there's definitely more competition leading to downward price pressure in Europe, in my view.

g_p 6y ago

For background context around telecoms for anyone reading this, there is an underlying difference in how telecoms networks are designed and architected - in the Telco world, links between networks were predicated on trust. Originally, telecoms networks were run by national level quasi-government operators, one per country. You interconnected with other "known entities". Even now, you likely have 3, 4 or 5 national mobile operators in any one country. They negotiate their own roaming agreements in order for you to get roaming access. It's all driven by these kinds of relationships predicated on trusting other networks. In IT, we are rapidly moving towards zero trust (due to the internet), but circuit switched (legacy) voice is still all designed to be sent over private circuits between operators who trust each other. The legacy protocols (see SS7), used to route calls between operators are functional, but also lack access control and authentication, as it's assumed only trusted parties are on the network and able to use them. Those assumptions are no longer valid, and there's a huge challenge in dealing with this - hence SMS and call interception and rerouting attacks to steal 2FA tokens etc.

est31 6y ago
Isn't the internet in the same situation, with BGP being assumed to be done between trusted parties?
- darkr 6y ago
  Yes, but increasingly traffic is strongly encrypted, with users able to exert some level of control over that encryption
- pas 6y ago
  The first rule of BGP is to filter what you get. Don't just blindly accept whatever the peer advertises. The second rule is obvious, but for the third there's also a lot of knobs for traffic engineering with BGP. And on top of that there's RPKI [ https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki/ ] I guess all of the big telcos have some homegrown ossified hacky "solution" that also serve as a minimal kind of "firewall" for SS7. (Basically I imagine that there's a lot of hardcoded rules for phone numbers, country codes and operators. Sure, they probably are an opposite of a problem for national intelligence services, after all it's easier to go by unnoticed in the noise, but they at least help with a total BGP-like hijack of a whole country code by an operator.)

kanisae 6y ago

In the past I worked at a mid level 4G provider in the US who had to deal with the larger providers on a regular basis. I was always astounded at how little they knew about their own networks. Regarding the articles statement of providers wanting an "all-in-one" solution, I have seen that in person, where management forced it, found it was horrible and then gave in and let us build the mixed vendor solution that worked well. I've personally mixed enode-b's from 2 different vendors to 3 different vendors SGW's and a different vendors PGW with no issues. The "One Throat To Choke" idea doesn't work if your business depends on that throat to operate so you end up with the vendor calling the shots instead of the business.

g_p 6y ago
The bigger they are, the harder they fall... On the whole, the technical standards should allow the kind of interoperability you described. That's the kind of fun real-world engineering that techies love. The bean-counters don't, because it's more devices needing support packages, it's more suppliers on the books, and ultimately it's probably (slightly) less profit than buying a single box. I've seen big household name operators in Europe stop even pretending they're doing the work, and straight up pass on contact details and a mobile number for the person at their tier-1 vendor partner, so you can liaise directly with them. It seems in these "5G" days even more than before, operators are retreating into the business of connectivity service, and leaving more and more for their vendor partners to do. When you're not even hiding the fact to a client that they may as well speak directly to the vendor, that says it all(!)
sgt101 6y ago
>>The "One Throat To Choke" idea but it sounds soooooo good in meetings !

baq 6y ago

> We recently asked a large European service provider why only part of their customers get IPv6 service, and how they pick which parts do or do not get such service. They could not tell us, and informed us they too would like to know woah. as a EU citizen, i'm terrified. i wanted to say surprised, but after a moment's thought, turns out it's only a moderate misalignment of expectations.

JPLeRouzic 6y ago
I retired in 2012, but at that time my employer had completely subcontracted the operation and maintenance of its mobile network to Ericsson, Huawei and Nokia. It was in France. Once I asked for a one day snapshot of all mobile data for a cooperative R&D project. The saga went on for months with repeated requests at various hierarchical levels, but to no avail. It's not that they refused, but I guess that the guys in charge simply were unable to get the requested information from the subcontractors.
iagovar 6y ago
I work for a large EU Telco and I can tell you the inside battles to get stuff done are absolutely ridiculous. Of course I work for a subcontractor too.

afarviral 6y ago

Wow ... I read this whole thing thinking it was a human. On a re-read I'm wondering what I was thinking. One thing that stands out on the account's previous posts is the paragraph lengths are quite consistent. I'm going to have to train my bot senses. I got conned. Any tips? Mindlessly reading me is not prepared...

afarviral 6y ago
Oh dang.. Was trying to reply to ttty comment that got flagged as likely not hijacked account. Off topic but interesting.
- monkey_monkey 6y ago
  Ah ok, I got terribly confused, because the sam_lowry account seemed quite legit!
  - afarviral 6y ago
    Apologies... But also: question everything. He he.
g_p 6y ago
One tell-tale sign I've noticed is sentences that don't go anywhere - the last sentence of the first paragraph starts going somewhere, then suddenly stops. If we assume people communicate for the reason of conveying some meaning, always dig for the meaning. If it's not there, it's likely some kind of vapid content-less babble... If you read critically with a view of "what are they actually saying?", you tend to spot this fairly quickly. The ending with some irrelevant babble gave the game away a bit though.
- afarviral 6y ago
  That's an excellent tip. Form an idea of what "they" are trying to convey... If it evaporates unexpectedly it might not be trying to convey anything...cause it's just an ML algo. The trouble is having clarity of thought myself, I barely know what point I'm trying to make let alone following the precise points of others half the time. These things really nail verisimilitude of chatter about an unfamiliar topic or of a smarter person, for instance, where you can't detect the BS because you assume lack of knowledge.

deleted 6y ago

[deleted]

one2three4 6y ago

This is gold. >> In reality, most service providers have not been operating on this model for decades. Driven by balance-sheet mechanics and consultants, service providers have been highly incentivised to outsource anything that could possibly be outsourced, and then some. >> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor. Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers. I think it's quite a safe bet that no operator in China went that way by buying and outsourcing from/to Western companies.

dragonelite 6y ago
Ericsson threaten to leave Sweden, if actions by Swedish government meant that they lose access to the Chinese markets and running/future contracts.
tguvot 6y ago
It's nearly impossible for telecom to deploy/configure/maintain their networks by themselves due to the scale. For example, I just googled, AT&T seems to have 67000 towers/macro cell sites. Let's say they want to update all of them to install modern 5G equipment. In many cases this equipment may come from different vendors and to deploy it might be multi-day job. Of the top of my head, about 20% of site visits fail due to various reasons (with good percent of them failing even before starting due to scheduling issues, sickness, not delivered at time equipment, etc) . How much time and people it will take to AT&T to do all the work on it sown ?
- tgv 6y ago
  They did it before, even had to wire everything together, coast to coast.
  - tguvot 6y ago
    "Before" it was simple. Now it's very complex. It's very complex exercise in large scale planning, logistics and coordination. If company tries to do it by itself, it something that will take years of work of hundreds to thousands of dedicated to this task people to accomplish. Market and customers require faster pace.

commandlinefan 6y ago

> Driven by balance-sheet mechanics and consultants It’s not just telcos...

ng55QPSK 6y ago
No only by balance-sheet mechanics, there is some politics attached. In my work place we're currently discussing moving in-house data centers to a cloud-provider. At the first glance it looks like at a higher cost (in-house cheaper). Now some balance-sheet-makeup sets in.

Semaphor 6y ago

Cynical me certainly can believe all this. But on the other hand, I’m wary of just reinforcing what I believe anyway. How trustworthy is this? There seems to be a lot of inside information, where did they get it from? Does anyone have corroborating links? All article links are either general, or US specific.

g_p 6y ago
Much of this is fairly widely known in the telecoms sector, and is "open secrets". The sector is a pretty "closed shop" though, full of trade secrets and "proprietary" things. Underneath it all though, actually it's fairly simple once you get your head around it. If you work closely with an operator, even as a client, you'll see examples of this - the number of people brought to meetings from the vendor, versus from the operator. Who answers the questions. For a public example, see the Telefonica O2 outage in the UK (and Japan, I believe) due to an Ericsson certificate outage, and how much of a role Ericsson played in this. (https://www.theregister.com/2018/12/06/ericsson_o2_telefonica_uk_outage/ https://www.theregister.com/2018/12/06/ericsson_o2_telefonic...) Press releases also give bits and pieces away: https://www.ericsson.com/en/press-releases/2019/11/orange-opts-for-ericsson-operations-engine-in-managed-services-extension https://www.ericsson.com/en/press-releases/2019/11/orange-op... https://www.mobileeurope.co.uk/press-wire/9588-three-uk-joins-telefonica-by-outsourcing-core-management-to-huawei https://www.mobileeurope.co.uk/press-wire/9588-three-uk-join... Although they might not give the level of detail you're looking for, it should hopefully corroborate things.
- Semaphor 6y ago
  No exactly what I was looking for, but a good start. Thank you!
ahubert 6y ago
Hi - author here. By all means ask around. I can only tell you that I've received may corroborating anecdotes over the past year. Many telcos even assumed I was writing about them specifically, when I wasn't! I also have a second post that has some more logos and names where I based this article on -> https://berthub.eu/articles/posts/how-tech-loses-out/ https://berthub.eu/articles/posts/how-tech-loses-out/

walrus01 6y ago

>> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor If you think this is bad in some place like the UK, you should see how ISPs and mobile network operators are set up in some countries in the developing world, where the vendor has fully captured the Telco as basically a hostage to its technical services. This is what happens when you have a mixture of institutional corruption, kickbacks and bribes, lack of local technical resources to develop a domestic network engineering talent pool, and a vendor that knows how weak the client entity's negotiating position is.

kazen44 6y ago
What also does not help is that becoming good at networking engineering at scale is something that is hard, if not impossible to learn on your own. Software Engineering is something one can teach themselves on a laptop, learning to design, build and operate networks at a large scale across geographies is simply not possible without being part of the industry. Network engineering talent is incredibly hard to come by in most regions of the world, especially if you consider that ISP networking deals with arcane technologies not really used in most "enterprise" networks. (BGP in various ways, MPLS is a big one, and arcane transports like SONET or DWDM solutions). Sure, one might be able to learn how to configure BGP, how ip works etc from their laptop using GNS3 or a couple of second hand routers/switches, but learning how to design networks at scale is completely different beast. Most people seem to enter the field by getting hired as tech support at a NOC and working their way up from there, which is kind of a grind compared to some more lucrative positions available to people who posses the technical talent.
- walrus01 6y ago
  I agree with 100% of that - a much harder problem to solve when learning real network engineering is much more capital intensive and requires real financial resources that may be beyond the reach of many.
  - kazen44 6y ago
    Also, it is a really difficult problem to solve. Most curriculums of universities and polytechnical schools only focus on the technical, theoretical aspects of network engineering, but getting hand-ons experience in actually running an ISP network is very, very difficult. The market for people who want to get into this line of work is also very small compared to software engineers, programmers and system administrators. The work is usually high risk, technically very complex and it can be very stressful. (It being vital infrastructure, and breaking something can result in catastrophe for your customers in some cases) I sometimes wonder what will happen if we have no one left to maintain the systems so many layers of software and systems depend upon.

magwa101 6y ago

Interesting insights on who operates these networks. However author equates US eavesdropping to Chinese eavesdropping. That is simply not serious. The "ship has left the harbor" view of existing infrastructure is not sensible, ownership matters.

ng55QPSK 6y ago

Ehm. You all are aware that 5G was created with the expectation that in the future all networks (and core functions) will be cloud-based? The last remaining HW will be the physical antenna and some PA/LNA and some local signal processing. You connect fiber to that and everything else is a operator-as-a-service model - running on AWS/Azure/GCP.

walrus01 6y ago
You're aware that telcos have massive amounts of physical stuff needed to make that happen first? You can't "cloud" hand wave away things like massive metro scale dark fiber networks for backhaul. Or things like inter city long haul DWDM networks. I assure you there's a lot more going on hidden behind the scenes of a modern ISP or 4G/5G carrier than just some antennas and software in a VM somewhere.
- ng55QPSK 6y ago
  This will not happen this year. But in the long run you don't need much own HW to be an operator. And fiber connectivity is (as the startpoint of this comments) outsourced in large scale.
  - walrus01 6y ago
    "in the long run you don't need that much HW to be an operator" - have you ever worked for a facilities based ISP or Telco and visited the interiors of dozens of different POPs? Please do so and then come tell us all about how telecom infrastructure on a national and global scale isn't composed of massive amounts of hardware all over the place.
    - ng55QPSK 6y ago
      I work in a team that created 5G. And around ~2014 it was clear: operators want to go away from own HW.
      - 6y ago

not1ofU 6y ago

Related: This white paper was published within the last 2 weeks relating to "5G Network Slicing" Quote from the author: “Currently, the impact on real-world applications of this network slicing attack is only limited by the number of slices live in 5G networks globally. The risks, if this fundamental vulnerability in the design of 5G standards had gone undiscovered, are significant. Having brought this to the industry’s attention through the appropriate forums and processes, we are glad to be working with the operator and standards communities to highlight this issue and promote best practice going forward.” PDF can be downloaded from here: https://info.adaptivemobile.com/5g-network-slicing-security https://info.adaptivemobile.com/5g-network-slicing-security

Clewza313 6y ago

> Since the early 2000s at least, most billing has been outsourced. This works by sending all Call Detail Records (CDRs) to a third party, often from Israel or China. This is quite misleadingly written: telcos are not shipping reams of CDRs to some cubicle farm in Haifa or Chongqing. Yes, almost every telco outsources its billing software to other companies, notably Amdocs (founded in Israel, now HQ's in the US). However, billing info is some of the most sensitive data a telco has for both privacy and commercial reasons, so that software always runs in a closed environment from where it cannot dial home. Historically that's been on-prem, it's slowly moving to the Cloud but even there it's going to be firewalled off very carefully.

waheoo 6y ago
Oh yea, super secure, their outsourced security team is all over it.
sgt101 6y ago
There's a big gap between what happens in the larger telcos that run infrastructure, and the white label operators.
- tguvot 6y ago
  White label operators can't usually afford Amdocs solution from one side. From the other side, Amdocs doesn't have a server farm in Israel to do CDR processing :) Everything happens either on client premises (and Amdocs can deliver end-to-end solution, including buildout of complete datacenter if it required by client) or "in the cloud"
Thr0wawayDocs 6y ago
Let's not oversell the fact that Amdocs' official HQ is in the US: It's an Israeli company through and through. It did more development in the US back in 2001 than it does now: Today, their US footprint is mostly customer sites. And you will find people brought in from Israel everywhere. It's always been bad enough that managers that don't speak Hebrew knew they were always at a disadvantage. Your best bet for saying it's not an Israeli company is to say that a whole lot of R&D is being done in other countries with cheaper labor. While it's true that the installations are on-prem (having been to quite a few of those), Amdocs business model isn't about dropping code and going away: They are so embedded with your typical deployment that there's plenty of opportunity to exfiltrate data. Sending every CDR to Haifa? Probably not: The Sysadmins on your typical large telco are iffy, but not that iffy. And carefully firewalled? The talent was never great, and the security practices were never all that serious: I've been handed production shells that I had no business having, because it was convenient at the time. Once again, I'd say that the best argument to claim that there's no data exfiltration is that the people writing the code aren't good enough to do this under the customer's nose.

lifeisstillgood 6y ago

>>> what remains in the other half are IT Architects who do not get closer to actual operations than an Excel sheet or a Visio diagram. the only light point in an otherwise depressing read

carlosf 6y ago

> In a modern telecommunications service provider, new equipment is deployed, configured, maintained and often financed by the vendor. Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers. > The host service provider often has no detailed insight in what is going on, and would have a hard time figuring this out through their remaining staff. Rampant outsourcing has meant that most local expertise has also left the company, willingly or unwillingly. 100% reflects my experience working in Huawei BR a few years ago. Carriers are mostly customer facing companies and very limited technically. Our customer (million + subscribers BR carrier) often hadn't the slightest idea how their own network was built and worked. Banning Huawei is absolutely impossible, at least in Brazil.

tguvot 6y ago

I worked for a bunch of years in biggest Israeli company that is selling OSS/BSS and related outsourcing services to telecoms (those who can afford it's solutions), and had some first hand experience with them >Since the early 2000s at least, most billing has been outsourced. This works by sending all Call Detail Records (CDRs) to a third party, often from Israel or China. A CDR stores who called whom and for how long. More data might be attached, for example the location of the customer, or where the customer was roaming abroad etc. Don't know about software from China, but the one that we sold doesn't send anything back to Israel. There are a lot of rules and restrictions upon CDRs and we had a bunch of training with regards to it. Everything is running on client site, usually on hardware deployed by us at their data centers and managed by dedicated team of people who relocate to live next to the client in order to provide 24/7 support of the systems on site >Typical service providers have hundreds of thousands of network elements. Surprisingly perhaps, many of these are actually maintained manually (!). Thousands of networking engineers labour to keep all this infrastructure operating well. This is a mix of half-truths and lies. None of the operators have thousands of people to manually configure day-to-day network stuff. Operators have rather sophisticated automation systems (aka OSS) that deal with provision and configuration of everything in their networks. Or almost everything. Any given operator whose life span is a decade or two today has a boatload of equipment (thousands of different types of hardware from same amount of vendors). In many cases this equipment was bought and deployed 10 or 20+ years ago. Companies that made it do not exist for many years. This hardware can't be replaced with anything, because nobody does this type of systems anyway. Those systems tend to have proprietary interfaces and in many cases can be managed only through Element Manager which can be managed only manually through some ancient windows or java application. >Meanwhile, modern large scale internet companies (like Google, Netflix, Facebook) have automated all such maintenance. Automation in this context means that no configuration states are edited manually but instead, entire networks get provisioned and configured from central templates. >With such automation, small teams of engineers can control and operate vast networks with relative ease - especially if good use is made of continuous integration and real life testing. I also worked for a while in one of FAANGS. They have it easy: all the hardware with modern with nice interfaces. You can actually automate it. Also their networks are much-much smaller compared to mid-sized telecom, much simpler and much more homogeneous. Automation that FAANG I worked for was a joke compared to automation systems that run telecom networks. My job was near network engineering team and during conversations they admitted that what they have is crap. I believe that at one point of time they considered to buy telecom level OSS system but bailed out because they couldn't get a source code .

rijoja 6y ago

"Just to let that sink in, Huawei (and their close partners) already run and directly operate the mobile telecommunication infrastructure for over 100 million European subscribers". Which providers are using and which are not using Huawei?

erikerikson 6y ago

The author would find Pivotal Commware[0] interesting. Not only stateside but offering advances in hardware efficiency and durability. Full disclosure: employee, soon investor [edit: also, they are hiring] [0] https://pivotalcommware.com/ https://pivotalcommware.com/

BenoitP 6y ago

> One even went so far as to state during an all-hands meeting with technical staff that ‘running a communication network’ was by no means a core competence for them. This is an outraging but very widely spread phenomenon. No industry is spared from the MBA hawks. Everything now is rent-seeking and moat building. Innovation has been packaged away and can only happen when the market makers say it can. What can an engineer do about that?

quadrifoliate 6y ago
> What can an engineer do about that? Lots! - Name the companies in question - Stop working for them, and start working for companies that favor engineering expertise - As a consumer, advertise the good companies and call bullshit on the bad ones - Raise awareness about these practices among your elected political representatives and their constituencies If all of their skilled engineers leave, the bad company cannot run only on the basis of MBAs juggling balance sheets. Unfortunately most of my fellow engineers are far more likely to sit around blaming “the MBAs” over drinks than take any of the above actions.
- aembleton 6y ago
  > If all of their skilled engineers leave, the bad company cannot run only on the basis of MBAs juggling balance sheets. The MBAs can just outsource the work.

quelsolaar 6y ago

My assumption is that all intelligence services in all countries would love to have access to fully compromised networks. They spend all their time thinking about how to access information, so they would be fairly incompetent not to consider this. Does that mean that all systems are compromised? No, because there are risks associated with tapping in to these systems. Partly it depends on if they have access to the systems, but mostly on the possible blow-back if they get caught. Example: Swedens FRA (NSA equivalent) could in theory ask Ericsson (a Swedish company), to install a backdoor. But, Sweden has a fairly free press, and there are good chances that someone would leak this information. If it got leaked it would be a major scandal that could go as far as toppling the government and destroy one of Swedens most important export companies. Its very risky, and its a risk no one wants to take, so the parts made in sweden are probably not compromised. China, on the other hand has almost no risks associated with adding backdoor. No free press, hard suppression of whistle blowers, and since most foreign intelligence services already assume the equipment is compromised, there is no real reputational damage either. I assume they are all compromised, why wouldn't they be? The US is somewhere in between. Sometimes companies are compromised by intelligence services, but much more often I think its employees. Why try to change Tim Cooks stance on privacy, when all you need to do is find one Apple employee, willing to take a sack of money to "do their country a great service"?

marsven_422 6y ago
"Sweden has a fairly free press" That's so wrong it hurts! All our press are dependent on government "presstöd" aka handouts.
- Swenrekcah 6y ago
  The press can be free regardless. If the allocations are according to some objective metrics then I don’t see a problem.
  - whatshisface 6y ago
    Objective metrics can be as biased as subjective metrics.

ksec 6y ago

Probably Off Topic, but may be a chance if anyone within the industry might know. What are the current / purposed patent licensing terms of NR-U; finalised and related with 3GPP Rel 16 are going to be? Specific to standalone NR-U ( As in MultiFire in 4G. ) which could compete with WiFi 6e.

deleted 6y ago

[deleted]

rcarmo 6y ago

There is an amazing amount of FUD in this article. I have worked in the telco industry for the better part of 30 years, and am back on it now after a 5-year hiatus in cloud computing. Before I "left" there was certainly a trend towards outsourcing and large "swaps" of radio gear (Nortel-Ericsson in my case, and Motorola-Huwawei at a direct competitor, to quote only two examples), but there was no way in $UNDERWORLD that we would let a vendor have direct access to our gear unsupervised (be it Cisco, Ericsson, whatever). Remote troubleshooting was possible, but usually via jump boxes and VNC (only very seldom we would let anyone VPN in, and even then it was only to sub-sections of the network). Nothing left our O&M network. Nothing came in, either, because upgrades were rolled out from internal servers. And it is still very much the same thing today. Although there are outsourcers and vendors who work alongside core staff in my telco customers (like myself now), we don't have access to anything but lab or dev environments, and even then mostly with MFA and very stringent limitations. Outsourced staff _does_ do field service of various kinds, and they do have access to base stations, DSLAMs and various other physical infrastructure, but that's usually done with (usually much cheaper) local technicians and not vendor staff. There are certifications for those. The reality is that most telco services are being "automated out" and moved to virtualized stacks that are easier to manage. And yes, VoIP on the core (no more SS7 if anyone can help it) and Kubernetes everywhere... But what I found to be really weird was the notion of outsourcing billing. Besides being a GDPR nightmare (and I'm in Europe, like the author, so I find it doubly unsettling), that was only done "off-prem" when all companies involved were in the same group (which was customary when fixed and mobile operators were separate). These days billing is, comparatively, greatly simplified (thanks to flat fees, real-time billing systems for prepaid and streamlined bundles), so the only data that actually leaves the BSS core goes to the (smaller and smaller) printing facilities. So I would take it all with a massive dollop of salt.

thu2111 6y ago
Which telco is that? Is it UK based by any chance? Article says things are different in the UK.

varispeed 6y ago

Isn't outsourcing essentially a tax avoidance? Companies look for cheap labour overseas because they don't want to pay local rates, which typically include higher tax and cost of adhering to any regulations. If a person tried to do this - for example outsourced their bank account to tax haven and asked their salary to be paid there, the authorities would be all over it. So now the fact that companies are not only not hiding the fact they are trying to bypass the system, they also are lobbying governments to make it easier! I for a second don't believe that there is no money under the table involved. Why otherwise politicians would choose to funk up the local population to appease a big corporation? Bottom line is that outsourcing should only be possible if it was not possible to create a product locally or companies should pay any difference in tax locally, so that people who got put out of jobs because of this can at least get benefits. And finally I don't understand why even discussing doing any deals with China does not amount to farting in a room.

cptskippy 6y ago
Outsourcing is done for many reasons. Sometimes companies have more projects than staff and outsourcing offers them flexible bandwidth. The downside to this approach is that it puts stress on staff who have to train or maintain the outsourcer.
- varispeed 6y ago
  I would understand outsourcing to countries that value human rights, have workers' protections in place and so on. But simply going for the cheapest possible option, where you get forced labour and children making your product simply wrong and indefensible. We at least should lobby online and offline shops to show country of origin on the labels. I wish if I could go on Amazon and have ability to filter out anything that comes from China. If you want to buy something responsibly it is very time consuming and some manufacturers go to great length to hide where their products are really made. Those people who exploited labour in Asia now got filthy rich and they are in position to shush any politician looking to put a stop to this or make sure such filters would not be implemented.

elzbardico 6y ago

Just another instance of how Harvard Business School completely f*ing up things. Future historians will have trouble understanding how we let MBAs destroy our civilization.

phreeza 6y ago

What I don't get in this entire conversation is how is 5G different from any other transport layer? Can't secure communication be achieved by encrypting the communications at a higher level?

mensetmanusman 6y ago
Metadata still exists in that framework.

mmaunder 6y ago

Steel production capability is considered strategically important in case we go to war, and it has been so since World War II. Steps have been taken to retain domestic production capacity for this reason. Until we have a planet of one people and one nation, we’re stuck thinking this way about things that are critically important, should we find ourselves at war with a former partner. Tech sovereignty has become such a thing. And the bad news is that we have lost. I’ll leave others to debate why, but we can’t manufacture our own chips, we cant make our own telco networks, and the cloud systems that provide back end services are almost lost. The state of play here is dire for the US and it’s strategic partners. I’d say that surveillance is less worrying than the simple fact that a potential future adversary has an off switch for these things that they can toggle at will: no more chips, no more telco products and no more cloud services - now, let’s have that South China Sea conversation one more time...

peter_d_sherman 6y ago

5G Vs. Starlink Which will win? ? Or, more likely, does the future hold a coming broadband internet connectivity price war? ? And if so... ...who will be the "last IP address standing"? ?

Foivos 6y ago

This is my experience interacting with mobile telcos as well. Even to get some simple logs from a base station you need to either ask an Ericsson engineer or, worse, wait for the Telco employee with the relevant knowledge to find time to do it. Telco employees with such knowledge are very few compared to the amount of workload they have to do, so it is hard to get them to dedicate time to help you.

philjohn 6y ago

My concern wouldn't be with data being exfiltrated, seeing as so much is E2E encrypted these days. But what if there is a remote kill switch - taking down a cellular network could cause a whole heap of problems in the 21st century.

dd_roger 6y ago

I work in OT security in an industry completely unrelated to Telecoms but which is also a matter of national security and everything in this blog post doesn't really sound believable from my experience dealing with sensitive infrastructures. Different industries, different countries, etc. I get that things can differ a bit, but going from "the infrastructure is airgapped" (in my industry) to "the infrastructure is managed remotely by a foreign entity" (as claimed by the author) seems too big to be true. Not gonna lie, I'm a bit sceptical about the veracity of some of these claims.

tim333 6y ago

I wonder if it really matters if networks are insecure. I generally connect to the internet via whatever hotspot is available or 4G if not with no expectation that it's fully secure. That's what https and other encryption is for. And I don't generally have anything to hide - there are a bunch of tools like tor and VPNs for people who do. That's probably the way forward - secure tech on your device rather than trusting to the kindness of strangers. Talking of phone network security one thing that does piss me off is my phone company just transferred my phone number of 10 years to fraudsters who presumably called customer support with some sob story. You'd think they could have some standards to stop that like at least sending an email to your usual address saying "There's been transfer request - you good?"

spixy 6y ago
Ok, but what about SMS, MMS, phone calls, getting your approximate position, ...? What if they decided to shut down the whole network because of some reason? Then https wont help you.
- tim333 6y ago
  You could drop SMS, MMS and phone calls and use something like whatsapp instead, which a lot of people seem to be moving towards anyway.

WhyNotHugo 6y ago

I can’t help notice that we always keep relying on closed-source for these kind of things, and then distrust providers because we don’t know what that secret code does. Am I the only one noticing that the obvious solution is to stop using closed-source tools for such delicate infrastructure that’s meant to serve the general population? I honestly can even grasp how we got into this mess in the first place: publicly funded software for public infrastructure that has such delicate security implications should obviously be open source only.