36 ms·

The Miserable State of Modems and Mobile Network Operators

181 points by hasheddan 2y ago

PaulHoule 2y ago
> The number one reason we see customers opt to not use cellular when > another connectivity option is on the table is due to concerns around > cost and reliability. For "reliability" read "availability". Cellular networks are profitable because carriers serve customers that are easy to serve and leave the rest alone. Vehicle 2 Vehicle communications mediated by cellular is a non-starter because people drive a lot in places where cell phones don't work. Wireless internet has been a false hope that's spoiled realistic plans for universal service because it's only affordable in markets that are already served. Large-scale government and corporate IoT programs find that 20% or more of stations are in places without service, etc.
- amluto 2y ago
  On top of this, there are markets that are not well covered due to local politics. I can go just a few blocks away into a neighborhood that has plenty of density to be interesting to the major carriers but has basically no usable cellular data. A couple years ago, there was a big fight between Crown Castle (big cellular network contractor) and the local government. Crown Castle wanted to build cell towers. The residents wanted service but didn’t want cell towers in their back yard. The government tried, and failed, to avoid issuing permits, because federal law sensibly prevents local government from effectively prohibiting the construction of cell towers. Win? No! The local government is very proud of itself for having pulled a rabbit out of its hat. It turns out that a cell tower is useless without backhaul, and the government rolled out so much red tape that Crown Castle determined that they would never recover the costs of installing the backhaul and abandoned the entire project. The local government somehow considers this to be a victory.
  - PaulHoule 2y ago
    What I remember in the 3G era was that my MVNO phone [1] worked great in medium sized cites like Rochester, NY but poorly in places with lower density and even worse in the highest density places like NYC and Hollywood. 4G had the capacity to serve super density urban areas really well. [1] I've got a suspicion that Tracfones ride the back of the bus and get worse service than you'd get with a premium-priced plan.
    - lotsofpulp 2y ago
      https://www.reddit.com/r/NoContract/comments/1cyfjpp/data_prioritization_policies_of_the_carriers_and/ https://www.reddit.com/r/NoContract/comments/1cyfjpp/data_pr... > Other TracFone brands are up in the air. I have gotten reports of priority data (and seen it myself on SafeLink, a TracFone brand) but I’ve also gotten reports of deprioritized data, complete with screenshots. It seems to be random what people get so if you absolutely require priority data, it might be best to look outside of the TracFone brands. https://en.wikipedia.org/wiki/QoS_Class_Identifier https://en.wikipedia.org/wiki/QoS_Class_Identifier
Delphiza 2y ago
As part of our IoT offerings, we tried quite hard to build devices with mobile connectivity about 7 years ago. This was just as low power 5G, NB-IoT and similar technologies were going to become a thing. We gave up because it was too much effort for little return, and it was better to focus on doing things with data, rather than collecting it. Even getting prototypes up with powered fanless PCs (SBCs) and Mini-PCIe or M.2 modems was harder than it should have been - you would think it would be easy with off-the-shelf devices and drivers. Small Internet connected devices are still needed, despite the perception that IoT is dead. Mobile networks and the modem supply chain are definitely holding the market back. On the plus side, for stationary powered devices most people are happy to connect to wi-fi. For low power devices, LoRa, with private gateways, seems to be a standard. Mobile that is used in outdoor vehicle and asset tracking is still stuck with fighting with modems as per OP.
- jon-wood 2y ago
  You might want to take another look, I went through a similar process a couple of years ago, and am now doing it again because our original equipment supplier decided they'd like a life (RIP PC Engines). There's a thriving industry of companies providing fanless ARM based machines either shipping with LTE modems, or with M2 & SIM slots on the board to provide your own. We install in industrial environments where an accessible internet connection is far from a given, having LTE on all our devices means that we can almost always give the device a way to call home. I can strongly recommend Compulab's devices, which you can purchase as a fully assembled unit that just needs a SIM card put in it (I can't recommend the Linux distribution they run, but you're free to flash them with whatever you'd like).
  - sumtechguy 2y ago
    > There's a thriving industry of companies providing fanless ARM based machines either shipping with LTE modems, or with M2 & SIM slots on the board to provide your own When I did this about 10 years ago we had quite a number to pick from. Build your own was possible. But that was only because of our organization had the capability and expertise to do it. We settled on 3 off the shelf ones. That was dependent on cost and number of I/O the customer was needing. DiY basically is 'first make your own computer with the ARM/MIPS/x86 chipsets' then 'spin your own special firmware for it'. Then 'build your own ground up linux distro or similar with compiler chain and SDK to work with it'. You may be able to get someone to sell/give you a reference layout/SDK. Then after all of that. You are now ready to add in a modem. Also prepare for the certifications of all the mobile networks you want to run on. Plus software for you to interact with the cell modem. Oh also you need to work on getting yourself provisioned correctly in the mobile networks. When you do for five devices it is a couple hours of playing with an API. But you probably want hundreds of them so be prepared for managing that, plus billing. Oh also you need to manage EoL for your parts. Many IoT installations are looking at you hanging around for 10+ years.
kmeisthax 2y ago
> The patents are referred to as Standards Essential Patents, or SEPs. Because of this arrangement, a vendor like Nordic probably can’t open source their modem firmware even if they wanted to. Why would a SEP license forbid publishing modem firmware? The IP[0] involved with the modem firmware is copyrighted and trade secret material; publishing the firmware does not (and cannot) grant a patent license to those licensed standards-essential patents. It only imperils your implementation of the modem firmware, not the whole patent spec. To wit, Cisco publishes an H.264 codec with BSD-licensed code, even though H.264 is under a shitload of patents that they are licensing. OpenH264 does ship with an additional patent sublicense that applies if you are using their build and installing it in a specific way, but that is particular to MPEG-LA's licensing structure[1]. Publishing an implementation of a patent you are licensing does not automatically sublicense the patent. [0] Laws that grant the ability to dictate the conduct of your competitors [1] Cisco is abusing(?) the MPEG-LA royalty fee cap to shield FOSS entities from having to take a license, basically, by paying for them.
- gjsman-1000 2y ago
  I suspect a big reason is that there’s middleware: some of this stuff is so complex, that like video games, there’s probably a cottage industry of companies selling partially implemented bits and pieces, or licensing across companies of source code, or companies only licensing the patents if you agree to never release code. There’s also the IP angle: if you do manage to figure it out, there’s no honor helping Huawei.
  - baby_souffle 2y ago
    > I suspect a big reason is that there’s middleware I’d bet it’s simpler. Saying “yes” to the ‘can we share this outside of this company?’ Question means a lot of work and meetings and red tape and somebody is going to have to rope in legal and … and … and … Or, “no, get back to work”.
    - gjsman-1000 2y ago
      True, but who owns most of the patents? Some of the most litigious companies on earth (Qualcomm, Broadcom, Cisco). How many legal cases are more expensive than patent litigation? Also, it’s very simple: Imagine you approach Qualcomm. The script probably goes something like this: “Sure, we’ll license you that patent, for $0.92 per device, a $25K per month maintenance fee, and compliance with our terms and conditions - chiefly, don’t license this patent to anyone else, and don’t share the source code of your implementation.”
aftbit 2y ago
Wow that was quite the technical deep dive! I bet there's not one person in the world who understands all the interactions in this system.
- linuxguy2 2y ago
  I too greatly enjoyed the article. Love those deep troubleshooting write-ups!
awelkie 2y ago
I think the miserable state of cellular modems ultimately comes from the power dynamics of private spectrum. The 3GPP protocols are complex because they're not really beholden to the implementers, but rather to the network operators. The operators are fine with additional complexity if it serves their interests, and the cost will just be passed on to the OEMs. The network operators have all the power because they're the ones with the licenses. WiFi tends to be simpler in part because the protocol authors are working more in the interest of the implementers, since it's really they who decide whether to adopt or not. Obviously a gross simplification but I think it's at the heart of the problem.
- dilyevsky 2y ago
  Vendors wrote those standards though, not the operators. They are complex because it’s a defensive moat against competition imho
  - tinktank 2y ago
    They are complex because, usually, they are a union of everything every vendor in the consortium has done and/or wants to do.
    - astrobe_ 2y ago
      Looking at the overview of the architecture on the network side [1] tells the tale, I think. My take is that telecom operators forcefully had to evolve from Morse and copper to IP and radio, hacking stuff left and right every step of the way. [1] https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#Architecture https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#Archit...
      - dilyevsky 2y ago
        Long time ago i worked on WiMAX devices which was a competing standard for the term “4g” from ieee and while it was simpler it wasn’t a whole lot simpler and performance sucked, so there’s that
hasheddan 2y ago
Hey folks, author here. Appreciate the comments and discussion on the post! Happy to continue the discussion or answer any follow-on questions folks have about our investigation and resolution.
- sharpshadow 2y ago
  Good read thanks. To the point of having a open platform, they would probably lose their monopoly. I could then just spin up my own mobile network and provide service like a wifi network with any credentials and do what I want. I guess there needs to be a big initial effort with lots of maintaining afterwards. If everything goes as expected the AIs could provide that in the future.
- nimish 2y ago
  Would just be easier to implement an open source one from scratch given that most IOT modems are just (poorly written) software with an SDR anyway. CEVAwaves or RivieraWaves certainly is. If it can be done for BLE, 802.15.4 and WiFi, it can be done for cellular; ORAN is enough for base-stations so it's not totally secret. But at the end of the day, aside from the stupid ITU protocols, LTE/5g isn't rocket science. It's bog standard OFDMA radio -- nothing fancy these days. I ended up reverse engineering parts of Sony's Altair derived modems but got bored, maybe I should publish that
cellhn 2y ago
[dead]
ForOldHack 2y ago
The complaint is that people cannot connect, and that hardware vendors dont want to go down the pit of other peoples sh*(blobs). Magnificent article. Clearly shows his expertise going down the sh*/rabbit hole, the more I read, the more dismal it got. Clearly the way to get around the technical side of the DNS bug, is to do what cell providers do: Start the device in the office of the provider, and let the DNS resolve there, and if it does not work, get a ticket started right then and there. Out in the field... use another devices Wifi, and do the same thing. If you cannot get cell phone service or Wifi, then ( I literally took about 45 mins on the phone to T-Mobile, which immediately refused to take responsibility, and also refused to put anything in writing: Then their recommendation is to get a satellite link with, StarLink!. I would rather be slapped in the face with a cold fish. Cell phone service providers all let this happen, until of course they almost all sold out to T-Moble. ( LifeLine, Assurnace, etc.. )
netfortius 2y ago
It's [almost] always DNS :)
jiehong 2y ago
First, it was quite interesting! Second, it made me think about the new Apple C1 modem. They must have had to comb through those specs as well, and it makes me wonder if will allow them to better integrate the modem in the kernel in the end?

181 points by hasheddan 2y ago

> The number one reason we see customers opt to not use cellular when > another connectivity option is on the table is due to concerns around > cost and reliability. For "reliability" read "availability". Cellular networks are profitable because carriers serve customers that are easy to serve and leave the rest alone. Vehicle 2 Vehicle communications mediated by cellular is a non-starter because people drive a lot in places where cell phones don't work. Wireless internet has been a false hope that's spoiled realistic plans for universal service because it's only affordable in markets that are already served. Large-scale government and corporate IoT programs find that 20% or more of stations are in places without service, etc.

amluto 2y ago
On top of this, there are markets that are not well covered due to local politics. I can go just a few blocks away into a neighborhood that has plenty of density to be interesting to the major carriers but has basically no usable cellular data. A couple years ago, there was a big fight between Crown Castle (big cellular network contractor) and the local government. Crown Castle wanted to build cell towers. The residents wanted service but didn’t want cell towers in their back yard. The government tried, and failed, to avoid issuing permits, because federal law sensibly prevents local government from effectively prohibiting the construction of cell towers. Win? No! The local government is very proud of itself for having pulled a rabbit out of its hat. It turns out that a cell tower is useless without backhaul, and the government rolled out so much red tape that Crown Castle determined that they would never recover the costs of installing the backhaul and abandoned the entire project. The local government somehow considers this to be a victory.
- PaulHoule 2y ago
  What I remember in the 3G era was that my MVNO phone [1] worked great in medium sized cites like Rochester, NY but poorly in places with lower density and even worse in the highest density places like NYC and Hollywood. 4G had the capacity to serve super density urban areas really well. [1] I've got a suspicion that Tracfones ride the back of the bus and get worse service than you'd get with a premium-priced plan.
  - lotsofpulp 2y ago
    https://www.reddit.com/r/NoContract/comments/1cyfjpp/data_prioritization_policies_of_the_carriers_and/ https://www.reddit.com/r/NoContract/comments/1cyfjpp/data_pr... > Other TracFone brands are up in the air. I have gotten reports of priority data (and seen it myself on SafeLink, a TracFone brand) but I’ve also gotten reports of deprioritized data, complete with screenshots. It seems to be random what people get so if you absolutely require priority data, it might be best to look outside of the TracFone brands. https://en.wikipedia.org/wiki/QoS_Class_Identifier https://en.wikipedia.org/wiki/QoS_Class_Identifier

Delphiza 2y ago

As part of our IoT offerings, we tried quite hard to build devices with mobile connectivity about 7 years ago. This was just as low power 5G, NB-IoT and similar technologies were going to become a thing. We gave up because it was too much effort for little return, and it was better to focus on doing things with data, rather than collecting it. Even getting prototypes up with powered fanless PCs (SBCs) and Mini-PCIe or M.2 modems was harder than it should have been - you would think it would be easy with off-the-shelf devices and drivers. Small Internet connected devices are still needed, despite the perception that IoT is dead. Mobile networks and the modem supply chain are definitely holding the market back. On the plus side, for stationary powered devices most people are happy to connect to wi-fi. For low power devices, LoRa, with private gateways, seems to be a standard. Mobile that is used in outdoor vehicle and asset tracking is still stuck with fighting with modems as per OP.

jon-wood 2y ago
You might want to take another look, I went through a similar process a couple of years ago, and am now doing it again because our original equipment supplier decided they'd like a life (RIP PC Engines). There's a thriving industry of companies providing fanless ARM based machines either shipping with LTE modems, or with M2 & SIM slots on the board to provide your own. We install in industrial environments where an accessible internet connection is far from a given, having LTE on all our devices means that we can almost always give the device a way to call home. I can strongly recommend Compulab's devices, which you can purchase as a fully assembled unit that just needs a SIM card put in it (I can't recommend the Linux distribution they run, but you're free to flash them with whatever you'd like).
- sumtechguy 2y ago
  > There's a thriving industry of companies providing fanless ARM based machines either shipping with LTE modems, or with M2 & SIM slots on the board to provide your own When I did this about 10 years ago we had quite a number to pick from. Build your own was possible. But that was only because of our organization had the capability and expertise to do it. We settled on 3 off the shelf ones. That was dependent on cost and number of I/O the customer was needing. DiY basically is 'first make your own computer with the ARM/MIPS/x86 chipsets' then 'spin your own special firmware for it'. Then 'build your own ground up linux distro or similar with compiler chain and SDK to work with it'. You may be able to get someone to sell/give you a reference layout/SDK. Then after all of that. You are now ready to add in a modem. Also prepare for the certifications of all the mobile networks you want to run on. Plus software for you to interact with the cell modem. Oh also you need to work on getting yourself provisioned correctly in the mobile networks. When you do for five devices it is a couple hours of playing with an API. But you probably want hundreds of them so be prepared for managing that, plus billing. Oh also you need to manage EoL for your parts. Many IoT installations are looking at you hanging around for 10+ years.

kmeisthax 2y ago

> The patents are referred to as Standards Essential Patents, or SEPs. Because of this arrangement, a vendor like Nordic probably can’t open source their modem firmware even if they wanted to. Why would a SEP license forbid publishing modem firmware? The IP[0] involved with the modem firmware is copyrighted and trade secret material; publishing the firmware does not (and cannot) grant a patent license to those licensed standards-essential patents. It only imperils your implementation of the modem firmware, not the whole patent spec. To wit, Cisco publishes an H.264 codec with BSD-licensed code, even though H.264 is under a shitload of patents that they are licensing. OpenH264 does ship with an additional patent sublicense that applies if you are using their build and installing it in a specific way, but that is particular to MPEG-LA's licensing structure[1]. Publishing an implementation of a patent you are licensing does not automatically sublicense the patent. [0] Laws that grant the ability to dictate the conduct of your competitors [1] Cisco is abusing(?) the MPEG-LA royalty fee cap to shield FOSS entities from having to take a license, basically, by paying for them.

gjsman-1000 2y ago
I suspect a big reason is that there’s middleware: some of this stuff is so complex, that like video games, there’s probably a cottage industry of companies selling partially implemented bits and pieces, or licensing across companies of source code, or companies only licensing the patents if you agree to never release code. There’s also the IP angle: if you do manage to figure it out, there’s no honor helping Huawei.
- baby_souffle 2y ago
  > I suspect a big reason is that there’s middleware I’d bet it’s simpler. Saying “yes” to the ‘can we share this outside of this company?’ Question means a lot of work and meetings and red tape and somebody is going to have to rope in legal and … and … and … Or, “no, get back to work”.
  - gjsman-1000 2y ago
    True, but who owns most of the patents? Some of the most litigious companies on earth (Qualcomm, Broadcom, Cisco). How many legal cases are more expensive than patent litigation? Also, it’s very simple: Imagine you approach Qualcomm. The script probably goes something like this: “Sure, we’ll license you that patent, for $0.92 per device, a $25K per month maintenance fee, and compliance with our terms and conditions - chiefly, don’t license this patent to anyone else, and don’t share the source code of your implementation.”

aftbit 2y ago

Wow that was quite the technical deep dive! I bet there's not one person in the world who understands all the interactions in this system.

linuxguy2 2y ago
I too greatly enjoyed the article. Love those deep troubleshooting write-ups!

awelkie 2y ago

I think the miserable state of cellular modems ultimately comes from the power dynamics of private spectrum. The 3GPP protocols are complex because they're not really beholden to the implementers, but rather to the network operators. The operators are fine with additional complexity if it serves their interests, and the cost will just be passed on to the OEMs. The network operators have all the power because they're the ones with the licenses. WiFi tends to be simpler in part because the protocol authors are working more in the interest of the implementers, since it's really they who decide whether to adopt or not. Obviously a gross simplification but I think it's at the heart of the problem.

dilyevsky 2y ago
Vendors wrote those standards though, not the operators. They are complex because it’s a defensive moat against competition imho
- tinktank 2y ago
  They are complex because, usually, they are a union of everything every vendor in the consortium has done and/or wants to do.
  - astrobe_ 2y ago
    Looking at the overview of the architecture on the network side [1] tells the tale, I think. My take is that telecom operators forcefully had to evolve from Morse and copper to IP and radio, hacking stuff left and right every step of the way. [1] https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#Architecture https://en.wikipedia.org/wiki/IP_Multimedia_Subsystem#Archit...
    - dilyevsky 2y ago
      Long time ago i worked on WiMAX devices which was a competing standard for the term “4g” from ieee and while it was simpler it wasn’t a whole lot simpler and performance sucked, so there’s that

hasheddan 2y ago

Hey folks, author here. Appreciate the comments and discussion on the post! Happy to continue the discussion or answer any follow-on questions folks have about our investigation and resolution.

sharpshadow 2y ago
Good read thanks. To the point of having a open platform, they would probably lose their monopoly. I could then just spin up my own mobile network and provide service like a wifi network with any credentials and do what I want. I guess there needs to be a big initial effort with lots of maintaining afterwards. If everything goes as expected the AIs could provide that in the future.
nimish 2y ago
Would just be easier to implement an open source one from scratch given that most IOT modems are just (poorly written) software with an SDR anyway. CEVAwaves or RivieraWaves certainly is. If it can be done for BLE, 802.15.4 and WiFi, it can be done for cellular; ORAN is enough for base-stations so it's not totally secret. But at the end of the day, aside from the stupid ITU protocols, LTE/5g isn't rocket science. It's bog standard OFDMA radio -- nothing fancy these days. I ended up reverse engineering parts of Sony's Altair derived modems but got bored, maybe I should publish that

cellhn 2y ago

[dead]

ForOldHack 2y ago

The complaint is that people cannot connect, and that hardware vendors dont want to go down the pit of other peoples sh*(blobs). Magnificent article. Clearly shows his expertise going down the sh*/rabbit hole, the more I read, the more dismal it got. Clearly the way to get around the technical side of the DNS bug, is to do what cell providers do: Start the device in the office of the provider, and let the DNS resolve there, and if it does not work, get a ticket started right then and there. Out in the field... use another devices Wifi, and do the same thing. If you cannot get cell phone service or Wifi, then ( I literally took about 45 mins on the phone to T-Mobile, which immediately refused to take responsibility, and also refused to put anything in writing: Then their recommendation is to get a satellite link with, StarLink!. I would rather be slapped in the face with a cold fish. Cell phone service providers all let this happen, until of course they almost all sold out to T-Moble. ( LifeLine, Assurnace, etc.. )

netfortius 2y ago

It's [almost] always DNS :)

jiehong 2y ago

First, it was quite interesting! Second, it made me think about the new Apple C1 modem. They must have had to comb through those specs as well, and it makes me wonder if will allow them to better integrate the modem in the kernel in the end?