4 ms·

This blog post has been quite the effort—coming into it I thought I'd just learn some new things, document a simple process, and be done. But along the way I p

This blog post has been quite the effort—coming into it I thought I'd just learn some new things, document a simple process, and be done. But along the way I probably spent 40 hours (and counting) trying to figure out the ins-and-outs of 4G LTE modems and Linux networking when it comes to these oddball devices. I hope some parts of this post help anyone else new to LTE networking in Linux, and maybe somebody can help clear up some of my open questions, like: Why doesn't Quectel have any firmware downloads on their site? (And is there any way to get them besides downloading random files posted to forums?) Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)? Why does it seem there's no real documentation for LTE modems on Linux? What little _good_ information I found was in random presentation slides scattered around the Internet. Other than that it's either extremely technical documentation pages that expect a very high level of familiarity already, or some very cookie-cutter blog posts that give a walkthrough to get Internet going, but don't really explain anything. (Also in the course of my research I learned a lot from OSMOCOM's work reverse-engineering some of the LTE modems (including the QC25 I'm using): https://osmocom.org/projects/quectel-modems/wiki/EC25 https://osmocom.org/projects/quectel-modems/wiki/EC25 ).

drblah 5y ago
As someone who as also spent way more hours on LTE/5G modems than I want to, this post resonates with me. I also learned a few things, namely about the ECM. I will have to look into that. Thanks! As for the firmware. It seems to just be how these companies do things, unfortunately. At work, we use modems from both Quectel and Simcom and the experience described in OP is very similar. For Simcom we have gotten access to a "secret-but-public" Onedrive share with documentation, drivers and firmware where they put up stuff sporadically. Sometimes the firmware updates even works! I have a theory that Qualcomm who makes the chipsets used by Quectel and Simcom (at least for the 5G modems) imposes some kind of heavy NDA which prevents free distribution of firmware and documentation. Everything seems very "hush, hush" at least. While I can't help answer your remaining questions, I can point to wiki I have found really helpful: http://trac.gateworks.com/wiki/wireless/modem http://trac.gateworks.com/wiki/wireless/modem
pd0wm 5y ago
Have you experimented with modemmanager/networkmanager to handle the configuration of the modem? For me that has worked quite reliably across a pretty large fleet of devices with a Quectel EG25 (qmi over usb). Regarding FW updates, have you tried opening a support ticket with Quectel? For me they have been extremely responsive and helpful, providing extra documentation on special AT commands and stuff.
- geerlingguy 5y ago
  I have not—that was what I was trying (mmcli) early on in my exploration, but I kept hitting speedbumps, so switched gears to configuring the modem over TTY and using QMI utils directly. I would like to use modemmanager though, it _seems_ like it's the nicer path.
  - pd0wm 5y ago
    With modemmanager you get a lot of nice stuff for "free", like automatic reconnect, DHCP, good logging, and an APN database (useful if you don't provide your own SIMs). It also allows you to easily set up correct priorities between WiFi and LTE, both for routing and DNS. Especially DNS is something I didn't think about at first, but the DNS server provided by your LTE connection is not always reachable over your WiFi/ethernet connection. This is the config I'm using: https://github.com/commaai/agnos-builder/blob/master/userspace/files/lte.nmconnection https://github.com/commaai/agnos-builder/blob/master/userspa... You can run modemmanager in debug mode, which allows you to send AT commands over dbus (or mmcli) in case you still want to configure some stuff manually or get debug info.
0xbadc0de5 5y ago
If firmware is your thing - you may want to look into Sierra Wireless EM7355, EM7455, etc series. They are linux compatible and have firmware, docs and SDK available from the Sierra site. https://source.sierrawireless.com/devices/mc-series/mc7455/ https://source.sierrawireless.com/devices/mc-series/mc7455/
samueldr 5y ago
You might want to look around the Pinephone and Pinephone Pro communities. Both phones use the EG25-G modem, chosen for better worldwide band availabilities, rather than splitting the Pinephone in "EU" and "NA" batches. Users also are working on providing a mostly liberated firmware for the modem: - https://github.com/Biktorgj/pinephone_modem_sdk https://github.com/Biktorgj/pinephone_modem_sdk
anonymouse008 5y ago
My god, this community and the things we toil and labor over, then give freely to one another... is just... dang inspiring. It makes life worth living. Thank you kind, brave soul.
rcarmo 5y ago
Yeah, things have become wildly fragmented since the "plain" 3G modem days, and there are no effective standards for USB multi-point devices (LTE made Ethernet emulation more popular, but weird pseudo-serials to configure them just became... weirder).
- geerlingguy 5y ago
  Heh, when I started researching and found that, yes, uart and AT commands are still the preferred way to interact with these modems, I was wondering if I had time-warped back to the 90s!
  - rcarmo 5y ago
    Yep. Amazing how the AT command reference I tweeted at you is still 50% valid since 2004 :D
Nextgrid 5y ago
> Why doesn't Quectel have any firmware downloads on their site? The telecoms industry in general is pretty shit and does a lot of anti-user nastiness like SIM locking or other arbitrary restrictions. It is not in the their best interests to let the user get too close to the inner workings of things. Security-wise it's also often terrible and security by obscurity is still very much the norm and would be threatened by more transparency. There's also only a handful of manufacturers of the underlying modem chips (Qualcomm is the main one) and they're all in bed with the industry and perpetuate the issue, and the barriers to entry to build your own module are huge both in terms of skill, factories/equipment needed to produce the chips and patents so there's little chance for a more open competitor to step up. > And is there any way to get them besides downloading random files posted to forums? If you can find a production-grade device that embeds the same modem that you have, search for firmware updates for that device - chances are it may embed the firmware for the modem too, which you can extract with a bit of reverse engineering. It's not great, but at least you're still getting the firmware from a more trusted source than random forum posts. > Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)? If you look carefully at the `ip addr` output of QMI vs ECM you would see that the IPs you got aren't the same. In QMI mode, the LTE interface is directly connected to your system, the 10.... IP you got is the one assigned by your carrier (I am not sure where the initial 169.254.231.106 IP it had before you established the connection came from) and your machine is directly reachable from the Internet. In ECM mode, you are given a fake Ethernet interface that's connected to a router and NAT gateway ran by the LTE module itself; this is also where that web interface comes from. In this mode, the module's router part is the one exposed to the internet and acts as a router/firewall/NAT gateway - your computer is not directly reachable unless you use that web interface to forward a port. Given the usual "quality" of these (as you witnessed by looking at the UI) I would not be surprised if there are exploits or bugs in there. There should also be a third mode, where the modem exposes one or more serial ports that accept AT commands and you can establish a connection over it using wvdial/pppd just like good old dial-up. This is worse than QMI mode but preferable to ECM mode as you still get your system directly connected to the internet without relying on the modem's router/gateway part. In general, I would recommend QMI mode - it is a breath of fresh air compared to the old school way of the modem exposing multiple serial ports and having to do dial-up on it. It does require some work but it's a solved problem - for desktop Linux look into ModemManager, or alternatively look into how OpenWRT deals with it for a potentially lighter alternative.

drblah 5y ago

As someone who as also spent way more hours on LTE/5G modems than I want to, this post resonates with me. I also learned a few things, namely about the ECM. I will have to look into that. Thanks! As for the firmware. It seems to just be how these companies do things, unfortunately. At work, we use modems from both Quectel and Simcom and the experience described in OP is very similar. For Simcom we have gotten access to a "secret-but-public" Onedrive share with documentation, drivers and firmware where they put up stuff sporadically. Sometimes the firmware updates even works! I have a theory that Qualcomm who makes the chipsets used by Quectel and Simcom (at least for the 5G modems) imposes some kind of heavy NDA which prevents free distribution of firmware and documentation. Everything seems very "hush, hush" at least. While I can't help answer your remaining questions, I can point to wiki I have found really helpful: http://trac.gateworks.com/wiki/wireless/modem http://trac.gateworks.com/wiki/wireless/modem

pd0wm 5y ago

Have you experimented with modemmanager/networkmanager to handle the configuration of the modem? For me that has worked quite reliably across a pretty large fleet of devices with a Quectel EG25 (qmi over usb). Regarding FW updates, have you tried opening a support ticket with Quectel? For me they have been extremely responsive and helpful, providing extra documentation on special AT commands and stuff.

geerlingguy 5y ago
I have not—that was what I was trying (mmcli) early on in my exploration, but I kept hitting speedbumps, so switched gears to configuring the modem over TTY and using QMI utils directly. I would like to use modemmanager though, it _seems_ like it's the nicer path.
- pd0wm 5y ago
  With modemmanager you get a lot of nice stuff for "free", like automatic reconnect, DHCP, good logging, and an APN database (useful if you don't provide your own SIMs). It also allows you to easily set up correct priorities between WiFi and LTE, both for routing and DNS. Especially DNS is something I didn't think about at first, but the DNS server provided by your LTE connection is not always reachable over your WiFi/ethernet connection. This is the config I'm using: https://github.com/commaai/agnos-builder/blob/master/userspace/files/lte.nmconnection https://github.com/commaai/agnos-builder/blob/master/userspa... You can run modemmanager in debug mode, which allows you to send AT commands over dbus (or mmcli) in case you still want to configure some stuff manually or get debug info.

0xbadc0de5 5y ago

If firmware is your thing - you may want to look into Sierra Wireless EM7355, EM7455, etc series. They are linux compatible and have firmware, docs and SDK available from the Sierra site. https://source.sierrawireless.com/devices/mc-series/mc7455/ https://source.sierrawireless.com/devices/mc-series/mc7455/

samueldr 5y ago

You might want to look around the Pinephone and Pinephone Pro communities. Both phones use the EG25-G modem, chosen for better worldwide band availabilities, rather than splitting the Pinephone in "EU" and "NA" batches. Users also are working on providing a mostly liberated firmware for the modem: - https://github.com/Biktorgj/pinephone_modem_sdk https://github.com/Biktorgj/pinephone_modem_sdk

anonymouse008 5y ago

My god, this community and the things we toil and labor over, then give freely to one another... is just... dang inspiring. It makes life worth living. Thank you kind, brave soul.

rcarmo 5y ago

Yeah, things have become wildly fragmented since the "plain" 3G modem days, and there are no effective standards for USB multi-point devices (LTE made Ethernet emulation more popular, but weird pseudo-serials to configure them just became... weirder).

geerlingguy 5y ago
Heh, when I started researching and found that, yes, uart and AT commands are still the preferred way to interact with these modems, I was wondering if I had time-warped back to the 90s!
- rcarmo 5y ago
  Yep. Amazing how the AT command reference I tweeted at you is still 50% valid since 2004 :D

Nextgrid 5y ago

> Why doesn't Quectel have any firmware downloads on their site? The telecoms industry in general is pretty shit and does a lot of anti-user nastiness like SIM locking or other arbitrary restrictions. It is not in the their best interests to let the user get too close to the inner workings of things. Security-wise it's also often terrible and security by obscurity is still very much the norm and would be threatened by more transparency. There's also only a handful of manufacturers of the underlying modem chips (Qualcomm is the main one) and they're all in bed with the industry and perpetuate the issue, and the barriers to entry to build your own module are huge both in terms of skill, factories/equipment needed to produce the chips and patents so there's little chance for a more open competitor to step up. > And is there any way to get them besides downloading random files posted to forums? If you can find a production-grade device that embeds the same modem that you have, search for firmware updates for that device - chances are it may embed the firmware for the modem too, which you can extract with a bit of reverse engineering. It's not great, but at least you're still getting the firmware from a more trusted source than random forum posts. > Is there any advantage to using QMI (`wwan0` / qmi_wwan) vs ECM (`usb0` / cdc_ether)? If you look carefully at the `ip addr` output of QMI vs ECM you would see that the IPs you got aren't the same. In QMI mode, the LTE interface is directly connected to your system, the 10.... IP you got is the one assigned by your carrier (I am not sure where the initial 169.254.231.106 IP it had before you established the connection came from) and your machine is directly reachable from the Internet. In ECM mode, you are given a fake Ethernet interface that's connected to a router and NAT gateway ran by the LTE module itself; this is also where that web interface comes from. In this mode, the module's router part is the one exposed to the internet and acts as a router/firewall/NAT gateway - your computer is not directly reachable unless you use that web interface to forward a port. Given the usual "quality" of these (as you witnessed by looking at the UI) I would not be surprised if there are exploits or bugs in there. There should also be a third mode, where the modem exposes one or more serial ports that accept AT commands and you can establish a connection over it using wvdial/pppd just like good old dial-up. This is worse than QMI mode but preferable to ECM mode as you still get your system directly connected to the internet without relying on the modem's router/gateway part. In general, I would recommend QMI mode - it is a breath of fresh air compared to the old school way of the modem exposing multiple serial ports and having to do dial-up on it. It does require some work but it's a solved problem - for desktop Linux look into ModemManager, or alternatively look into how OpenWRT deals with it for a potentially lighter alternative.