4 ms·

> it suddenly ballooned in size in April 2025 after its operators breached a TotoLink router firmware update server and infected approximately 100,000 devices

> it suddenly ballooned in size in April 2025 after its operators breached a TotoLink router firmware update server and infected approximately 100,000 devices This is scary. Everyone lauds open source projects like OpenWRT but... who is watching their servers? I imagine you can't run an army of security people on donations and a shoestring budget. Does OpenWRT use digital signing to mitigate this?

sam_lowry_ 11mo ago
This is exactly why OpenWRT has no unattended updates by default )
- shoddydoordesk 11mo ago
  You are dismissing the seriousness of this. Their package manager is widely used. One would only need to compromise their build servers to wreak havoc. Didn't they have a vulnerability in their firmware download tool like a minute ago? The difference between OpenWRT and Linux distros is the amount of testing and visibility. OpenWRT is loaded on to residential devices and forgotten about, it doesn't have professional sysadmins babysitting it 24/7. Remember the xz backdoor was only discovered because some autist at Microsoft noticed a microsecond difference in performance testing.
  - jacobgkau 11mo ago
    I'm confused why you're so honed in on OpenWRT as a third-party open-source project here when the vulnerability you quoted (TotoLink) was the official firmware update server of a brand of devices. Is it "scary" to think about OpenWRT potentially getting hacked? If you get scared by theoretical possibilities in software, sure. Is it relevant? Not exactly. Are companies' official servers more secure than an open-source project's servers? In this case, apparently not.
    - danudey 11mo ago
      What's scary is that OpenWRT is a project created by people who wanted a better solution than what was out there, and are therefore largely driven by a desire to create a good product. Meanwhile, corporations are driven entirely by profit motive, so as long as it's more expensive to be vigilant about security than it is to be lax about it they will never improve. Until companies which produce (and do not update) vulnerable equipment are penalized (e.g. charged with criminal negligence) for DDoS attacks using their hardware then the open-source projects are going to continue to be far more trustworthy and less vulnerable than corporations which mass-produce the cheapest hardware they can and then designating it as obsolete and unsupported as fast as possible to force more updates.
      - AnthonyMouse 11mo ago
        The disappointing thing is that the companies don't just ship the open source firmware on their devices from the factory. They rarely if ever have any marketable features the open source firmware doesn't -- it's more often the other way around -- and then you don't have a zillion unpatched devices when they decide to stop caring because the community continues to maintain the code.
whatshisface 11mo ago
As always, hundreds watch the open repositories, maybe one watches a company's build servers, if they're lucky. :-)
- TylerE 11mo ago
  Hundreds watch, but how closely? Plenty of stories of fairly major projects having evil commits snuck in that remain for months.
  - alphager 11mo ago
    Name a few.
    - deleted 11mo ago
      [deleted]
    - TylerE 11mo ago
      https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://medium.com/@aleksamajkic/fake-sms-how-deep-does-the-rabbit-hole-really-go-17e25c42f986#345c https://medium.com/@aleksamajkic/fake-sms-how-deep-does-the-... https://blog.linuxmint.com/?p=2994 https://blog.linuxmint.com/?p=2994 https://www.bleepingcomputer.com/news/linux/malicious-package-found-on-the-ubuntu-snap-store/ https://www.bleepingcomputer.com/news/linux/malicious-packag... https://www.cnx-software.com/2021/04/22/phd-students-willfully-committed-known-malicious-changes-to-mainline-linux/ https://www.cnx-software.com/2021/04/22/phd-students-willful... I could go on but I trust this is a sufficient number of examples.
      - pona-a 11mo ago
        Only two of these were actual malicious commits. Two others were malware inserted into the repositories (if Twitter could be thought of as a meta-repo), which is bad but not on the same scale.
      - dxxvi 11mo ago
immibis 11mo ago
Digital signing wouldn't defend you from a compromised build server.
- mbilker 11mo ago
  What in that act says OpenWrt would be made illegal? If anything, OpenWrt would roll out automated security updates for a supported branched release to comply with these regulations. Also, if you actually read it, there are exceptions for open source software!
  - majorchord 11mo ago
    OP claims almost daily that some benign thing is actually illegal but practically never provides any useful proof when asked. (please prove me wrong, Alex)
- pabs3 11mo ago
  Reproducible Builds and multiple distributed builders would though. https://reproducible-builds.org/ https://reproducible-builds.org/
tempest_ 11mo ago
I don't follow. > run an army of security people Do you think these private companies do this? They don't. They pay as little as humanly possible to cover their ass. Botnets comprised of compromised routers is common and commercial/consumer routers are a far juicer target than openwrt.
- bigiain 11mo ago
  > They pay as little as humanly possible to cover their ass. They probably spend more on the team who ends up writing the "We take your security very seriously" breach notification message than they do on "security people". At least until then get forced into brand-name external Cyber Security Consultants to "investigate" their breach and work out who they can plausibly blame it on that's not part of the C suite.
- Aeolun 11mo ago
  > They pay as little as humanly possible to cover their ass. It’s probably helpful that open source teams aren’t hampered by standards and 20 year outdated audit processes either.
nine_k 11mo ago
Why, OpenWRT firmware and packages are both signed, of course. You can manually and independently check the image signature before flashing an update. The build infrastructure is, of course, a juicy target: infect the artifact after building but before signing, and pwn millions of boxes before this is detected. This is why bit-perfect reproducible builds are so important. OpenWRT in particular have that: https://openwrt.org/docs/guide-developer/security#reproducible_builds https://openwrt.org/docs/guide-developer/security#reproducib...
- tetha 11mo ago
  Bit-Reproducible infrastructure could also result in some of the wildest build distribution architectures if you think about it. You could publish sources and have people register like in APT mirrors to provide builds, and at the end of the day, the build from the largest bit-equal group is published. I do see the Tor-Issue - a botnet or a well-supplied malicious actor could just flood it. And if you flip it - if you'd need agreement about the build output, it could also be poisoned with enough nodes to prevent releases for a critical security issue. I agree, I don't solve all supply chain issues in one comment :) But that in turn could be helped with reputation. Maybe a node needs to supply 6 months of perfect builds - for testing as well - to become eligible. Which would be defeated by patience, but what isn't? It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer. This combination of reproducible, deterministic builds, tests across a number of probably-trustworthy sources is quite interesting, as it allows very heavy decentralization. I could just run an old laptop or two here to support. And then come compromise hundreds of these all across the world.
  - smt88 11mo ago
    The distribution system you're describing exists and has been in use for decades. You just distribute the build using bittorrent.
    - cluckindan 11mo ago
      And if someone invests in having >90% of the peers offer a malicious file and serve DHTs matching that file?
sidewndr46 11mo ago
The post is nothing more than "but what about security" meant to deflect away from the discussion at hand and towards OpenWRT
Quothling 11mo ago
I recently had some issues getting one of our embeded devices connect through passive ftp. Because the exact same device worked at a different site I knew it wasn't the device or it's settings. Long story short, it turned out the problematic site hadn't been updating its routers which meant they couldn't VPN passive FTP traffic. Anyway, we have literal thousands of those routers maintained by hundreds of different companies, who are mainly there to maintain the actual mechanical equipment and not the network. Turned out the site where the technicians updated things weren't in the majority. I'm in the process of getting the business to implement better security, and it's going better than you might expect. If it wasn't because having a plan for how to update your OT security is required to meet EU compliance, however, I doubt we would've done anything beyond making sure we could do passive FTP when it was needed. As an example, there is still no plans to deal with the OT which we know has build in hardware backdoors from the manufactures. Wnich is around 70% of our dataloggers, but the EU has no compliance rules on that...

sam_lowry_ 11mo ago

This is exactly why OpenWRT has no unattended updates by default )

shoddydoordesk 11mo ago
You are dismissing the seriousness of this. Their package manager is widely used. One would only need to compromise their build servers to wreak havoc. Didn't they have a vulnerability in their firmware download tool like a minute ago? The difference between OpenWRT and Linux distros is the amount of testing and visibility. OpenWRT is loaded on to residential devices and forgotten about, it doesn't have professional sysadmins babysitting it 24/7. Remember the xz backdoor was only discovered because some autist at Microsoft noticed a microsecond difference in performance testing.
- jacobgkau 11mo ago
  I'm confused why you're so honed in on OpenWRT as a third-party open-source project here when the vulnerability you quoted (TotoLink) was the official firmware update server of a brand of devices. Is it "scary" to think about OpenWRT potentially getting hacked? If you get scared by theoretical possibilities in software, sure. Is it relevant? Not exactly. Are companies' official servers more secure than an open-source project's servers? In this case, apparently not.
  - danudey 11mo ago
    What's scary is that OpenWRT is a project created by people who wanted a better solution than what was out there, and are therefore largely driven by a desire to create a good product. Meanwhile, corporations are driven entirely by profit motive, so as long as it's more expensive to be vigilant about security than it is to be lax about it they will never improve. Until companies which produce (and do not update) vulnerable equipment are penalized (e.g. charged with criminal negligence) for DDoS attacks using their hardware then the open-source projects are going to continue to be far more trustworthy and less vulnerable than corporations which mass-produce the cheapest hardware they can and then designating it as obsolete and unsupported as fast as possible to force more updates.
    - AnthonyMouse 11mo ago
      The disappointing thing is that the companies don't just ship the open source firmware on their devices from the factory. They rarely if ever have any marketable features the open source firmware doesn't -- it's more often the other way around -- and then you don't have a zillion unpatched devices when they decide to stop caring because the community continues to maintain the code.

whatshisface 11mo ago

As always, hundreds watch the open repositories, maybe one watches a company's build servers, if they're lucky. :-)

TylerE 11mo ago
Hundreds watch, but how closely? Plenty of stories of fairly major projects having evil commits snuck in that remain for months.
- alphager 11mo ago
  Name a few.
  - deleted 11mo ago
    [deleted]
  - TylerE 11mo ago
    https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://medium.com/@aleksamajkic/fake-sms-how-deep-does-the-rabbit-hole-really-go-17e25c42f986#345c https://medium.com/@aleksamajkic/fake-sms-how-deep-does-the-... https://blog.linuxmint.com/?p=2994 https://blog.linuxmint.com/?p=2994 https://www.bleepingcomputer.com/news/linux/malicious-package-found-on-the-ubuntu-snap-store/ https://www.bleepingcomputer.com/news/linux/malicious-packag... https://www.cnx-software.com/2021/04/22/phd-students-willfully-committed-known-malicious-changes-to-mainline-linux/ https://www.cnx-software.com/2021/04/22/phd-students-willful... I could go on but I trust this is a sufficient number of examples.
    - pona-a 11mo ago
      Only two of these were actual malicious commits. Two others were malware inserted into the repositories (if Twitter could be thought of as a meta-repo), which is bad but not on the same scale.
    - dxxvi 11mo ago

immibis 11mo ago

Digital signing wouldn't defend you from a compromised build server.

mbilker 11mo ago
What in that act says OpenWrt would be made illegal? If anything, OpenWrt would roll out automated security updates for a supported branched release to comply with these regulations. Also, if you actually read it, there are exceptions for open source software!
- majorchord 11mo ago
  OP claims almost daily that some benign thing is actually illegal but practically never provides any useful proof when asked. (please prove me wrong, Alex)
pabs3 11mo ago
Reproducible Builds and multiple distributed builders would though. https://reproducible-builds.org/ https://reproducible-builds.org/

tempest_ 11mo ago

I don't follow. > run an army of security people Do you think these private companies do this? They don't. They pay as little as humanly possible to cover their ass. Botnets comprised of compromised routers is common and commercial/consumer routers are a far juicer target than openwrt.

bigiain 11mo ago
> They pay as little as humanly possible to cover their ass. They probably spend more on the team who ends up writing the "We take your security very seriously" breach notification message than they do on "security people". At least until then get forced into brand-name external Cyber Security Consultants to "investigate" their breach and work out who they can plausibly blame it on that's not part of the C suite.
Aeolun 11mo ago
> They pay as little as humanly possible to cover their ass. It’s probably helpful that open source teams aren’t hampered by standards and 20 year outdated audit processes either.

nine_k 11mo ago

Why, OpenWRT firmware and packages are both signed, of course. You can manually and independently check the image signature before flashing an update. The build infrastructure is, of course, a juicy target: infect the artifact after building but before signing, and pwn millions of boxes before this is detected. This is why bit-perfect reproducible builds are so important. OpenWRT in particular have that: https://openwrt.org/docs/guide-developer/security#reproducible_builds https://openwrt.org/docs/guide-developer/security#reproducib...

tetha 11mo ago
Bit-Reproducible infrastructure could also result in some of the wildest build distribution architectures if you think about it. You could publish sources and have people register like in APT mirrors to provide builds, and at the end of the day, the build from the largest bit-equal group is published. I do see the Tor-Issue - a botnet or a well-supplied malicious actor could just flood it. And if you flip it - if you'd need agreement about the build output, it could also be poisoned with enough nodes to prevent releases for a critical security issue. I agree, I don't solve all supply chain issues in one comment :) But that in turn could be helped with reputation. Maybe a node needs to supply 6 months of perfect builds - for testing as well - to become eligible. Which would be defeated by patience, but what isn't? It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer. This combination of reproducible, deterministic builds, tests across a number of probably-trustworthy sources is quite interesting, as it allows very heavy decentralization. I could just run an old laptop or two here to support. And then come compromise hundreds of these all across the world.
- smt88 11mo ago
  The distribution system you're describing exists and has been in use for decades. You just distribute the build using bittorrent.
  - cluckindan 11mo ago
    And if someone invests in having >90% of the peers offer a malicious file and serve DHTs matching that file?

sidewndr46 11mo ago

The post is nothing more than "but what about security" meant to deflect away from the discussion at hand and towards OpenWRT

Quothling 11mo ago

I recently had some issues getting one of our embeded devices connect through passive ftp. Because the exact same device worked at a different site I knew it wasn't the device or it's settings. Long story short, it turned out the problematic site hadn't been updating its routers which meant they couldn't VPN passive FTP traffic. Anyway, we have literal thousands of those routers maintained by hundreds of different companies, who are mainly there to maintain the actual mechanical equipment and not the network. Turned out the site where the technicians updated things weren't in the majority. I'm in the process of getting the business to implement better security, and it's going better than you might expect. If it wasn't because having a plan for how to update your OT security is required to meet EU compliance, however, I doubt we would've done anything beyond making sure we could do passive FTP when it was needed. As an example, there is still no plans to deal with the OT which we know has build in hardware backdoors from the manufactures. Wnich is around 70% of our dataloggers, but the EU has no compliance rules on that...