18 ms·

What happened with ASUS routers this morning?

736 points by zdw 3y ago

Anonasty 3y ago
Asus needs to get their act together. First the AMD mobo problem and now this.
- ddtaylor 3y ago
  I think at this point they have eroded enough brand trust it would be difficult to recover from. I'm not hyper-vigilant in this area, but I can say with confidence I have absolutely no expectation to do anything related to ASUS hardware for any of my upcoming tech purchases. That's a decision that will likely affect my purchasing for the next 5 years since I don't often re-build my PC.
delabay 3y ago
I had this exact issue today. Starting in the wee hours of the morning, the router would just crash every 30-40 minutes. I had to reboot it to get the router back. I updated firmware and it seemed to fix it..
- deleted 3y ago
  [deleted]
- argc 3y ago
  This happened to me too! After a day of dealing with the router crashing every 30 minutes, I finally updated the firmware and am hoping the issue is fixed now.
the-anarchist 3y ago
And yet people keep asking what the point of OpenWRT is.
- awestroke 3y ago
  Nobody is asking that
  - smashed 3y ago
    Hmm maybe not but reading all the linked forum posts and threads, people are at a complete lost for information. "I installed version X.Y.Z and so far so good" "I tried all available updates and it keeps crashing". There is no transparency in the published updates. Take that firmware blob (which is probably 99% GPL licensed by the way) and shut up. That's basically what the vendors are saying. What's ironic is that most OEM are using some sort of out of date openwrt derivatives as a base for their closed source firmwares, which are obvious GPL violations.
    - Nextgrid 3y ago
      > What's ironic is that most OEM are using some sort of out of date openwrt derivatives as a base for their closed source firmwares, which are obvious GPL violations. Do you have a source of that or am I out of date? Based on my knowledge, outdated, GPL-violation OpenWRT would still be a massive upgrade over what's usually developed in-house by OEMs.
      - IntelMiner 3y ago
        Linksys WRT 1200/1900/3200/32X touted their OpenWRT based firmware as a feature (though those routers are about a decade old now, and their Wi-Fi chips have unfixed bugs due to Marvell selling their Wi-Fi division to NXP) Musk's "Starlink" router is also OpenWRT based
        sgtnoodle 3y ago
        I use a WRT1200AC running openWRT as my router, wifi disabled, and it works pretty well. I use two WRT1900AC running openWRT to make a WDS bridge and can get about 400-500Mbps of iperf3 throughput, and one of the two serves as an access point. Even though the wifi does indeed have its limitations in the form of missing features, it's been reliable and performant for me.
albert_e 3y ago
Interesting. May be i am old fashioned but shouldn't hardware appliances be designed as standalone devices that have minimal external dependencies. If there was no firmware update or patch applied -- the functionality of the device shouldn't change. Do all routers these days "phone home" anyway and modify settings on autopilot? Even if user has chosen to turn off updates?
- vorpalhex 3y ago
  Routers tend to be complex devices that have vulnerabilities. This has led to bot armies of routers. Users won't apply fixes. Therefore auto-updates became the norm.
  - akira2501 3y ago
    This _was_ a bot army of routers. It just so happened that the army's orders were to do something uncoordinated and useless. You can trust the users implicitly, or you can trust ASUS implicitly. It seems that the appropriate place to deal with network traffic security issues is with the network bandwidth provider, which would be your ISP.
    - derefr 3y ago
      > It seems that the appropriate place to deal with network traffic security issues is with the network bandwidth provider, which would be your ISP. This would imply that the ISP has abilities to interpose between malicious traffic and the user that, with modern technologies like DNS-over-HTTP, they simply don’t have. To most ISPs (that don’t force you to install MITM CA certs), user traffic is increasingly a black box. Which means that attacker traffic can blend right in as part of said black box. The only thing that can see into the black box is the device on the end of the line.
      - akira2501 3y ago
        In most cases the device is getting it's IP information on boot by negotiation or request from the ISP. All the ISP needs is a database of blacklisted equipment and some middleware box to receive connections from insecure home devices when it detects them. It's not about catching related traffic when it occurs, more managing the immediate implications from insecure devices on the network. The ISP can see the device, it doesn't need to inspect the traffic.
shmerl 3y ago
Wipe out Asus firmware and install something like DD-WRT to never have such problems again.
- cmehdy 3y ago
  Unfortunately it's becoming increasingly expensive to find routers for casual home usage that allow for alternative supported firmwares. A decade ago slapping tomato on something cheap would do the trick, but now with high speed internet and wifi 6 there aren't many devices having that flexibility within entry-level price ranges (at least from what I've seen in Europe and Canada).
  - kelnos 3y ago
    [0] would seem to suggest there are 45 options for things with WiFi 6 that you can install OpenWRT on. 16 of them are supported in an actual release and don't require running a snapshot. Not amazing, to be sure, and I don't know what those devices cost, but I think the situation is less dire than your post would suggest. [0] https://openwrt.org/toh/views/toh_available_16128_ax-wifi https://openwrt.org/toh/views/toh_available_16128_ax-wifi
    - londons_explore 3y ago
      None of them are things your ISP will ship you free, nor things you'll find on eBay for $20. (Despite routers capable of gigabit speeds being commonly both free and available on eBay for pennies)
  - mjevans 3y ago
    With the price of high end routers rising the last time I needed one (that could get near to 1gbit) I went with a mini board that was well supported by OpenWRT. Some of these can also support WiFi via laptop style PCIe add-in cards. https://openwrt.org/toh/views/toh_single-board-computers https://openwrt.org/toh/views/toh_single-board-computers
- buccal 3y ago
  If you plan on using third party firmware you should look at compatibility with OpenWRT BEFORE buying the device. Router manufacturers are unwilling and unable to provide a future-proof software. Latest proof of that in my case was with a new Wi-Fi 6 router Netgear WAX206. After indeterminate time clients would just stop receiving packets and would require reconnecting. WAX206 latest factory firmware is based on OpenWRT from 2016 (Netgear still refuses to open source parts of it). After flashing current OpenWRT the connection is rock solid.
userbinator 3y ago
This seems to be related to “ASUS Healing System” which I don’t even know if I have enabled or not. That name already sounds creepy enough, but searching for that string (with the quotes) currently returns only 4 results, of people asking what it is. My guess is some sort of hidden backdoor, disguised as an ostensibly useful feature.
- atoav 3y ago
  To me that sounds like some sort of adaptive signal conditioning/denoising/filtering system just by the name of it. So DSP magic?
- krackers 3y ago
  So what exactly is the Asus Healing System? I can't find any details about it online.
  - ddtaylor 3y ago
    I assume that ASUS routers are based on Linux, so shouldn't the source for these routers be readily available? I am able to find custom third party mods (asuswrt-merlin) but I can't actually find a clear copy of the original sources! This is very common when I look to find source for embedded devices like this. What I expect is the next step is that you will find (or be given) a borderline useless blob of source that doesn't explain any of it's build process, which is absurd because the GPL clearly defines the build "glue" as part of the source. Is ASUS another company that is doing a poor job of GPL compliance in this space? Is it intentional?
    - IshKebab 3y ago
      Why would ASUS release the source code for its router firmware? It doesn't have to be GPL.
      - ddtaylor 3y ago
        > AsusWRT is a derivative of Tomato which is itself one of the descendants of HyperWRT, a Linux distribution for low-end network appliances such as routers.
gkanai 3y ago
I have a set of ASUS mesh routers and have been experiencing this issue today. Glad I see it's not my specific problem although it looks like I have to watch for firmware updates.
- garbagewoman 3y ago
  probably consider swapping them out for something less sus, too
  - CincinnatiMan 3y ago
    Any recommendations for someone that doesn’t spend much time tinkering these days?
legohead 3y ago
This happened to me this morning. I even called Cox, since they were out just last week doing scheduled maintenance and the internet went down a few times. Cox said the line looks fine (and tried to sell me their cable modem). I logged into my router and checked for new firmware, then installed it, and it started working again.
chillbill 3y ago
I have the same device, didn’t notice any issues, possibly because I’m using Asuswrt Merlin https://www.asuswrt-merlin.net/ https://www.asuswrt-merlin.net/
- rlbaker 3y ago
  Did not experience any issues with my RT-AX86U running Asuswrt-Merlin either.
- kqr 3y ago
  Oddly, I'm also running Merlin but I did have a problem where my laptop thought it was connected to the AP but couldn't get local network traffic routed to it. Easiest solution turned out to be to just reconnect, so I don't actually know the deeper problem. I don't think it has any relation, but since it's the first time it's happened, it was kind of a freaky coincidence!
- cheald 3y ago
  Ditto, RT-AC68U running merlin with no issues.
chx 3y ago
Never put a router without open source firmware on the open Internet. It's that simple. The right tool for the job. (OTOH I advice against using Linux as your daily driver on any modern laptop. Once again, the right tool for the job and that's Windows + WSL.)
- kelnos 3y ago
  > Windows + WSL Hard pass. Not interested in Microsoft's latest malware/adware of an OS. I've been running Linux on laptops for nearly 20 years now. Sure, it was a pain in the ass in the earlier days, but nowadays things just work, as long as you're willing to do your research ahead of time to choose a laptop where the hardware is fully supported.
  - ricardobayes 3y ago
    You're biased. Win 11 doesn't have third-party ads of any kind, unlike iOS's app store, for example. It was always news, videos etc. Recommended content. Moreover, ubuntu famously had an Amazon button on the home page, yet no is going around touting "Linux is riddled with adware".
    - fsflover 3y ago
      I looks like Ubuntu was created just in order to be able to dismiss Linux as "riddled with adware". It's just a single distribution out of a hundred, and far from the best, so it's completely wrong of course.
      - rejectfinite 3y ago
        >I looks like Ubuntu was created just in order to be able to dismiss Linux as "riddled with adware". Yes of course it was. The only reason. Delusuional. You think some conspiracy created Ubuntu so people can point to the Amazon thing?
        msla 3y ago
inamberclad 3y ago
Problems today too. This is a huge pain and really making me question my router. What's a good hardware platform for OpenWRT?
- aetherspawn 3y ago
  I can't recommend the tp-link AX50 highly enough. It has been rock solid and stable now for probably around 4 years. I don't remember ever having to reset it. I recommended it to my friends when they changed their routers over, and it fixed a lot of issues for them as well. The ones who regularly used VR reported that the stuttering issues they had on VR over WiFi with their Netgear WiFi 6 routers were solved. Have owned both ASUS and Netgear in the past and the experience was terrible -- the high-end ASUS router regularly bricked itself during auto update. So I bought a more expensive ASUS router for around $600, and you guessed it, it did the same thing just less frequently. The NETGEAR routers never bricked themselves, but across the 3 models we've owned nearly all of them would lose most of their throughput (around 90%) on a daily basis and would need a daily reset to get proper ISP speeds again. I had a Dream Machine Pro for around a year before I moved it to my office to run the network in the office. Funnily enough it has probably been less stable than the tp-link router, but nonetheless the UDM Pro is still a great piece of kit and probably the last router you'd ever need if it's in your budget and you have a place to put it (awkward form factor for home). Re: Open WRT I ran this around 3 years ago on a D-Link router that I had, and the stability was unpredictable.. better than ASUS and NETGEAR but probably leaning on the side of poor compared to tp-link and UDM Pro. It would randomly get slow once per fortnight or so, and would need to be rebooted. Sometimes it would panic and completely lose internet connection. There are plugins designed to reboot the router as soon as a loss of connection is detected, but why is this even necessary? It's annoying too. Drops all your connections. Sorry for long.
- inamberclad 3y ago
  I followed one of the solutions in the comments and removed /jffs/asd/chknvram20230516 and the issue seems to have gone away. CPU usage went from 70-90% back down to 5%.
- nvarsj 3y ago
  I have one of those small Intel based nuc like devices and run x86 openwrt on that. It works great on a gigabit ipv6/ipv4 line. I also replaced the proprietary bios with coreboot for complete security.
55873445216111 3y ago
I had the same issue this morning on my RT-AC68U.
dude3 3y ago
I had issues too. Thought it was just my home network. After a few reboots of every device on ai mesh, was able to install the latest firmware. It mentioned it fixed a memory leak. I have 2 RT-AX55's.
crazy5sheep 3y ago
I thought it was just my asus router broken today, and I was about to buy a new one.
- garbagewoman 3y ago
  About to buy a new Asus router?
sneak 3y ago
This is the danger of allowing your vendor to automatically remotely execute arbitrary code on your devices.
- yosito 3y ago
  He had auto updates turned off, so there was no "allowing" happening. It's not even clear yet what happened, but one possibility is that this was someone taking advantage of an exploit on routers that had NOT been automatically updated.
  - garbagewoman 3y ago
    yes, as you suggest, it appears that Asus routers were targeted by a 0day that they are known to lack the skills to deal with
kenhwang 3y ago
Oh, that explains why my connection was slow from some rooms in the house today. I also have an ASUS RT-AC86U (running 3.0.0.4.386_51255), but as a node in the mesh, it's still online. However, my RT-AX55 (3.0.0.4.386_50460) in the mesh has been down all day and I haven't noticed before this post because the other nodes picked up the slack (with spotty/slow signal). Also have an RT-AX82U (3.0.0.4.388_22525) as the mesh leader and it also doesn't seem impacted.
- ok_dad 3y ago
  I had no issues with my 2 ax-92u routers in a mesh on the newest firmware 3.0.0.4.388_22525.
  - rahimnathwani 3y ago
    Ditto.
vrglvrglvrgl 3y ago
[dead]
olliej 3y ago
Wow I had this happen this morning with our axe6600/et8/whatever and assumed it was something actually failing
hbcondo714 3y ago
The power of blogging and HN. Asus or my ISP didn't tell me why my router/internet went out twice today. I honestly thought thieves stole my copper again: https://news.yahoo.com/rise-copper-theft-officials-concerned-195400800.html https://news.yahoo.com/rise-copper-theft-officials-concerned...
- DANmode 3y ago
  and brought it back in between? :D
  - TeMPOraL 3y ago
    Maybe they were spotted by the police and had to pretend they're there installing the wires so they don't get caught.
- itsphilos 3y ago
  again?
- mad0 3y ago
  I love when thieves are trying to steal copper, but all they get is a broken fiber optic ;) My dad works as a network engineer, and he told me a story that one of the banks in Poland lost one of the internet providers. They investigated and found out that thieves stole hundreds of meters of a fiber cable, because they thought that it's copper.
  - garbagewoman 3y ago
    that sounds like so much fun for all involved
  - maven29 3y ago
    Don't they typically have co-extruded copper in the conduit/pipes for Utility Locators to use, even if it's just fiber inside?
pendu 3y ago
Unbelievable. I struggled with this the whole day. Checked the ISP status. Thought, it could be overheating (router is in the garage) or bad RJ-45. Had to reboot multiple times today. Never ever I would have guessed it was a firmware issue. Thank you for writing this blogpost. I just upgraded firmware. So happy when something has an explanation. !
- garbagewoman 3y ago
  Why would your router overheat in the coldest place in the building?
  - tyingq 3y ago
    The garage could be the coldest area in the building, or the hottest. Things happen, too...dust clogging ventilation, animals build a nest around it, router placed near refrigerator coils, etc.
  - NBJack 3y ago
    I appreciate that you must live in a cooler climate. This is not the case everywhere. My garage cooks on a bad day. Great for the hot water heater, not so great for everything else.
jms703 3y ago
Good reason to consider running openwrt, opensense, vyos, or something like that, if you’re up to it.
- baq 3y ago
  MikroTik here, can recommend
  - AlbertVAustin 3y ago
    MikroTik is definitely very nice (I'm running my home network with them ~7 devices). Too bad it's not open-source like OpenWRT is :-/
    - intelVISA 3y ago
      I like MT but it being closed source always irked me for such a critical device: wonder how many Ghosts in the Shell lurk in there...
      - AlbertVAustin 3y ago
        I'm actually not that worried about them secretly doing something malicious (although that's also a valid concern) but rather in the "given enough eyeballs, all bugs are shallow" and the general risk factors (MikroTik going bankrupt and we're left with obsolete bug-ridden hardware). Sadly given the large amount of software that's closed source there I don't see them working towards open sourcing any of it as, even technically, it would be a massive effort.
        intelVISA 3y ago
        Agreed, even if it was not their intent it being closed source is malicious itself - we need to promote this discourse in public conciousness so people are less afraid to question the unaudited, outsourced binary blobs that fill their day to day lives.
  - 3y ago
simongray 3y ago
My Apple Airport router also crashed yesterday and had to be restarted. I wonder if this is related?
PeterStuer 3y ago
So far things seem fine here. I run an RT-AC87U (3.0.0.4.382_52545-ga0245cc) as the main router I have an RT-AC86U (3.0.0.4.386_51255-g486ded6) and an RT-AC68U (3.0.0.4.386_51255-g486ded6) running in AP mode. Tried checking whether there was an update available for the RT-AC87U, but it looks like the ASUS firmware site is unreachable (overloaded?) for me at the moment.
- PeterStuer 3y ago
  And the AC87U got bitten today. Running out of memory until dnsmasq can't allocate anymore. Reboots help only temporary.
dack 3y ago
Glad this wasn't just me - but we were going mad for a couple hours there not understanding what was going on... I still want to know what happened. It feels like a good excuse to get a new non-asus router barring any other explanation
bboygravity 3y ago
TLDR: someone at CCP and/or NSA tried out a backdoor in ASUS routers and did an oopsy.
- temptemptemp111 3y ago
  [dead]
JdeBP 3y ago
I don't have an ASUS router, but three things leap out at me: a string being logged over and over, running out of space on a filesystem, and rotated log files named something.1. It is trivially easy to blow right past the size capping on systems that use the old "newsyslog" style of external logfile rotation from the 20th century, and something that is logging a short string "[chknvram_action] Invalid string" over and over very fast is exactly how to do this. For those interested in investigation, therefore, I would suggest looking at logfile sizes, and seeing whether it was logs eating all of the free space on /jffs and /var . The underlying cause would be whatever is logging "[chknvram_action] Invalid string" thousands of times over, but the mechanism would be log files filling the tmpfs that the article mentions, which would explain why the system had no memory for forking new processes. My wild speculation about "[chknvram_action] Invalid string" is that something somewhere in whatever "chknvram" is, the name being suggestive of something checking non-volatile RAM, has either bad data or a broken parser, and the recovery semantics are to retry immediately, incessantly, as fast as possible.
- bartvk 3y ago
  Weird how syslog doesn't de-duplicate these identical log messages. Or maybe it does, but not enough.
  - inferiorhuman 3y ago
    BSD syslogd definitely does, perhaps whatever stripped down one ASUS is using doesn't. e.g. Dec 2 01:09:41 hostname syslogd: last message repeated 10 times The threshold's pretty low and most of the "repeated" messages say "repeated 1 times" however.
    - TeMPOraL 3y ago
      Does it store timestamps for those repeated occurrences? I wouldn't want my logs to "helpfully" coalesce multiple identical messages into a single one. For example, if I saw the equivalent of the text below in a program/OS while facing some issues, I'd be remarkably pissed. [2023-05-18 07:12:21] [E] Invalid event 0x1AF2 received from ... [2023-05-18 09:44:01] [I] Last message repeated 10 times I care less about how many times the message was repeated - I care about timestamps, which I might want to correlate to other activities.
Ninjaneered 3y ago
Super interesting, the same thing happened to me around 4am PDT, I have 2x ASUS ZenWiFi ET8 mesh routers. Every 10 minutes or so, we'd loose internet access for a minute, then it would come back. I called Charter and they blamed the router, but I was a bit suspect since in January of 2022, we had another issue where the call center blamed our router and after a bit of testing was able to prove to them it wasn't our router. This was kind of interesting as well, they finally sent out a second service tech that had some sort of spectrum analyzer and he found our line it was being overwhelmed with noise. Turns out the drop from the pole wasn't shielded and a new T-Mobile 5G tower was bleeding into the line (acting like an antennae). The tech put a new wire up and it immediately solved the issue. In both occasions, the help desk told us the signal coming to the router was fine though. This time a factory reset seemed to do the trick, but I seemed to have the latest firmware to start with so I'm not sure what's going on.
- garbagewoman 3y ago
  You're not sure what's going on, despite this new information?
  - egberts1 3y ago
    It would appear that he is not the troubleshooter but a trouble reporter.
    - Ninjaneered 3y ago
      Yeah, I did a factory reset before learning about other solutions here. I'll have to go through the thread again, but I'm not sure we know what the root cause is yet? Log files were filling up, but I didn't catch why.
gmac 3y ago
Interesting. My CT8 mesh routers seem OK and have no new updates available so far.
deleted 3y ago
[deleted]
theonlybutlet 3y ago
Vodafone gigabox did the same today. Never been sure who manufactures it?
shampto3 3y ago
Interesting. I have asus zenwifi mesh routers and today one of the nodes went offline and it took me a while to realize and then fix it. The fix was manually updating all of the firmware for every mesh router, so it seems like it was a similar problem. EDIT: crap, I just checked again and two of my nodes are offline now. Only the root node is working. Too late in the evening to fix it now, guess I’ll be working on that in the morning.
- JdeBP 3y ago
  The local fix is, reportedly, to delete the malware signature file that is breaking ASUS's proprietary malware scanner. * https://news.ycombinator.com/item?id=35984706 https://news.ycombinator.com/item?id=35984706 * https://news.ycombinator.com/item?id=35984725 https://news.ycombinator.com/item?id=35984725 The service fix is for ASUS to make a malware scanner that doesn't have such bad behaviour in the case of a signature file that it cannot process, and to not use a logging mechanism that we knew had this flaw in the 1980s, and improved upon in the 1990s. (-:
- garbagewoman 3y ago
  UPDATE: the fix appears to be swapping out the ASUS routers for anything else
- shampto3 3y ago
  UPDATE: I followed the instructions in https://news.ycombinator.com/item?id=35986375 https://news.ycombinator.com/item?id=35986375 for each of the nodes and it appears to have solved the problem, at least for now. I will keep an eye on them to make sure they don't die again. The frustrating part was that my home assistant raspberry pi was hardwired into one of the nodes that died, so my automation to change my thermostat wasn't working and it was stuck on a setting that I normally only set for one hour in the day. Obviously I could change it manually, but I didn't realize that it hadn't changed until it had been like 5 hours.
e3bc54b2 3y ago
Boy am I glad I replaced stock firmware with OpenWRT the moment my router came out of box last week. It was also extremely painless experience, and I'd really recommend people to buy routers with OpenWRT support, even if they cost a little more. A router is something you buy for a decade or more, and it's worth the investment. Our livelihood depends on network availability, and depending on whims of terrible router firmware is not something to rely on.
- gat1 3y ago
  OpenWRT is a bliss to setup a router. We configured a lot of little customed subnetworks at work and it made everything so much easier.
  - intelVISA 3y ago
    Can't reccommend OpenWRT enough, really solid.
- iforgotpassword 3y ago
  I've given up on trying to find one with perfect openwrt support. Something is always broken, mostly hardware offloading, so speeds suck.
  - wtallis 3y ago
    The kind of hardware offloading included in consumer router hardware is fundamentally broken by design. Relying on the ethernet switch to handle NAT instead of the CPU makes it impossible to do software-based QoS (eg. the SQM module's cake or fq-codel) or any other packet processing that said ethernet switch isn't equipped for.
- enlyth 3y ago
  There's also this one for Asus routers, I've been using it without issues for a long time: https://www.asuswrt-merlin.net/ https://www.asuswrt-merlin.net/
ww520 3y ago
I used to have the same model of ASUS router for several years. It lost internet connections quite often. Had to reboot to get it back. Also the performance felt sluggish. When WiFi 6 comes along, I just replace it with a router from another brand.
- vardump 3y ago
  Wifi 6 has been around for about 4 years. Maybe you meant Wifi 7?
  - garbagewoman 3y ago
    some of us are using ASUS, so wifi 6 is news to us, along with the news of how devastatingly terrible our routers are
    - vardump 3y ago
      Wifi 6 routers such as Asus RT-AX56U etc. were released back in 2019.
  - ww520 3y ago
    It’s Wi-Fi 6. I didn’t get it right at the beginning of the release cycle.
sampa 3y ago
People here recommend openwrt. For Asus routers it is wrong. Install Asuswrt-Merlin.
- pixelpanic360 3y ago
  [dead]
- egberts1 3y ago
  You should list the benefits of ASUS-Merlin over ASUS here. Leaving us ASUS users dangling with that comment is like leaving us hanging in there ... pondering WHY?
  - drunner 3y ago
    https://www.asuswrt-merlin.net/features https://www.asuswrt-merlin.net/features
RektBoy 3y ago
I cringe every time I see these "gamer" routers.
- Aeolun 3y ago
  They look horrid, but they work surprisingly well.
Sarkie 3y ago
I just installed 2 ASUS routers in my house on Monday. What a ride.
- garbagewoman 3y ago
  what led you to this decision?
adamparsons 3y ago
I lost my absolute mind this morning trying to figure this out. Dual wan and neither working. Got a few hours into the work day and ended up digging out an old WRT router just to get some work done. I saw a bunch of kernel panics and assumed the router finally gave up a the ghost after all these years. I guess I can go fish it back out of the bin
user070223 3y ago
I had brief disconnect from one of my routers earlier Now I try to update the ASUS firmware and get this Error "The router cannot connect to ASUS server to check for the firmware update. Please download the latest firmware from ASUS Product Support Site and do manually upgrade. Check on FAQ for more instructions."
shekhar123456 3y ago
[flagged]
rjmunro 3y ago
Any word from ASUS on this? If not, I know to avoid ASUS products in future.
- garbagewoman 3y ago
  If you have reached this point in time without having extreme distrust in the company, you may be interested in a bridge I have for sale at an extreme discount
tanepiper 3y ago
Ahh this explains why my upstairs router had no internet.
Hard_Space 3y ago
I am lucky enough to live in a time-zone where this crisis happened while I was asleep. Woke up to find that some network event had clearly happened, but that the pending Asus AC88U update that I had been putting off for months auto-triggered. So far, all normal.
rswail 3y ago
Off topic, but where can I find: 1. A modem that handles ADSL/VDSL/fibre incoming and 10GB ethernet outgoing with passthrough to: 2. A good wifi 6 mesh router + APs (eg Netgear or equivalent). I don't want them to do anything but: a) Support both DHCP and fixed assigned IPv4 configuration b) IPv6 /56 subnet assignment c) NAT outgoing IPv4, no incoming connections allowed d) IPv6 ingress/egress I don't want them scanning the traffic, protecting me from malware, upgrading themselves, doing anything fancy. I want the bandwidth of my home ISP connection to be supported and I want the bandwidth of my 5G wifi and 10GB internal LAN to be fully supported.
- stingraycharles 3y ago
  I’m a big fan of Ruckus for anything wireless, but they’re not cheap. But they’re extremely reliable, high-quality hardware that work especially well in multi-AP setups. You’ll probably want a separate modem+router device, though.
- sschueller 3y ago
  Your best bet is still to run your own hardware and software like opnsense [1]. An old PC would do or if you need more a newer one. You can easily add cards for 10 or 25 GBit and upgrade later I needed with more memory etc. [1] https://sschueller.github.io/posts/wiring-a-home-with-fiber/#router https://sschueller.github.io/posts/wiring-a-home-with-fiber/...
- denkmoon 3y ago
  A modem that handles ADSL, and VDSL, and Fibre, with 10GBASE-T? Why would such a thing exist? All ADSL and VDSL connections known to man will have adequate bandwidth over 1000BASE-T, simple and cheap modems that support RFC 1483 bridging abound. If you've got >1Gb/s fibre, presumably your ISP provides equipment to support such, and why not simply have a router with SFP+?
  - Aeolun 3y ago
    > If you've got >1Gb/s fibre, presumably your ISP provides equipment to support suc Why would they do that? If they sell me 10GBit but I can only reasonably use 1, then they can sell the same thing to 10x the people with the same hardware.
nuc1e0n 3y ago
The timestamp in the article of 1684335272 is oddly familiar to me. 16777216 is the maximum value you can store in 24 bits, but imagine if asus were storing the fractions of a second (0-100) in an 8 bit number, packed together with epoch seconds in a 32-bit integer. Maybe this was some timestamp related overflow?
- easytiger 3y ago
  It's Linux. It will be from a timeval struct or similar. It translates to Wed May 17 2023 14:54:32
Tor3 3y ago
Nothing happened to my RT-AC68U, and I just checked - it's (at least) a couple of firmware releases behind. I also checked my RT-AC2900, and nothing happened there either (I haven't checked the firmware version, but I have not updated anything since I bought it somewhen last year)
Gabrys1 3y ago
Fix: 1. Reboot the router via pulling and reconnecting the power cord 2. Log in to WWW interface 3. Go to Administration > System. Enable SSH (enable login/password as well, choose a port of your taste) 4. SSH to your router: ssh admin@192.168.50.1 -p 2424 (assuming your user name is admin, the IP is 192.168.50.1 and you chose port 2424 for ssh). Password is the same as for the web UI 5. In SSH session, type: rm /jffs/asd/chknvram20230516 6. In SSH session, type: reboot Seems to have done the trick for me.
- wincy 3y ago
  I’d recommend renaming the file instead of straight up deleting it.
  - scrapheap 3y ago
    Given the error they had in the log was no space left on device, then the fix probably does require the removal of the file.
    - shawnz 3y ago
      It was no space left on the tmpfs, since the bug causes some kind of memory leak. It's not a matter of disk usage.
  - smileybarry 3y ago
    It’s still used by asd next time it launches, so you have to at least move it out of that folder.
- Aeolun 3y ago
  I mean, this might work for me, but how do we deal with the fact that if I were my wife ASUS has basically turned my very expensive router into a very expensive brick?
  - Zolt 3y ago
    Agreed. I had enough issues with my router yesterday I purchased a different brand router and decommissioned my ASUS. I was not aware of this wide-spread issue until today. Much like HP printers, I will think twice about recommending or purchasing an ASUS.
daneel_w 3y ago
I use a $25 Asus "Wi-Fi repeater" which runs OpenWRT instead of their crippled firmware, allowing it be a plain Wi-Fi access point acting as a VLAN extension of my little SoHo switch. An entirely uncomplicated setup that just won't misbehave on its own.
egberts1 3y ago
I deleted /jffs/asd/chknvram20230516, rebooted, and back to normal. Pesky upgrade. Now have full wire capturing going on on public side.
citrin_ru 3y ago
Does anyone knows why Wi-Fi routers vendors keep writing (and maintaining for a short period) own firmware which in most cases has lower quality than OpenWRT when they can use OpenWRT and save money?
- alibert 3y ago
  Actually, there is a large number of manufacturers using OpenWrt as a baseline for their firmware. Unfortunately, these are often outdated version of a forked OpenWrt with heavy modifications to include firmware blob for hardware acceleration and such.
- pinebox 3y ago
  Because router vendors need some kind of "secret sauce" in order to keep selling routers, otherwise the whole thing becomes a commodity you can buy from anywhere. This is also why OpenWRT is unable to utilize all the features on a lot of routers: The closed source firmware and software pieces needed to do so have yet to be deciphered.
  - citrin_ru 3y ago
    Vendors can ship OpenWRT + closed source drivers. Anyway they use Linux kernel (yet to see a closed source BSD router; PfSense is occasionally used but its open source) relying on GPL exceptions allowing to use closed source drivers (and firmware blobs).
vijucat 3y ago
> "The fact that they were both at exactly 23 seconds is probably just a crazy coincidence" Immediately brought a smile to my face. This profession makes us detail-oriented detectives. It's a joyful power and a powerful joy :-) Now I know it's not a good personality trait. I guess one's hobbies should ideally be complete opposites, based on intuition and a feel for things.
- chatmasta 3y ago
  I can definitely relate. This sort of pattern recognition is a blessing and a curse... it's great when it saves some time during the hypothesizing phase, but not so great when it leads you on a wild goose chase :)
- kraussvonespy 3y ago
  https://xkcd.com/2259/ https://xkcd.com/2259/ I found that one just after spending all day chasing an odd WAN issue for a customer. A few years ago in the era of half/full duplex in ethernet, duplex mismatches sometimes manifested in some truly weird ways. It was at that point that I did start to believe in ghosts. At least ghosts in the network.
lightlyused 3y ago
My ac66u-b1 has been acting weird for the past several months. I first thought it was an Ubuntu bug. Wifi wouldn't always reconnect after I'd closed my laptop. Yesterday, however, it seemed to just die randomly. Error was "no space left on device." SSHing in to it and it does show root volume as 100% used. So, whatever Asus was doing has pushed it over the edge space wise and it can no long write the dnsmasq.leases file when enough devices connect. So, what wifi ap should I buy to replace this POS?
- trey-jones 3y ago
  Not offering advice on purchasing an AP, but if you are interested in trying to get more life out of this one: My guess based on reading the article is that it's mostly logs filling the disk. You can find out for sure with some version of du -a | sort -hr | head Then just start deleting logs at or near the top of the list - I suppose this would get you back running. Sure, constantly writing to logs sounds terrible for performance, but did you even notice when it was working properly? If it's actually Ubuntu running on this router, you could even install (or configure if it's already there) logrotate to probably prevent this from happening again at all. Then you still have a POS router, but one that might get the job done.
cloin 3y ago
I was fighting with this all day. I have a 3 pack Zenwifi XT8 that has always given me trouble so I didn’t even think to check if it was a widespread issue. I’ve just been enduring the pain long enough that my baseline pain is already so high. I originally bought these on a recommendation while I was looking for gigabit mesh systems. I’m back in the market after yesterday.
- jeremyjh 3y ago
  Eero 6 Plus, it will change your life. The things just work with no hassle, and I have very good coverage and throughput even out on our patio.
  - Skates1616 3y ago
    Running the same in 6 family member homes, I have not had one “tech support” call needed since installation last year. I moved them all from the original OnHubs which were also set it and forget it…
Eumenes 3y ago
I checked my logs and my T-Mobile (AC-1900) seems unaffected ...
XxCincinnatusxX 3y ago
Same router here & it did that to me all day yesterday but I'm not as tech savvy as some of you guys so I don't have a clue how to fix it. Hopefully I'll figure it out by the end of the day
ncphil 3y ago
Probably the most painful home Internet experience you can have is a consumer router fail. Suddenly you really need all those basic sysamin features a business system provides. I used OpenWRT on Linksys back in the day, but later had an early PCEngines APU running PfSense. Those worked well enough until my ISP increased our bandwidth. Made the jump to a little Mikrotik RB2011 on the top floor and added an extra AP (actuall a RB 900 series router) on the first. That did the job up until we got 1 Gb fiber. Not willing to use the ISP's router+mesh, I got a Ubiquiti EdgrRouter and a couple of Netgear APs (by then all our switches were ProSafe managed). The ER has been great. After mastering RouterOS, Vyatta was no big deal. I regret now not getting UI APs, Netgear changes its crappy prosumer interface with every model it seems, and they're painful to manage without paying for the cloud service. The switches are OK, but the once again the UX annoyingly varies by model. Of course, Ubiquiti has its own issues mostly in how the company itself is run. If I have to, I'm ready to go back to Mikrotik (whose web ui the Vault 7 goons could only crack during an interlude between firmwares: as if anyone leaves that UI enabled). In all the years I ran it, Mikrotik never caused me a problem with a RouterOS update (neither has EdgeOS).
- rejectfinite 3y ago
  >Suddenly you really need all those basic sysamin features a business system provides It's good for a learning experience :) And now, the ISP routers have caught up imo I just use that now and it is fine. Plug in as much as possible with Ethernet. If that fails, it's probably a router issue or ISP network issue, and then it's on the ISP side to fix.
  - hedora 3y ago
    In the US, ISPs sell all sorts of network snooping data to advertisers and the police. Letting one of their devices on to your LAN is asking for trouble.
- l33tman 3y ago
  To be fair, "business system features" are no help often, I have a $1200 Netgate pfSense firewall that just failed, probably due to the Intel Atom C2000 bug. I just have to throw it away and buy a new. That it was a business thingy didn't change anything except made it much more expensive to replace.
rejectfinite 3y ago
Right now, I just use the ISP modem/router combo. DOCSIS. Plug as much into ethernet as I can, mobiles and tablets on wifi. Laptop even using a dock for work, much more stable than wifi as I am in an apartment.
xmdx 3y ago
Is there any information on why this started happening in the last 2 days? Without any change to the firmware? Considering flashing asuswrt-merlin to avoid this in the future.
51Cards 3y ago
I have an AC-68U but had been keeping my firmware up to date. Fortunately it has been rock solid, including yesterday and today.
funOtter 3y ago
Any recommendations for good 4-6 port wifi routers?
tgtweak 3y ago
I stopped using Asus routers around the RT-68U generation since that was the third one where the radios burnt out on it. Been using tplink since and 0 issues. Asus has been pretty good about open source firmware and being relatively transparent about their operating system and hardware, but I can't afford to replace my router/AP every 2 years because of a crappy radio IC that is prone to failure.
carbocation 3y ago
Wait. Is it happening again this morning?
mydogmuppet 3y ago
Oh there is a god in heaven. When Asustek ran out of overpriced RT AX86Us I bit the Mikrotik bullet. Bought a hAP Ax3. Works like a dream, medium painful learning curve.
Havoc 3y ago
I've got 2x ASUS RT-AC92U on my network and all looks pretty normal on smokeping-like metrics.
t312227 3y ago
hello, idk ... but if your model is in the device compatibility-list, your should always be able to change to an alternative FOSS firmware. for example to https://openwrt.org https://openwrt.org or to some maintained tomato-clone like https://freshtomato.org https://freshtomato.org ps. i'm a happy owner of a 10+ years old RT N16 - it runs really great on openwrt ... cheers, v
- bioemerl 3y ago
  You can also build an X86 pfsense box that you can upgrade over time just like any other computer.
- stevefan1999 3y ago
  The problem is those routers usually have secret kernel modules that enables hardware acceleration, for example NAT offloading engine and ASIC acceleration for routing. Those are bound to a specific kernel version and is hard to reverse engineer for open source use because some of them are obfuscated binary blob
replwoacause 3y ago
Same exact issue here on my RT-AX58U. "dnsmasq-dhcp[1489]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)”
- johnbellone 3y ago
  I have the same on my router, too.
KomoD 3y ago
RT-AC58U, no issues.
xtat 3y ago
From the top comments it seems like we're skipping the fact that the guy has automatic updates turned off and it's pulling down and execing arbitrary code anyway.
- ed25519FUUU 3y ago
  There’s many, many things going wrong here.
sharno 3y ago
Wow, I thought I only had this problem. Contacted my ISP and they checked on their end and everything was fine. So I reset the router and it seems to be normal now.
- wood_spirit 3y ago
  It will likely crash again in a few minutes, then … :( I’ve just restarted mine and gone into the admin page and told it to update its firmware. No expectation it really helps, but it seems a good step before sshing in to delete a file…?
synergy20 3y ago
I have been running openwrt on my router for decades, buy one that openwrt supports.
jbluepolarbear 3y ago
Is this device specific? I have the RT-AX82U (great router by the way) and haven’t had any issues.
pnemonic 3y ago
This was driving me up the wall last night and this morning. Worst part is that fiber infra is being laid in my neighborhood, so I had it chalked up to my ISP changing hosts or something and it going sort of badly. Was planning to call them tonight if it was still f'ed when I got home. Fortunately, I checked for a firmware update from the car before I left and found the golden patch. Kicked it off and left for work. Figuring all is fine now. What a wild trip though. Wild trip? Man I am getting old.
tinglymintyfrsh 3y ago
These sort of vendor YOLO changes are more of the domain of Microsoft's MDE. I use an Deciso OPNsense where it doesn't insist on breaking itself on the orders of corporate overlords.
cfield 3y ago
fwiw, for the RT-AC68U, ASUS currently[0] recommends downgrading the firmware to the version at this link: https://dlcdnets.asus.com/pub/ASUS/wireless/RT-AC68U/FW_RT_AC68U_300438651255.zip?model=RT-AC68U https://dlcdnets.asus.com/pub/ASUS/wireless/RT-AC68U/FW_RT_A... To downgrade, they pointed to the 'Method 2: Update Manually' section of this support FAQ: https://www.asus.com/support/FAQ/1008000/#a2 https://www.asus.com/support/FAQ/1008000/#a2 Other comments from ASUS chat support: - Asus is "in the process of fixing the issue". - When a new firmware update is available "it will be just on the ASUS website to the router itself". [I presume they're referencing the router's web GUI admin tool?] - When asked for an 'official Asus forum or support site' to monitor for updates regarding this issue, they said: "We don't have a forum page for newly released updates but this should be posted on the ASUS website main page." ---- [0] ASUS chat support session ending at 12:32:46 PT on 2023-05-18
NBJack 3y ago
Unbelievable. I just ran into this myself. Same root cause: no space, old firmware.
hrseoservices 3y ago
[flagged]
hrseoservices 3y ago
[flagged]
KomoD 3y ago
https://www.asus.com/us/news/6ks8psh9bd8e8g0e/ https://www.asus.com/us/news/6ks8psh9bd8e8g0e/
Ninjaneered 3y ago
Still no complete answer about why those of us who didn't have automatic updates turned on were affected, but some updates: https://arstechnica.com/information-technology/2023/05/asus-fixes-error-that-caused-mass-router-outage-worldwide-for-2-days/ https://arstechnica.com/information-technology/2023/05/asus-...
abrookewood 3y ago
Asus have acknowledged the issue: https://www.asus.com/support/FAQ/1050466 https://www.asus.com/support/FAQ/1050466

736 points by zdw 3y ago

Anonasty 3y ago

Asus needs to get their act together. First the AMD mobo problem and now this.

ddtaylor 3y ago
I think at this point they have eroded enough brand trust it would be difficult to recover from. I'm not hyper-vigilant in this area, but I can say with confidence I have absolutely no expectation to do anything related to ASUS hardware for any of my upcoming tech purchases. That's a decision that will likely affect my purchasing for the next 5 years since I don't often re-build my PC.

delabay 3y ago

I had this exact issue today. Starting in the wee hours of the morning, the router would just crash every 30-40 minutes. I had to reboot it to get the router back. I updated firmware and it seemed to fix it..

deleted 3y ago
[deleted]
argc 3y ago
This happened to me too! After a day of dealing with the router crashing every 30 minutes, I finally updated the firmware and am hoping the issue is fixed now.

the-anarchist 3y ago

And yet people keep asking what the point of OpenWRT is.

awestroke 3y ago
Nobody is asking that
- smashed 3y ago
  Hmm maybe not but reading all the linked forum posts and threads, people are at a complete lost for information. "I installed version X.Y.Z and so far so good" "I tried all available updates and it keeps crashing". There is no transparency in the published updates. Take that firmware blob (which is probably 99% GPL licensed by the way) and shut up. That's basically what the vendors are saying. What's ironic is that most OEM are using some sort of out of date openwrt derivatives as a base for their closed source firmwares, which are obvious GPL violations.
  - Nextgrid 3y ago
    > What's ironic is that most OEM are using some sort of out of date openwrt derivatives as a base for their closed source firmwares, which are obvious GPL violations. Do you have a source of that or am I out of date? Based on my knowledge, outdated, GPL-violation OpenWRT would still be a massive upgrade over what's usually developed in-house by OEMs.
    - IntelMiner 3y ago
      Linksys WRT 1200/1900/3200/32X touted their OpenWRT based firmware as a feature (though those routers are about a decade old now, and their Wi-Fi chips have unfixed bugs due to Marvell selling their Wi-Fi division to NXP) Musk's "Starlink" router is also OpenWRT based
      - sgtnoodle 3y ago
        I use a WRT1200AC running openWRT as my router, wifi disabled, and it works pretty well. I use two WRT1900AC running openWRT to make a WDS bridge and can get about 400-500Mbps of iperf3 throughput, and one of the two serves as an access point. Even though the wifi does indeed have its limitations in the form of missing features, it's been reliable and performant for me.

albert_e 3y ago

Interesting. May be i am old fashioned but shouldn't hardware appliances be designed as standalone devices that have minimal external dependencies. If there was no firmware update or patch applied -- the functionality of the device shouldn't change. Do all routers these days "phone home" anyway and modify settings on autopilot? Even if user has chosen to turn off updates?

vorpalhex 3y ago
Routers tend to be complex devices that have vulnerabilities. This has led to bot armies of routers. Users won't apply fixes. Therefore auto-updates became the norm.
- akira2501 3y ago
  This _was_ a bot army of routers. It just so happened that the army's orders were to do something uncoordinated and useless. You can trust the users implicitly, or you can trust ASUS implicitly. It seems that the appropriate place to deal with network traffic security issues is with the network bandwidth provider, which would be your ISP.
  - derefr 3y ago
    > It seems that the appropriate place to deal with network traffic security issues is with the network bandwidth provider, which would be your ISP. This would imply that the ISP has abilities to interpose between malicious traffic and the user that, with modern technologies like DNS-over-HTTP, they simply don’t have. To most ISPs (that don’t force you to install MITM CA certs), user traffic is increasingly a black box. Which means that attacker traffic can blend right in as part of said black box. The only thing that can see into the black box is the device on the end of the line.
    - akira2501 3y ago
      In most cases the device is getting it's IP information on boot by negotiation or request from the ISP. All the ISP needs is a database of blacklisted equipment and some middleware box to receive connections from insecure home devices when it detects them. It's not about catching related traffic when it occurs, more managing the immediate implications from insecure devices on the network. The ISP can see the device, it doesn't need to inspect the traffic.

shmerl 3y ago

Wipe out Asus firmware and install something like DD-WRT to never have such problems again.

cmehdy 3y ago
Unfortunately it's becoming increasingly expensive to find routers for casual home usage that allow for alternative supported firmwares. A decade ago slapping tomato on something cheap would do the trick, but now with high speed internet and wifi 6 there aren't many devices having that flexibility within entry-level price ranges (at least from what I've seen in Europe and Canada).
- kelnos 3y ago
  [0] would seem to suggest there are 45 options for things with WiFi 6 that you can install OpenWRT on. 16 of them are supported in an actual release and don't require running a snapshot. Not amazing, to be sure, and I don't know what those devices cost, but I think the situation is less dire than your post would suggest. [0] https://openwrt.org/toh/views/toh_available_16128_ax-wifi https://openwrt.org/toh/views/toh_available_16128_ax-wifi
  - londons_explore 3y ago
    None of them are things your ISP will ship you free, nor things you'll find on eBay for $20. (Despite routers capable of gigabit speeds being commonly both free and available on eBay for pennies)
- mjevans 3y ago
  With the price of high end routers rising the last time I needed one (that could get near to 1gbit) I went with a mini board that was well supported by OpenWRT. Some of these can also support WiFi via laptop style PCIe add-in cards. https://openwrt.org/toh/views/toh_single-board-computers https://openwrt.org/toh/views/toh_single-board-computers
buccal 3y ago
If you plan on using third party firmware you should look at compatibility with OpenWRT BEFORE buying the device. Router manufacturers are unwilling and unable to provide a future-proof software. Latest proof of that in my case was with a new Wi-Fi 6 router Netgear WAX206. After indeterminate time clients would just stop receiving packets and would require reconnecting. WAX206 latest factory firmware is based on OpenWRT from 2016 (Netgear still refuses to open source parts of it). After flashing current OpenWRT the connection is rock solid.

userbinator 3y ago

This seems to be related to “ASUS Healing System” which I don’t even know if I have enabled or not. That name already sounds creepy enough, but searching for that string (with the quotes) currently returns only 4 results, of people asking what it is. My guess is some sort of hidden backdoor, disguised as an ostensibly useful feature.

atoav 3y ago
To me that sounds like some sort of adaptive signal conditioning/denoising/filtering system just by the name of it. So DSP magic?
krackers 3y ago
So what exactly is the Asus Healing System? I can't find any details about it online.
- ddtaylor 3y ago
  I assume that ASUS routers are based on Linux, so shouldn't the source for these routers be readily available? I am able to find custom third party mods (asuswrt-merlin) but I can't actually find a clear copy of the original sources! This is very common when I look to find source for embedded devices like this. What I expect is the next step is that you will find (or be given) a borderline useless blob of source that doesn't explain any of it's build process, which is absurd because the GPL clearly defines the build "glue" as part of the source. Is ASUS another company that is doing a poor job of GPL compliance in this space? Is it intentional?
  - IshKebab 3y ago
    Why would ASUS release the source code for its router firmware? It doesn't have to be GPL.
    - ddtaylor 3y ago
      > AsusWRT is a derivative of Tomato which is itself one of the descendants of HyperWRT, a Linux distribution for low-end network appliances such as routers.

gkanai 3y ago

I have a set of ASUS mesh routers and have been experiencing this issue today. Glad I see it's not my specific problem although it looks like I have to watch for firmware updates.

garbagewoman 3y ago
probably consider swapping them out for something less sus, too
- CincinnatiMan 3y ago
  Any recommendations for someone that doesn’t spend much time tinkering these days?

legohead 3y ago

This happened to me this morning. I even called Cox, since they were out just last week doing scheduled maintenance and the internet went down a few times. Cox said the line looks fine (and tried to sell me their cable modem). I logged into my router and checked for new firmware, then installed it, and it started working again.

chillbill 3y ago

I have the same device, didn’t notice any issues, possibly because I’m using Asuswrt Merlin https://www.asuswrt-merlin.net/ https://www.asuswrt-merlin.net/

rlbaker 3y ago
Did not experience any issues with my RT-AX86U running Asuswrt-Merlin either.
kqr 3y ago
Oddly, I'm also running Merlin but I did have a problem where my laptop thought it was connected to the AP but couldn't get local network traffic routed to it. Easiest solution turned out to be to just reconnect, so I don't actually know the deeper problem. I don't think it has any relation, but since it's the first time it's happened, it was kind of a freaky coincidence!
cheald 3y ago
Ditto, RT-AC68U running merlin with no issues.

chx 3y ago

Never put a router without open source firmware on the open Internet. It's that simple. The right tool for the job. (OTOH I advice against using Linux as your daily driver on any modern laptop. Once again, the right tool for the job and that's Windows + WSL.)

kelnos 3y ago
> Windows + WSL Hard pass. Not interested in Microsoft's latest malware/adware of an OS. I've been running Linux on laptops for nearly 20 years now. Sure, it was a pain in the ass in the earlier days, but nowadays things just work, as long as you're willing to do your research ahead of time to choose a laptop where the hardware is fully supported.
- ricardobayes 3y ago
  You're biased. Win 11 doesn't have third-party ads of any kind, unlike iOS's app store, for example. It was always news, videos etc. Recommended content. Moreover, ubuntu famously had an Amazon button on the home page, yet no is going around touting "Linux is riddled with adware".
  - fsflover 3y ago
    I looks like Ubuntu was created just in order to be able to dismiss Linux as "riddled with adware". It's just a single distribution out of a hundred, and far from the best, so it's completely wrong of course.
    - rejectfinite 3y ago
      >I looks like Ubuntu was created just in order to be able to dismiss Linux as "riddled with adware". Yes of course it was. The only reason. Delusuional. You think some conspiracy created Ubuntu so people can point to the Amazon thing?
      - msla 3y ago

inamberclad 3y ago

Problems today too. This is a huge pain and really making me question my router. What's a good hardware platform for OpenWRT?

aetherspawn 3y ago
I can't recommend the tp-link AX50 highly enough. It has been rock solid and stable now for probably around 4 years. I don't remember ever having to reset it. I recommended it to my friends when they changed their routers over, and it fixed a lot of issues for them as well. The ones who regularly used VR reported that the stuttering issues they had on VR over WiFi with their Netgear WiFi 6 routers were solved. Have owned both ASUS and Netgear in the past and the experience was terrible -- the high-end ASUS router regularly bricked itself during auto update. So I bought a more expensive ASUS router for around $600, and you guessed it, it did the same thing just less frequently. The NETGEAR routers never bricked themselves, but across the 3 models we've owned nearly all of them would lose most of their throughput (around 90%) on a daily basis and would need a daily reset to get proper ISP speeds again. I had a Dream Machine Pro for around a year before I moved it to my office to run the network in the office. Funnily enough it has probably been less stable than the tp-link router, but nonetheless the UDM Pro is still a great piece of kit and probably the last router you'd ever need if it's in your budget and you have a place to put it (awkward form factor for home). Re: Open WRT I ran this around 3 years ago on a D-Link router that I had, and the stability was unpredictable.. better than ASUS and NETGEAR but probably leaning on the side of poor compared to tp-link and UDM Pro. It would randomly get slow once per fortnight or so, and would need to be rebooted. Sometimes it would panic and completely lose internet connection. There are plugins designed to reboot the router as soon as a loss of connection is detected, but why is this even necessary? It's annoying too. Drops all your connections. Sorry for long.
inamberclad 3y ago
I followed one of the solutions in the comments and removed /jffs/asd/chknvram20230516 and the issue seems to have gone away. CPU usage went from 70-90% back down to 5%.
nvarsj 3y ago
I have one of those small Intel based nuc like devices and run x86 openwrt on that. It works great on a gigabit ipv6/ipv4 line. I also replaced the proprietary bios with coreboot for complete security.

55873445216111 3y ago

I had the same issue this morning on my RT-AC68U.

dude3 3y ago

I had issues too. Thought it was just my home network. After a few reboots of every device on ai mesh, was able to install the latest firmware. It mentioned it fixed a memory leak. I have 2 RT-AX55's.

crazy5sheep 3y ago

I thought it was just my asus router broken today, and I was about to buy a new one.

garbagewoman 3y ago
About to buy a new Asus router?

sneak 3y ago

This is the danger of allowing your vendor to automatically remotely execute arbitrary code on your devices.

yosito 3y ago
He had auto updates turned off, so there was no "allowing" happening. It's not even clear yet what happened, but one possibility is that this was someone taking advantage of an exploit on routers that had NOT been automatically updated.
- garbagewoman 3y ago
  yes, as you suggest, it appears that Asus routers were targeted by a 0day that they are known to lack the skills to deal with

kenhwang 3y ago

Oh, that explains why my connection was slow from some rooms in the house today. I also have an ASUS RT-AC86U (running 3.0.0.4.386_51255), but as a node in the mesh, it's still online. However, my RT-AX55 (3.0.0.4.386_50460) in the mesh has been down all day and I haven't noticed before this post because the other nodes picked up the slack (with spotty/slow signal). Also have an RT-AX82U (3.0.0.4.388_22525) as the mesh leader and it also doesn't seem impacted.

ok_dad 3y ago
I had no issues with my 2 ax-92u routers in a mesh on the newest firmware 3.0.0.4.388_22525.
- rahimnathwani 3y ago
  Ditto.

vrglvrglvrgl 3y ago

[dead]

olliej 3y ago

Wow I had this happen this morning with our axe6600/et8/whatever and assumed it was something actually failing

hbcondo714 3y ago

The power of blogging and HN. Asus or my ISP didn't tell me why my router/internet went out twice today. I honestly thought thieves stole my copper again: https://news.yahoo.com/rise-copper-theft-officials-concerned-195400800.html https://news.yahoo.com/rise-copper-theft-officials-concerned...

DANmode 3y ago
and brought it back in between? :D
- TeMPOraL 3y ago
  Maybe they were spotted by the police and had to pretend they're there installing the wires so they don't get caught.
itsphilos 3y ago
again?
mad0 3y ago
I love when thieves are trying to steal copper, but all they get is a broken fiber optic ;) My dad works as a network engineer, and he told me a story that one of the banks in Poland lost one of the internet providers. They investigated and found out that thieves stole hundreds of meters of a fiber cable, because they thought that it's copper.
- garbagewoman 3y ago
  that sounds like so much fun for all involved
- maven29 3y ago
  Don't they typically have co-extruded copper in the conduit/pipes for Utility Locators to use, even if it's just fiber inside?

pendu 3y ago

Unbelievable. I struggled with this the whole day. Checked the ISP status. Thought, it could be overheating (router is in the garage) or bad RJ-45. Had to reboot multiple times today. Never ever I would have guessed it was a firmware issue. Thank you for writing this blogpost. I just upgraded firmware. So happy when something has an explanation. !

garbagewoman 3y ago
Why would your router overheat in the coldest place in the building?
- tyingq 3y ago
  The garage could be the coldest area in the building, or the hottest. Things happen, too...dust clogging ventilation, animals build a nest around it, router placed near refrigerator coils, etc.
- NBJack 3y ago
  I appreciate that you must live in a cooler climate. This is not the case everywhere. My garage cooks on a bad day. Great for the hot water heater, not so great for everything else.

jms703 3y ago

Good reason to consider running openwrt, opensense, vyos, or something like that, if you’re up to it.

baq 3y ago
MikroTik here, can recommend
- AlbertVAustin 3y ago
  MikroTik is definitely very nice (I'm running my home network with them ~7 devices). Too bad it's not open-source like OpenWRT is :-/
  - intelVISA 3y ago
    I like MT but it being closed source always irked me for such a critical device: wonder how many Ghosts in the Shell lurk in there...
    - AlbertVAustin 3y ago
      I'm actually not that worried about them secretly doing something malicious (although that's also a valid concern) but rather in the "given enough eyeballs, all bugs are shallow" and the general risk factors (MikroTik going bankrupt and we're left with obsolete bug-ridden hardware). Sadly given the large amount of software that's closed source there I don't see them working towards open sourcing any of it as, even technically, it would be a massive effort.
      - intelVISA 3y ago
        Agreed, even if it was not their intent it being closed source is malicious itself - we need to promote this discourse in public conciousness so people are less afraid to question the unaudited, outsourced binary blobs that fill their day to day lives.
- 3y ago

simongray 3y ago

My Apple Airport router also crashed yesterday and had to be restarted. I wonder if this is related?

PeterStuer 3y ago

So far things seem fine here. I run an RT-AC87U (3.0.0.4.382_52545-ga0245cc) as the main router I have an RT-AC86U (3.0.0.4.386_51255-g486ded6) and an RT-AC68U (3.0.0.4.386_51255-g486ded6) running in AP mode. Tried checking whether there was an update available for the RT-AC87U, but it looks like the ASUS firmware site is unreachable (overloaded?) for me at the moment.

PeterStuer 3y ago
And the AC87U got bitten today. Running out of memory until dnsmasq can't allocate anymore. Reboots help only temporary.

dack 3y ago

Glad this wasn't just me - but we were going mad for a couple hours there not understanding what was going on... I still want to know what happened. It feels like a good excuse to get a new non-asus router barring any other explanation

bboygravity 3y ago

TLDR: someone at CCP and/or NSA tried out a backdoor in ASUS routers and did an oopsy.

temptemptemp111 3y ago
[dead]

JdeBP 3y ago

I don't have an ASUS router, but three things leap out at me: a string being logged over and over, running out of space on a filesystem, and rotated log files named something.1. It is trivially easy to blow right past the size capping on systems that use the old "newsyslog" style of external logfile rotation from the 20th century, and something that is logging a short string "[chknvram_action] Invalid string" over and over very fast is exactly how to do this. For those interested in investigation, therefore, I would suggest looking at logfile sizes, and seeing whether it was logs eating all of the free space on /jffs and /var . The underlying cause would be whatever is logging "[chknvram_action] Invalid string" thousands of times over, but the mechanism would be log files filling the tmpfs that the article mentions, which would explain why the system had no memory for forking new processes. My wild speculation about "[chknvram_action] Invalid string" is that something somewhere in whatever "chknvram" is, the name being suggestive of something checking non-volatile RAM, has either bad data or a broken parser, and the recovery semantics are to retry immediately, incessantly, as fast as possible.

bartvk 3y ago
Weird how syslog doesn't de-duplicate these identical log messages. Or maybe it does, but not enough.
- inferiorhuman 3y ago
  BSD syslogd definitely does, perhaps whatever stripped down one ASUS is using doesn't. e.g. Dec 2 01:09:41 hostname syslogd: last message repeated 10 times The threshold's pretty low and most of the "repeated" messages say "repeated 1 times" however.
  - TeMPOraL 3y ago
    Does it store timestamps for those repeated occurrences? I wouldn't want my logs to "helpfully" coalesce multiple identical messages into a single one. For example, if I saw the equivalent of the text below in a program/OS while facing some issues, I'd be remarkably pissed. [2023-05-18 07:12:21] [E] Invalid event 0x1AF2 received from ... [2023-05-18 09:44:01] [I] Last message repeated 10 times I care less about how many times the message was repeated - I care about timestamps, which I might want to correlate to other activities.

Ninjaneered 3y ago

Super interesting, the same thing happened to me around 4am PDT, I have 2x ASUS ZenWiFi ET8 mesh routers. Every 10 minutes or so, we'd loose internet access for a minute, then it would come back. I called Charter and they blamed the router, but I was a bit suspect since in January of 2022, we had another issue where the call center blamed our router and after a bit of testing was able to prove to them it wasn't our router. This was kind of interesting as well, they finally sent out a second service tech that had some sort of spectrum analyzer and he found our line it was being overwhelmed with noise. Turns out the drop from the pole wasn't shielded and a new T-Mobile 5G tower was bleeding into the line (acting like an antennae). The tech put a new wire up and it immediately solved the issue. In both occasions, the help desk told us the signal coming to the router was fine though. This time a factory reset seemed to do the trick, but I seemed to have the latest firmware to start with so I'm not sure what's going on.

garbagewoman 3y ago
You're not sure what's going on, despite this new information?
- egberts1 3y ago
  It would appear that he is not the troubleshooter but a trouble reporter.
  - Ninjaneered 3y ago
    Yeah, I did a factory reset before learning about other solutions here. I'll have to go through the thread again, but I'm not sure we know what the root cause is yet? Log files were filling up, but I didn't catch why.

gmac 3y ago

Interesting. My CT8 mesh routers seem OK and have no new updates available so far.

deleted 3y ago

[deleted]

theonlybutlet 3y ago

Vodafone gigabox did the same today. Never been sure who manufactures it?

shampto3 3y ago

Interesting. I have asus zenwifi mesh routers and today one of the nodes went offline and it took me a while to realize and then fix it. The fix was manually updating all of the firmware for every mesh router, so it seems like it was a similar problem. EDIT: crap, I just checked again and two of my nodes are offline now. Only the root node is working. Too late in the evening to fix it now, guess I’ll be working on that in the morning.

JdeBP 3y ago
The local fix is, reportedly, to delete the malware signature file that is breaking ASUS's proprietary malware scanner. * https://news.ycombinator.com/item?id=35984706 https://news.ycombinator.com/item?id=35984706 * https://news.ycombinator.com/item?id=35984725 https://news.ycombinator.com/item?id=35984725 The service fix is for ASUS to make a malware scanner that doesn't have such bad behaviour in the case of a signature file that it cannot process, and to not use a logging mechanism that we knew had this flaw in the 1980s, and improved upon in the 1990s. (-:
garbagewoman 3y ago
UPDATE: the fix appears to be swapping out the ASUS routers for anything else
shampto3 3y ago
UPDATE: I followed the instructions in https://news.ycombinator.com/item?id=35986375 https://news.ycombinator.com/item?id=35986375 for each of the nodes and it appears to have solved the problem, at least for now. I will keep an eye on them to make sure they don't die again. The frustrating part was that my home assistant raspberry pi was hardwired into one of the nodes that died, so my automation to change my thermostat wasn't working and it was stuck on a setting that I normally only set for one hour in the day. Obviously I could change it manually, but I didn't realize that it hadn't changed until it had been like 5 hours.

e3bc54b2 3y ago

Boy am I glad I replaced stock firmware with OpenWRT the moment my router came out of box last week. It was also extremely painless experience, and I'd really recommend people to buy routers with OpenWRT support, even if they cost a little more. A router is something you buy for a decade or more, and it's worth the investment. Our livelihood depends on network availability, and depending on whims of terrible router firmware is not something to rely on.

gat1 3y ago
OpenWRT is a bliss to setup a router. We configured a lot of little customed subnetworks at work and it made everything so much easier.
- intelVISA 3y ago
  Can't reccommend OpenWRT enough, really solid.
iforgotpassword 3y ago
I've given up on trying to find one with perfect openwrt support. Something is always broken, mostly hardware offloading, so speeds suck.
- wtallis 3y ago
  The kind of hardware offloading included in consumer router hardware is fundamentally broken by design. Relying on the ethernet switch to handle NAT instead of the CPU makes it impossible to do software-based QoS (eg. the SQM module's cake or fq-codel) or any other packet processing that said ethernet switch isn't equipped for.
enlyth 3y ago
There's also this one for Asus routers, I've been using it without issues for a long time: https://www.asuswrt-merlin.net/ https://www.asuswrt-merlin.net/

ww520 3y ago

I used to have the same model of ASUS router for several years. It lost internet connections quite often. Had to reboot to get it back. Also the performance felt sluggish. When WiFi 6 comes along, I just replace it with a router from another brand.

vardump 3y ago
Wifi 6 has been around for about 4 years. Maybe you meant Wifi 7?
- garbagewoman 3y ago
  some of us are using ASUS, so wifi 6 is news to us, along with the news of how devastatingly terrible our routers are
  - vardump 3y ago
    Wifi 6 routers such as Asus RT-AX56U etc. were released back in 2019.
- ww520 3y ago
  It’s Wi-Fi 6. I didn’t get it right at the beginning of the release cycle.

sampa 3y ago

People here recommend openwrt. For Asus routers it is wrong. Install Asuswrt-Merlin.

pixelpanic360 3y ago
[dead]
egberts1 3y ago
You should list the benefits of ASUS-Merlin over ASUS here. Leaving us ASUS users dangling with that comment is like leaving us hanging in there ... pondering WHY?
- drunner 3y ago
  https://www.asuswrt-merlin.net/features https://www.asuswrt-merlin.net/features

RektBoy 3y ago

I cringe every time I see these "gamer" routers.

Aeolun 3y ago
They look horrid, but they work surprisingly well.

Sarkie 3y ago

I just installed 2 ASUS routers in my house on Monday. What a ride.

garbagewoman 3y ago
what led you to this decision?

adamparsons 3y ago

I lost my absolute mind this morning trying to figure this out. Dual wan and neither working. Got a few hours into the work day and ended up digging out an old WRT router just to get some work done. I saw a bunch of kernel panics and assumed the router finally gave up a the ghost after all these years. I guess I can go fish it back out of the bin

user070223 3y ago

I had brief disconnect from one of my routers earlier Now I try to update the ASUS firmware and get this Error "The router cannot connect to ASUS server to check for the firmware update. Please download the latest firmware from ASUS Product Support Site and do manually upgrade. Check on FAQ for more instructions."

shekhar123456 3y ago

[flagged]

rjmunro 3y ago

Any word from ASUS on this? If not, I know to avoid ASUS products in future.

garbagewoman 3y ago
If you have reached this point in time without having extreme distrust in the company, you may be interested in a bridge I have for sale at an extreme discount

tanepiper 3y ago

Ahh this explains why my upstairs router had no internet.

Hard_Space 3y ago

I am lucky enough to live in a time-zone where this crisis happened while I was asleep. Woke up to find that some network event had clearly happened, but that the pending Asus AC88U update that I had been putting off for months auto-triggered. So far, all normal.

rswail 3y ago

Off topic, but where can I find: 1. A modem that handles ADSL/VDSL/fibre incoming and 10GB ethernet outgoing with passthrough to: 2. A good wifi 6 mesh router + APs (eg Netgear or equivalent). I don't want them to do anything but: a) Support both DHCP and fixed assigned IPv4 configuration b) IPv6 /56 subnet assignment c) NAT outgoing IPv4, no incoming connections allowed d) IPv6 ingress/egress I don't want them scanning the traffic, protecting me from malware, upgrading themselves, doing anything fancy. I want the bandwidth of my home ISP connection to be supported and I want the bandwidth of my 5G wifi and 10GB internal LAN to be fully supported.

stingraycharles 3y ago
I’m a big fan of Ruckus for anything wireless, but they’re not cheap. But they’re extremely reliable, high-quality hardware that work especially well in multi-AP setups. You’ll probably want a separate modem+router device, though.
sschueller 3y ago
Your best bet is still to run your own hardware and software like opnsense [1]. An old PC would do or if you need more a newer one. You can easily add cards for 10 or 25 GBit and upgrade later I needed with more memory etc. [1] https://sschueller.github.io/posts/wiring-a-home-with-fiber/#router https://sschueller.github.io/posts/wiring-a-home-with-fiber/...
denkmoon 3y ago
A modem that handles ADSL, and VDSL, and Fibre, with 10GBASE-T? Why would such a thing exist? All ADSL and VDSL connections known to man will have adequate bandwidth over 1000BASE-T, simple and cheap modems that support RFC 1483 bridging abound. If you've got >1Gb/s fibre, presumably your ISP provides equipment to support such, and why not simply have a router with SFP+?
- Aeolun 3y ago
  > If you've got >1Gb/s fibre, presumably your ISP provides equipment to support suc Why would they do that? If they sell me 10GBit but I can only reasonably use 1, then they can sell the same thing to 10x the people with the same hardware.

nuc1e0n 3y ago

The timestamp in the article of 1684335272 is oddly familiar to me. 16777216 is the maximum value you can store in 24 bits, but imagine if asus were storing the fractions of a second (0-100) in an 8 bit number, packed together with epoch seconds in a 32-bit integer. Maybe this was some timestamp related overflow?

easytiger 3y ago
It's Linux. It will be from a timeval struct or similar. It translates to Wed May 17 2023 14:54:32

Tor3 3y ago

Nothing happened to my RT-AC68U, and I just checked - it's (at least) a couple of firmware releases behind. I also checked my RT-AC2900, and nothing happened there either (I haven't checked the firmware version, but I have not updated anything since I bought it somewhen last year)

Gabrys1 3y ago

Fix: 1. Reboot the router via pulling and reconnecting the power cord 2. Log in to WWW interface 3. Go to Administration > System. Enable SSH (enable login/password as well, choose a port of your taste) 4. SSH to your router: ssh admin@192.168.50.1 -p 2424 (assuming your user name is admin, the IP is 192.168.50.1 and you chose port 2424 for ssh). Password is the same as for the web UI 5. In SSH session, type: rm /jffs/asd/chknvram20230516 6. In SSH session, type: reboot Seems to have done the trick for me.

wincy 3y ago
I’d recommend renaming the file instead of straight up deleting it.
- scrapheap 3y ago
  Given the error they had in the log was no space left on device, then the fix probably does require the removal of the file.
  - shawnz 3y ago
    It was no space left on the tmpfs, since the bug causes some kind of memory leak. It's not a matter of disk usage.
- smileybarry 3y ago
  It’s still used by asd next time it launches, so you have to at least move it out of that folder.
Aeolun 3y ago
I mean, this might work for me, but how do we deal with the fact that if I were my wife ASUS has basically turned my very expensive router into a very expensive brick?
- Zolt 3y ago
  Agreed. I had enough issues with my router yesterday I purchased a different brand router and decommissioned my ASUS. I was not aware of this wide-spread issue until today. Much like HP printers, I will think twice about recommending or purchasing an ASUS.

daneel_w 3y ago

I use a $25 Asus "Wi-Fi repeater" which runs OpenWRT instead of their crippled firmware, allowing it be a plain Wi-Fi access point acting as a VLAN extension of my little SoHo switch. An entirely uncomplicated setup that just won't misbehave on its own.

egberts1 3y ago

I deleted /jffs/asd/chknvram20230516, rebooted, and back to normal. Pesky upgrade. Now have full wire capturing going on on public side.

citrin_ru 3y ago

Does anyone knows why Wi-Fi routers vendors keep writing (and maintaining for a short period) own firmware which in most cases has lower quality than OpenWRT when they can use OpenWRT and save money?

alibert 3y ago
Actually, there is a large number of manufacturers using OpenWrt as a baseline for their firmware. Unfortunately, these are often outdated version of a forked OpenWrt with heavy modifications to include firmware blob for hardware acceleration and such.
pinebox 3y ago
Because router vendors need some kind of "secret sauce" in order to keep selling routers, otherwise the whole thing becomes a commodity you can buy from anywhere. This is also why OpenWRT is unable to utilize all the features on a lot of routers: The closed source firmware and software pieces needed to do so have yet to be deciphered.
- citrin_ru 3y ago
  Vendors can ship OpenWRT + closed source drivers. Anyway they use Linux kernel (yet to see a closed source BSD router; PfSense is occasionally used but its open source) relying on GPL exceptions allowing to use closed source drivers (and firmware blobs).

vijucat 3y ago

> "The fact that they were both at exactly 23 seconds is probably just a crazy coincidence" Immediately brought a smile to my face. This profession makes us detail-oriented detectives. It's a joyful power and a powerful joy :-) Now I know it's not a good personality trait. I guess one's hobbies should ideally be complete opposites, based on intuition and a feel for things.

chatmasta 3y ago
I can definitely relate. This sort of pattern recognition is a blessing and a curse... it's great when it saves some time during the hypothesizing phase, but not so great when it leads you on a wild goose chase :)
kraussvonespy 3y ago
https://xkcd.com/2259/ https://xkcd.com/2259/ I found that one just after spending all day chasing an odd WAN issue for a customer. A few years ago in the era of half/full duplex in ethernet, duplex mismatches sometimes manifested in some truly weird ways. It was at that point that I did start to believe in ghosts. At least ghosts in the network.

lightlyused 3y ago

My ac66u-b1 has been acting weird for the past several months. I first thought it was an Ubuntu bug. Wifi wouldn't always reconnect after I'd closed my laptop. Yesterday, however, it seemed to just die randomly. Error was "no space left on device." SSHing in to it and it does show root volume as 100% used. So, whatever Asus was doing has pushed it over the edge space wise and it can no long write the dnsmasq.leases file when enough devices connect. So, what wifi ap should I buy to replace this POS?

trey-jones 3y ago
Not offering advice on purchasing an AP, but if you are interested in trying to get more life out of this one: My guess based on reading the article is that it's mostly logs filling the disk. You can find out for sure with some version of du -a | sort -hr | head Then just start deleting logs at or near the top of the list - I suppose this would get you back running. Sure, constantly writing to logs sounds terrible for performance, but did you even notice when it was working properly? If it's actually Ubuntu running on this router, you could even install (or configure if it's already there) logrotate to probably prevent this from happening again at all. Then you still have a POS router, but one that might get the job done.

cloin 3y ago

I was fighting with this all day. I have a 3 pack Zenwifi XT8 that has always given me trouble so I didn’t even think to check if it was a widespread issue. I’ve just been enduring the pain long enough that my baseline pain is already so high. I originally bought these on a recommendation while I was looking for gigabit mesh systems. I’m back in the market after yesterday.

jeremyjh 3y ago
Eero 6 Plus, it will change your life. The things just work with no hassle, and I have very good coverage and throughput even out on our patio.
- Skates1616 3y ago
  Running the same in 6 family member homes, I have not had one “tech support” call needed since installation last year. I moved them all from the original OnHubs which were also set it and forget it…

Eumenes 3y ago

I checked my logs and my T-Mobile (AC-1900) seems unaffected ...

XxCincinnatusxX 3y ago

Same router here & it did that to me all day yesterday but I'm not as tech savvy as some of you guys so I don't have a clue how to fix it. Hopefully I'll figure it out by the end of the day

ncphil 3y ago

Probably the most painful home Internet experience you can have is a consumer router fail. Suddenly you really need all those basic sysamin features a business system provides. I used OpenWRT on Linksys back in the day, but later had an early PCEngines APU running PfSense. Those worked well enough until my ISP increased our bandwidth. Made the jump to a little Mikrotik RB2011 on the top floor and added an extra AP (actuall a RB 900 series router) on the first. That did the job up until we got 1 Gb fiber. Not willing to use the ISP's router+mesh, I got a Ubiquiti EdgrRouter and a couple of Netgear APs (by then all our switches were ProSafe managed). The ER has been great. After mastering RouterOS, Vyatta was no big deal. I regret now not getting UI APs, Netgear changes its crappy prosumer interface with every model it seems, and they're painful to manage without paying for the cloud service. The switches are OK, but the once again the UX annoyingly varies by model. Of course, Ubiquiti has its own issues mostly in how the company itself is run. If I have to, I'm ready to go back to Mikrotik (whose web ui the Vault 7 goons could only crack during an interlude between firmwares: as if anyone leaves that UI enabled). In all the years I ran it, Mikrotik never caused me a problem with a RouterOS update (neither has EdgeOS).

rejectfinite 3y ago
>Suddenly you really need all those basic sysamin features a business system provides It's good for a learning experience :) And now, the ISP routers have caught up imo I just use that now and it is fine. Plug in as much as possible with Ethernet. If that fails, it's probably a router issue or ISP network issue, and then it's on the ISP side to fix.
- hedora 3y ago
  In the US, ISPs sell all sorts of network snooping data to advertisers and the police. Letting one of their devices on to your LAN is asking for trouble.
l33tman 3y ago
To be fair, "business system features" are no help often, I have a $1200 Netgate pfSense firewall that just failed, probably due to the Intel Atom C2000 bug. I just have to throw it away and buy a new. That it was a business thingy didn't change anything except made it much more expensive to replace.

rejectfinite 3y ago

Right now, I just use the ISP modem/router combo. DOCSIS. Plug as much into ethernet as I can, mobiles and tablets on wifi. Laptop even using a dock for work, much more stable than wifi as I am in an apartment.

xmdx 3y ago

Is there any information on why this started happening in the last 2 days? Without any change to the firmware? Considering flashing asuswrt-merlin to avoid this in the future.

51Cards 3y ago

I have an AC-68U but had been keeping my firmware up to date. Fortunately it has been rock solid, including yesterday and today.

funOtter 3y ago

Any recommendations for good 4-6 port wifi routers?

tgtweak 3y ago

I stopped using Asus routers around the RT-68U generation since that was the third one where the radios burnt out on it. Been using tplink since and 0 issues. Asus has been pretty good about open source firmware and being relatively transparent about their operating system and hardware, but I can't afford to replace my router/AP every 2 years because of a crappy radio IC that is prone to failure.

carbocation 3y ago

Wait. Is it happening again this morning?

mydogmuppet 3y ago

Oh there is a god in heaven. When Asustek ran out of overpriced RT AX86Us I bit the Mikrotik bullet. Bought a hAP Ax3. Works like a dream, medium painful learning curve.

Havoc 3y ago

I've got 2x ASUS RT-AC92U on my network and all looks pretty normal on smokeping-like metrics.

t312227 3y ago

hello, idk ... but if your model is in the device compatibility-list, your should always be able to change to an alternative FOSS firmware. for example to https://openwrt.org https://openwrt.org or to some maintained tomato-clone like https://freshtomato.org https://freshtomato.org ps. i'm a happy owner of a 10+ years old RT N16 - it runs really great on openwrt ... cheers, v

bioemerl 3y ago
You can also build an X86 pfsense box that you can upgrade over time just like any other computer.
stevefan1999 3y ago
The problem is those routers usually have secret kernel modules that enables hardware acceleration, for example NAT offloading engine and ASIC acceleration for routing. Those are bound to a specific kernel version and is hard to reverse engineer for open source use because some of them are obfuscated binary blob

replwoacause 3y ago

Same exact issue here on my RT-AX58U. "dnsmasq-dhcp[1489]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)”

johnbellone 3y ago
I have the same on my router, too.

KomoD 3y ago

RT-AC58U, no issues.

xtat 3y ago

From the top comments it seems like we're skipping the fact that the guy has automatic updates turned off and it's pulling down and execing arbitrary code anyway.

ed25519FUUU 3y ago
There’s many, many things going wrong here.

sharno 3y ago

Wow, I thought I only had this problem. Contacted my ISP and they checked on their end and everything was fine. So I reset the router and it seems to be normal now.

wood_spirit 3y ago
It will likely crash again in a few minutes, then … :( I’ve just restarted mine and gone into the admin page and told it to update its firmware. No expectation it really helps, but it seems a good step before sshing in to delete a file…?

synergy20 3y ago

I have been running openwrt on my router for decades, buy one that openwrt supports.

jbluepolarbear 3y ago

Is this device specific? I have the RT-AX82U (great router by the way) and haven’t had any issues.

pnemonic 3y ago

This was driving me up the wall last night and this morning. Worst part is that fiber infra is being laid in my neighborhood, so I had it chalked up to my ISP changing hosts or something and it going sort of badly. Was planning to call them tonight if it was still f'ed when I got home. Fortunately, I checked for a firmware update from the car before I left and found the golden patch. Kicked it off and left for work. Figuring all is fine now. What a wild trip though. Wild trip? Man I am getting old.

tinglymintyfrsh 3y ago

These sort of vendor YOLO changes are more of the domain of Microsoft's MDE. I use an Deciso OPNsense where it doesn't insist on breaking itself on the orders of corporate overlords.

cfield 3y ago

fwiw, for the RT-AC68U, ASUS currently[0] recommends downgrading the firmware to the version at this link: https://dlcdnets.asus.com/pub/ASUS/wireless/RT-AC68U/FW_RT_AC68U_300438651255.zip?model=RT-AC68U https://dlcdnets.asus.com/pub/ASUS/wireless/RT-AC68U/FW_RT_A... To downgrade, they pointed to the 'Method 2: Update Manually' section of this support FAQ: https://www.asus.com/support/FAQ/1008000/#a2 https://www.asus.com/support/FAQ/1008000/#a2 Other comments from ASUS chat support: - Asus is "in the process of fixing the issue". - When a new firmware update is available "it will be just on the ASUS website to the router itself". [I presume they're referencing the router's web GUI admin tool?] - When asked for an 'official Asus forum or support site' to monitor for updates regarding this issue, they said: "We don't have a forum page for newly released updates but this should be posted on the ASUS website main page." ---- [0] ASUS chat support session ending at 12:32:46 PT on 2023-05-18

NBJack 3y ago

Unbelievable. I just ran into this myself. Same root cause: no space, old firmware.

hrseoservices 3y ago

[flagged]

hrseoservices 3y ago

[flagged]

KomoD 3y ago

https://www.asus.com/us/news/6ks8psh9bd8e8g0e/ https://www.asus.com/us/news/6ks8psh9bd8e8g0e/

Ninjaneered 3y ago

Still no complete answer about why those of us who didn't have automatic updates turned on were affected, but some updates: https://arstechnica.com/information-technology/2023/05/asus-fixes-error-that-caused-mass-router-outage-worldwide-for-2-days/ https://arstechnica.com/information-technology/2023/05/asus-...

abrookewood 3y ago

Asus have acknowledged the issue: https://www.asus.com/support/FAQ/1050466 https://www.asus.com/support/FAQ/1050466