14 ms·

How Asus and a Microsoft Bug Almost Broke Remote Work

122 points by c0nsumer 4y ago

pixl97 4y ago
Wow, what a nightmare to troubleshoot and try to figure out. I feel lucky that a month before the March 2020 I switched to a new job doing more direct application support instead of general desktop/AD related work or I may have had to deal with that exact issue.
- c0nsumer 4y ago
  OP here. It was... I was really fortunate that some other company had figured it out in parallel with MS, because I was hitting a dead end. I got to the point where I was pretty sure that WPAD wasn't working right, and was in the midst of capturing all sorts of data and escalating that, when a side conversation prompted someone to ask "was it you I worked with on X?". Then it all came together. I think this is one of those things that ends up being so esoteric most folks haven't worked on it, then the chain of events leading to it... But the moment I heard the trigger I was able to repro it... That was quite a relief.
josephcsible 4y ago
Why do so many companies require proxy configurations to be set on the endpoint, rather than using transparent proxying? Wouldn't that completely avoid a whole class of issues?
- 9dev 4y ago
  SSL certificates would be my first guess.
  - josephcsible 4y ago
    Don't browsers use CONNECT to access HTTPS sites over a proxy? Isn't the certificate situation exactly the same inside of that vs. with transparent proxying?
- tryauuum 4y ago
  hm... Could it be related to malware? If there's a malicious device in network, it would need a proxy configuration. With a transparent proxy back-connecting to attacker would be easier On the other hand, attacker could probably always send messages out via DNS tunnel Please someone more knowledgeable do enlighten us
  - c0nsumer 4y ago
    Malware can easily just pick up and use the system proxy settings. Hell, on Windows it just has to use WinHTTP and it'll use the system's paths to the internet.
    - tryauuum 4y ago
      Yeah, but if you don't set system proxies and instead you set it through environmental variables only for apps that need it?
- c0nsumer 4y ago
  OP here. This isn't quite my space, but from what I recall there's a couple wrinkles with transparent proxying. It's overall a good idea, but has some edge cases: - Authentication can go sideways in weird ways. A 407 from what looks like the correct site can cause odd things. IIRC there isn't great support from vendors for authenticating transparent proxies, too. Sure, you could auth off the machine instead (X device is on the network, therefore it's allowed), but what about shared machines... Proxy auth as user is better, because it allows requests to be tracked to a user ID and login session, not just a device. - Client getting load balanced between proxies during a session can trigger reauth (because auth sessions not shared between proxies), this can confuse a client. - HTTPS sites can get weird. - Routing in very large private environments can complicate default routing to the public internet. Although this can be handled by doing transparent auth + optional manual config.
Randor 4y ago
As the blog correctly identifies, the WPAD specification is still in draft: https://datatracker.ietf.org/doc/html/draft-ietf-wrec-wpad-01 https://datatracker.ietf.org/doc/html/draft-ietf-wrec-wpad-0... In other words there really isn't a standard to follow. You can't add features to operating systems if there isn't a formal specification. I'm not sure who's to blame here.
- justsomehnguy 4y ago
  > You can't add features And you can't add features to browsers if there isn't a formal specification, right? Right?
- josteink 4y ago
  > I'm not sure who's to blame here. I would venture to say perhaps AzureAD? When a OS receives an instruction from the router/DHCP server that no proxy should be used on this network, and the OS adheres to that… And that alone just breaks AzureAD, that’s clearly a point where AzureAD needs to be made more resilient.
  - c0nsumer 4y ago
    Hey there, OP here. Azure AD is a cloud service. The bug was in WinHttpAutoProxySvc, but at some point ASUS (et al) must have noticed it and chose to send the blank Option 252 to take advantage of the result, without realizing that the result is poor behavior and reporting it to Microsoft. EDIT: More specifically, it was that a service running on the client was told, in a weird way, that there was no proxy to use. This weird way triggered a bug, so it kept using that setting even after it no longer received that signal (different DHCP on a different network) and a there-is-now-a-proxy setting was available (via DNS). EDIT 2: To be more clear, Azure AD needs access to the internet, uses WinHTTP (because this is Windows' standard HTTP libraries), and when a bug in one part of this stack was exposed, AAD didn't work. There were no problems with AAD here, even though that's where the user saw the error.
- fulafel 4y ago
  "Expires: December 1999". It was in draft in the last millennium but it died. It's also a terrible idea. For example now anyone running a evil DHCP server in a WLAN you joincan get your browser to follow a malicious PAC script which lets them MITM even HTTPS traffic... see eg https://www.pcworld.com/article/415991/disable-wpad-now-or-have-your-accounts-and-private-data-compromised.html https://www.pcworld.com/article/415991/disable-wpad-now-or-h... (This was of course back when Windows users were getting regularly pwned by a windows worm of the week so wasn't anything out of the ordinary)
Krisjohn 4y ago
Website blocked due to trojan Website blocked: nuxx.net Malwarebytes Browser Guard blocked this website because it may contain malware activity. We strongly recommend you do not continue. Nothing is ever easy.
- pseudo0 4y ago
  It's 100% clean on VirusTotal. I'm not sure what Malwarebytes has been doing lately, but I had to remove it from a relative's computer after it kept throwing false positives. https://www.virustotal.com/gui/url/54e060dd220272974a9a5bffa4aac87592721ef91f7c50facc23d2a492f03a4a/summary https://www.virustotal.com/gui/url/54e060dd220272974a9a5bffa...
  - c0nsumer 4y ago
    I think they are using some old list, or keep around positives even after they get removed from elsewhere. I'm the OP and nuxx.net was getting flagged by Yandex because I had an old perl script, I believe udpscanner.pl, in a directory as I referenced it in some old writeup. It was actually named udpscannerl.pl.txt, was served up as text/plain, on a server with no perl CGI, and was something that needed to be run interactively from an interpreter. Literally, sample code. Yet for some reason Yandex flagged it as a malicious site. And Malwarebytes picked that up... And apparently continues to do so years after I removed the file and got Yandex to rescan and mark the site as clean.
- c0nsumer 4y ago
  I've been fighting with this for years, and I'm pretty sure Yandex is the problem. Hell, I even signed up for Yandex's webmaster tools as part of trying to fix this. Years ago Yandex was flagging on some sample perl code that had a .txt extension (some something like udpscanner.pl.txt or so, IIRC) that I had sitting in a directory. There's no perl CGI, no way for it to execute on the server; just sample code. IIRC it was even served up as text/plain for easy reading in browsers. It was something that would be run directly on an OS, to do some fast scanning of open ports. For an end user to run it they would have to download it and get it executed by their perl interpreter. Definitely NOT an exploit in a browser. As I recall it was something super basic that I found on a compromised server years ago and referenced in an old writeup. For some reason it was flagged by Yandex as a browser exploit, they reported up to other places, and Malwarebytes flagged the whole site as malicious. Since fighting the technical reasons why their scan is flawed is Sisyphus, I ended up just removing it from the site and getting Yandex to rescan. They now list the site as clean, but some old tools still say something untoward is going on. It's frustrating.
PreInternet01 4y ago
Ah, yes, my old nemesis, WinHttpAutoProxySvc... For years, on both Windows 10 and 11, this has had the habit of randomly spiking the CPU core the service is running on to 100%, in some kind of busy-loop that's effectively preventing anything that uses the Win32 HTTP API from working. So, if symptoms include laptop fan thinks it's a jet engine, Start menu refusing to fully populate, Search not responding, and a lot of apps just not launching at all (or taking several minutes to do so), a quick look at the Services tab in Task Manager for WinHttpAutoProxySvc, followed by go to details and End task on the corresponding svchost.exe might just do the trick. You can ignore the big scary warning about this restarting the system: that's a lie. For a slightly more permanent fix, paste the following into a .reg file and merge it into your Registry: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinHttpAutoProxySvc] "Start"=dword:00000004 This will disable the service (which Microsoft has made impossible to do via regular GUI or CLI tooling), and after a reboot, you should be able to, like, use your PC for a while. Keep an eye open for rogue Windows Updates, though, as Microsoft really, really wants to re-enable this service using those. (Apparently, WinHttpAutoProxySvc does all kinds of important stuff, including address assignment for non-native IPv6 setups, none of which I care about, but before blindly following 'just disable this thing' advice from the Internet, just think for a while before rolling it out to your entire fleet). Playing whack-a-mole with WinHttpAutoProxySvc has been oddly satisfying so far: one of these days I might actually grab a debugger to see what's going on here (because, yes, also after updating to Windows 11 22H2, which re-enabled the service, I had the same-old symptoms within days -- I admire the writer of this article for getting a fix for their problem that worked!)
- danbolt 4y ago
  I sometimes wonder if someone will someday tweet just what were they thinking?!? with a Ghidra screenshot of some quick feature I wrote before a deadline.
- leeoniya 4y ago
  > Ah, yes, my old nemesis Windows. i remember when registry tweaks wouldnt randomly reset and random bloatware would not appear/reappear with each OS update. those were the "good" old days of XP/2000. anyways, i finally got fed up with Windows 10 Updates + Defender shenanigans and perma-switched to Linux (EndeavourOS/KDE). Oh, the joy of your machine only ever doing what you told it, and blazing fast filesystem access with low resource usage! Unthinkable!
chx 4y ago
This has been bothering me quite a bit, it's ... You know companies will, as regular practice just give you a laptop to work on. Some will even provide a phone or tablet. These are not cheap. Not cheap at all. I've yet to hear of a company which would fleet-buy routers reflash with OpenWRT and hand them out. Why?
- fragmede 4y ago
  What would the router do in this case? On the other hand, companies will loan out commercial hotspot products (aka what you get from the eg Verizon store so employees can get Internet access at home.
  - c0nsumer 4y ago
    OP here. And anecdotes, but almost everyone at this company as a Verizon mobile that has data enabled for tethering as needed, but then some people would... ...use it for Netflix and actually hit the "unlimited" data caps and get throttled. ...name their mobile network the same as their home network with the same password. ...place the phone somewhere weird (in the basement where they work, up against an earth-backed concrete/rebar wall, and complain that it's "too slow". There's almost no end to weird things people can do with tech and then blame just the tech. There are constant new problems that need to be handled on a case by case basis, because the root causes are not solvable solely by technical means.
    - fragmede 4y ago
      The ideal case would seem to be ship employees a laptop with LTE built in
  - jon-wood 4y ago
    I used to work for a company that does development of things like TV apps, and physical hardware. They’d distribute wireless APs to engineers which automatically connected to the corporate VPN and exposed that via the AP for devices which couldn’t be directly configured to do so but needed access to development servers only accessible internally.
withinboredom 4y ago
I’ve come to realize the Windows networking stack is bonkers: https://github.com/microsoft/WSL/issues/9354 https://github.com/microsoft/WSL/issues/9354 If I could get every game running on Linux, I’d never boot into it.
- bitcharmer 4y ago
  When was the last time you tried to run a windows game on linux via steam with proton. I am yet to find a title that doesn't work. This was discussed many times on HN.
  - bravetraveler 4y ago
    Not even who you're asking, but I'll preface this by saying... I'm a big fan of Proton. With that said, non-working games are fairly common. We should be clear about this; either they don't work at all, or don't work well. Personally, the biggest 'not working at all' game would be Destiny 2. I bring back my VM with a passed-through GPU occasionally for it At risk of arguing their merits, certain anticheats just simply aren't compatible I don't have many in the 'don't work well' category... but my PC is ridiculously overbuilt. There are certainly inefficiencies Whenever that happens it's often a Wayland problem and not a Proton problem
    - P5fRxh5kUvp2th 4y ago
      Agreed. There are only two windows machines in my house. 1. work laptop 2. gaming machine I tried converting the gaming machine over to Linux, but it's just not there yet. It's making huge strides and I'm super happy about it, but claiming that most or all games run on Linux is waaaaay overstating the reality.
      - bravetraveler 4y ago
        Thank you for the confirmation and support! I love where Linux gaming is going, but we really should be honest about it. The easiest way to leave a bad impression on newcomers is... telling them one thing and having them see the exact opposite. I recommend giving Linux a try, it's way more capable than it used to be. With that said, pack a parachute - you probably will find yourself needing to switch at times.
gumboza 4y ago
This doesn't surprise me. One thing I know about Windows is you can't rely on it when you need it. So in the middle of an outage yesterday, the Windows 10 start menu stopped working. You can open it but when you click on stuff nothing happens. Reboot doesn't fix it. Fortunately I had cmd pinned to the taskbar so I am literally starting programs from that.
- c0nsumer 4y ago
  FYI, this is frequently caused by an Explorer shell extension. I'd look at if you have any of those installed. Unfortunately a lot of random utilities will add them, even old stuff that might not work right anymore. Any why? Because lots of app developers seem to think that you'll want to quickly open anything with their app via a right-click popout...
fomine3 4y ago
Non-transparent proxy is the root of pain
patientplatypus 4y ago
[dead]
josteink 4y ago
> Beyond DNS there is also a Dynamic Host Configuration Protocol (DHCP) method where, along with the typical network address settings, the client receives the URL for downloading the PAC file. This is done via option 252, but isn’t widely supported, it’s normally not used, and we don’t use it it either. Consider me the odd one out but I’ve actually heard about (and used) DHCP option 252, but never once heard about WPAD. To me, this statement seems a bit presumptuous.
- justsomehnguy 4y ago
  Yes, you are the odd one. Or never administered enterprise network with a lot of Windows machines and services. Especially with ISA Server. Though for some time there is a push to just disable WPAD discovery, because it can be used for redirecting the traffic for nefarious purposes and most people don't use a classic web-proxies anyway.
  - c0nsumer 4y ago
    OP here. That's where WPAD conversations get interesting... I imagine that for a whole bunch of folks something like default-path-to-internet is sufficient, or having everything hit the proxy and then hairpin some things back internally. But when you start getting into really big private environments there are all sorts of edge cases that Proxy Auto-Config (PAC) files really help with. For those who aren't aware, PAC files are a single JavaScript function that implement tests on the requested URL (and a few other things) and return either DIRECT or PROXY for where that request should go. And this all runs on the client. It's like a super-robust exception list that's centrally hosted and the OS handles caching it, checking for changes, etc. Changes to it can be made centrally and the clients will generally use the new one within a maximum of 20 minutes. Trying to manage a proxy exception list on each client is... hard. Think situations like having numerous proxy servers all throughout the world, users that move around the world, and you don't want to change the client config to access different proxy servers. Or some sites that needs to go through some proxy servers, some through others (eg: for compliance or technical reasons), some direct even though they have a public-ish URL, or not wanting to bounce high-traffic internal sites off the proxies at all. It's basically client-side app layer selective routing to proxies and it's great. WPAD is just a way to automatically discover the PAC file. While it can be hijacked on a malicious network to point devices through nefarious proxies, if the company stuff is set up right, there's nothing any of those URLs could grab. Or better, tack on some always-on VPN or whatnot when not on the trusted network and all the better. IMO the DHCP way of doing it is kinda cool, but DNS is so much simplier and that's why it's widely supported. DNS is just DNS... Anything that can query DNS can implement WPAD logic, which is why it's supported on pretty much every browser platform and macOS and Windows and such. DHCP requires the dhcpcd to get the setting, somehow pass it to each app that needs it, etc. It requires a lot more integration. Windows does it, but only for things that use the Windows HTTP libraries (WinHTTP). macOS could, but I understand why they don't bother... Linux... Could... But proxy support on most Linuxes sucks. HTTP_PROXY/HTTPS_PROXY is awful if authenticating proxies are used (plaintext creds in an environment variable?!?), basically no PAC file support at all, no transparent Kerberos auth... Blah.
ytygg775 4y ago
Clickbaity title. It makes it sound like remote work for everybody everywhere almost broke because of this-or-that thing involving Microsoft and Asus. Not the case. The article is about some obscure issue for a particular company who trusted their IT to be handled by Active Directory in a Microsoft Azure cloud environment, involving Asus home routers. Hardly a general insight. The root cause for this is the inability of our industry to properly define standards and then enforcing/sticking to them. Everybody just hacks something that kinda-sorta works and whoever has the larger market share is right and everybody else has to suffer, even if they themselves want to do the right thing and do it properly. Let's face it, we all suck at this and have to pay for it with this kind of meaningless waste of time "troubleshooting".
- stingraycharles 4y ago
  I agree that the title is clickbaity -- I expected something much worse and more intentional than some obscure integration bug between Asus routers' DHCP server and proxy-auto-discovery. Why our industry typically fails to agree to formal standards is that it takes time and effort to agree upon standards. The W3C is a good example: browser vendors move faster than the W3C could keep up with, so they decided to bypass them. I don't fully share your negativism towards this: the ability to innovate quickly is important, and even when things get fully standardized, there is no guarantee that every vendor implements them correctly (again, see HTML and just how different things can behave with different browser vendors that all use the same standard).
  - xnickb 4y ago
    > the ability to innovate quickly is important That's a superstition.
    - ytygg775 4y ago
      No, it's a fact of life. If you don't, then somebody else will and you'll go under. If you first spend 5 years crafting the perfect invitation to a date for your crush, then meanwhile another guy will have not just asked her out, but by that time they will have rings on their fingers, a house and two kids.
irusensei 4y ago
After a fresh install of Windows and allowing the updates finish I was surprised by MyAsus nagging prompts asking for login. It never asked me if I want that absolute ad riddled crapware disguised as a support tool.
cesarb 4y ago
> Retrospectively I suspect confusion around what it means to shut down or restart a computer led to many of the reports of reboots/shutdowns/driving into the office fixing the problem or not [...] I read somewhere (I think it was a post on /r/sysadmin) that, on modern Windows, "shut down" actually means hibernate, so old-school people who learned that it's always better to power down and power back up instead of just rebooting (since it also restarts the hardware, not only the software) are led astray: on modern Windows, it's the opposite, "restart" is the option which does a clean start-up, while "shut down" just fakes it. Edit: it was this post and its comments: https://old.reddit.com/r/sysadmin/comments/qiqigu/when_i_ask_did_you_turn_it_off_and_on_again_and/ https://old.reddit.com/r/sysadmin/comments/qiqigu/when_i_ask...
- WirelessGigabit 4y ago
  Shutdown still shuts down the user session, but not the kernel. Shift+Shutdown to get an actual shutdown, or disable Fast Startup in powercfg.cpl. And that Fast Startup is not to be confused with Fast Boot, which skips the POST.
jrochkind1 4y ago
What occurs to me is how consumers have this home internet equipment -- of varying quality, although here it may be "lack of spec" rather than "out of spec" that's a problem -- without the capacity to troubleshoot it if it goes wrong. What if the problem had been triggered by specific routers, but did not bring down the whole server until reboot, but just caused a denial of service to the users with those routers? It would perhaps never have been figured out at all, certain users would just find themselves unable to connect, and have no idea what to do to fix it, with really no feasible way to find out. Router, modem, broadband provider, workstation, OS upgrade, who knows? With most Enterprise IT just saying "I don't know, works for everyone else, must be something wrong on your end, good luck with that." I wonder how often this happens.
- c0nsumer 4y ago
  > With most Enterprise IT just saying "I don't know, works for everyone else, must be something wrong on your end, good luck with that." This was something new and interesting with COVID WFH. Whereas previously WFH was an optional thing, a bit of a perk, and usually not long-term, we could fault someone's home internet and let them contact the ISP/whatever to figure it out, 2020 changed things. Since we were telling people they needed to work from home, and if they couldn't they'd need to contact HR, it really was best if we helped them out a bit more. Not full-on support, but talking through the problem and making a recommendation of how to make things better basically became the norm.

122 points by c0nsumer 4y ago

Wow, what a nightmare to troubleshoot and try to figure out. I feel lucky that a month before the March 2020 I switched to a new job doing more direct application support instead of general desktop/AD related work or I may have had to deal with that exact issue.

c0nsumer 4y ago
OP here. It was... I was really fortunate that some other company had figured it out in parallel with MS, because I was hitting a dead end. I got to the point where I was pretty sure that WPAD wasn't working right, and was in the midst of capturing all sorts of data and escalating that, when a side conversation prompted someone to ask "was it you I worked with on X?". Then it all came together. I think this is one of those things that ends up being so esoteric most folks haven't worked on it, then the chain of events leading to it... But the moment I heard the trigger I was able to repro it... That was quite a relief.

josephcsible 4y ago

Why do so many companies require proxy configurations to be set on the endpoint, rather than using transparent proxying? Wouldn't that completely avoid a whole class of issues?

9dev 4y ago
SSL certificates would be my first guess.
- josephcsible 4y ago
  Don't browsers use CONNECT to access HTTPS sites over a proxy? Isn't the certificate situation exactly the same inside of that vs. with transparent proxying?
tryauuum 4y ago
hm... Could it be related to malware? If there's a malicious device in network, it would need a proxy configuration. With a transparent proxy back-connecting to attacker would be easier On the other hand, attacker could probably always send messages out via DNS tunnel Please someone more knowledgeable do enlighten us
- c0nsumer 4y ago
  Malware can easily just pick up and use the system proxy settings. Hell, on Windows it just has to use WinHTTP and it'll use the system's paths to the internet.
  - tryauuum 4y ago
    Yeah, but if you don't set system proxies and instead you set it through environmental variables only for apps that need it?
c0nsumer 4y ago
OP here. This isn't quite my space, but from what I recall there's a couple wrinkles with transparent proxying. It's overall a good idea, but has some edge cases: - Authentication can go sideways in weird ways. A 407 from what looks like the correct site can cause odd things. IIRC there isn't great support from vendors for authenticating transparent proxies, too. Sure, you could auth off the machine instead (X device is on the network, therefore it's allowed), but what about shared machines... Proxy auth as user is better, because it allows requests to be tracked to a user ID and login session, not just a device. - Client getting load balanced between proxies during a session can trigger reauth (because auth sessions not shared between proxies), this can confuse a client. - HTTPS sites can get weird. - Routing in very large private environments can complicate default routing to the public internet. Although this can be handled by doing transparent auth + optional manual config.

Randor 4y ago

As the blog correctly identifies, the WPAD specification is still in draft: https://datatracker.ietf.org/doc/html/draft-ietf-wrec-wpad-01 https://datatracker.ietf.org/doc/html/draft-ietf-wrec-wpad-0... In other words there really isn't a standard to follow. You can't add features to operating systems if there isn't a formal specification. I'm not sure who's to blame here.

justsomehnguy 4y ago
> You can't add features And you can't add features to browsers if there isn't a formal specification, right? Right?
josteink 4y ago
> I'm not sure who's to blame here. I would venture to say perhaps AzureAD? When a OS receives an instruction from the router/DHCP server that no proxy should be used on this network, and the OS adheres to that… And that alone just breaks AzureAD, that’s clearly a point where AzureAD needs to be made more resilient.
- c0nsumer 4y ago
  Hey there, OP here. Azure AD is a cloud service. The bug was in WinHttpAutoProxySvc, but at some point ASUS (et al) must have noticed it and chose to send the blank Option 252 to take advantage of the result, without realizing that the result is poor behavior and reporting it to Microsoft. EDIT: More specifically, it was that a service running on the client was told, in a weird way, that there was no proxy to use. This weird way triggered a bug, so it kept using that setting even after it no longer received that signal (different DHCP on a different network) and a there-is-now-a-proxy setting was available (via DNS). EDIT 2: To be more clear, Azure AD needs access to the internet, uses WinHTTP (because this is Windows' standard HTTP libraries), and when a bug in one part of this stack was exposed, AAD didn't work. There were no problems with AAD here, even though that's where the user saw the error.
fulafel 4y ago
"Expires: December 1999". It was in draft in the last millennium but it died. It's also a terrible idea. For example now anyone running a evil DHCP server in a WLAN you joincan get your browser to follow a malicious PAC script which lets them MITM even HTTPS traffic... see eg https://www.pcworld.com/article/415991/disable-wpad-now-or-have-your-accounts-and-private-data-compromised.html https://www.pcworld.com/article/415991/disable-wpad-now-or-h... (This was of course back when Windows users were getting regularly pwned by a windows worm of the week so wasn't anything out of the ordinary)

Krisjohn 4y ago

Website blocked due to trojan Website blocked: nuxx.net Malwarebytes Browser Guard blocked this website because it may contain malware activity. We strongly recommend you do not continue. Nothing is ever easy.

pseudo0 4y ago
It's 100% clean on VirusTotal. I'm not sure what Malwarebytes has been doing lately, but I had to remove it from a relative's computer after it kept throwing false positives. https://www.virustotal.com/gui/url/54e060dd220272974a9a5bffa4aac87592721ef91f7c50facc23d2a492f03a4a/summary https://www.virustotal.com/gui/url/54e060dd220272974a9a5bffa...
- c0nsumer 4y ago
  I think they are using some old list, or keep around positives even after they get removed from elsewhere. I'm the OP and nuxx.net was getting flagged by Yandex because I had an old perl script, I believe udpscanner.pl, in a directory as I referenced it in some old writeup. It was actually named udpscannerl.pl.txt, was served up as text/plain, on a server with no perl CGI, and was something that needed to be run interactively from an interpreter. Literally, sample code. Yet for some reason Yandex flagged it as a malicious site. And Malwarebytes picked that up... And apparently continues to do so years after I removed the file and got Yandex to rescan and mark the site as clean.
c0nsumer 4y ago
I've been fighting with this for years, and I'm pretty sure Yandex is the problem. Hell, I even signed up for Yandex's webmaster tools as part of trying to fix this. Years ago Yandex was flagging on some sample perl code that had a .txt extension (some something like udpscanner.pl.txt or so, IIRC) that I had sitting in a directory. There's no perl CGI, no way for it to execute on the server; just sample code. IIRC it was even served up as text/plain for easy reading in browsers. It was something that would be run directly on an OS, to do some fast scanning of open ports. For an end user to run it they would have to download it and get it executed by their perl interpreter. Definitely NOT an exploit in a browser. As I recall it was something super basic that I found on a compromised server years ago and referenced in an old writeup. For some reason it was flagged by Yandex as a browser exploit, they reported up to other places, and Malwarebytes flagged the whole site as malicious. Since fighting the technical reasons why their scan is flawed is Sisyphus, I ended up just removing it from the site and getting Yandex to rescan. They now list the site as clean, but some old tools still say something untoward is going on. It's frustrating.

PreInternet01 4y ago

Ah, yes, my old nemesis, WinHttpAutoProxySvc... For years, on both Windows 10 and 11, this has had the habit of randomly spiking the CPU core the service is running on to 100%, in some kind of busy-loop that's effectively preventing anything that uses the Win32 HTTP API from working. So, if symptoms include laptop fan thinks it's a jet engine, Start menu refusing to fully populate, Search not responding, and a lot of apps just not launching at all (or taking several minutes to do so), a quick look at the Services tab in Task Manager for WinHttpAutoProxySvc, followed by go to details and End task on the corresponding svchost.exe might just do the trick. You can ignore the big scary warning about this restarting the system: that's a lie. For a slightly more permanent fix, paste the following into a .reg file and merge it into your Registry: Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinHttpAutoProxySvc] "Start"=dword:00000004 This will disable the service (which Microsoft has made impossible to do via regular GUI or CLI tooling), and after a reboot, you should be able to, like, use your PC for a while. Keep an eye open for rogue Windows Updates, though, as Microsoft really, really wants to re-enable this service using those. (Apparently, WinHttpAutoProxySvc does all kinds of important stuff, including address assignment for non-native IPv6 setups, none of which I care about, but before blindly following 'just disable this thing' advice from the Internet, just think for a while before rolling it out to your entire fleet). Playing whack-a-mole with WinHttpAutoProxySvc has been oddly satisfying so far: one of these days I might actually grab a debugger to see what's going on here (because, yes, also after updating to Windows 11 22H2, which re-enabled the service, I had the same-old symptoms within days -- I admire the writer of this article for getting a fix for their problem that worked!)

danbolt 4y ago
I sometimes wonder if someone will someday tweet just what were they thinking?!? with a Ghidra screenshot of some quick feature I wrote before a deadline.
leeoniya 4y ago
> Ah, yes, my old nemesis Windows. i remember when registry tweaks wouldnt randomly reset and random bloatware would not appear/reappear with each OS update. those were the "good" old days of XP/2000. anyways, i finally got fed up with Windows 10 Updates + Defender shenanigans and perma-switched to Linux (EndeavourOS/KDE). Oh, the joy of your machine only ever doing what you told it, and blazing fast filesystem access with low resource usage! Unthinkable!

chx 4y ago

This has been bothering me quite a bit, it's ... You know companies will, as regular practice just give you a laptop to work on. Some will even provide a phone or tablet. These are not cheap. Not cheap at all. I've yet to hear of a company which would fleet-buy routers reflash with OpenWRT and hand them out. Why?

fragmede 4y ago
What would the router do in this case? On the other hand, companies will loan out commercial hotspot products (aka what you get from the eg Verizon store so employees can get Internet access at home.
- c0nsumer 4y ago
  OP here. And anecdotes, but almost everyone at this company as a Verizon mobile that has data enabled for tethering as needed, but then some people would... ...use it for Netflix and actually hit the "unlimited" data caps and get throttled. ...name their mobile network the same as their home network with the same password. ...place the phone somewhere weird (in the basement where they work, up against an earth-backed concrete/rebar wall, and complain that it's "too slow". There's almost no end to weird things people can do with tech and then blame just the tech. There are constant new problems that need to be handled on a case by case basis, because the root causes are not solvable solely by technical means.
  - fragmede 4y ago
    The ideal case would seem to be ship employees a laptop with LTE built in
- jon-wood 4y ago
  I used to work for a company that does development of things like TV apps, and physical hardware. They’d distribute wireless APs to engineers which automatically connected to the corporate VPN and exposed that via the AP for devices which couldn’t be directly configured to do so but needed access to development servers only accessible internally.

withinboredom 4y ago

I’ve come to realize the Windows networking stack is bonkers: https://github.com/microsoft/WSL/issues/9354 https://github.com/microsoft/WSL/issues/9354 If I could get every game running on Linux, I’d never boot into it.

bitcharmer 4y ago
When was the last time you tried to run a windows game on linux via steam with proton. I am yet to find a title that doesn't work. This was discussed many times on HN.
- bravetraveler 4y ago
  Not even who you're asking, but I'll preface this by saying... I'm a big fan of Proton. With that said, non-working games are fairly common. We should be clear about this; either they don't work at all, or don't work well. Personally, the biggest 'not working at all' game would be Destiny 2. I bring back my VM with a passed-through GPU occasionally for it At risk of arguing their merits, certain anticheats just simply aren't compatible I don't have many in the 'don't work well' category... but my PC is ridiculously overbuilt. There are certainly inefficiencies Whenever that happens it's often a Wayland problem and not a Proton problem
  - P5fRxh5kUvp2th 4y ago
    Agreed. There are only two windows machines in my house. 1. work laptop 2. gaming machine I tried converting the gaming machine over to Linux, but it's just not there yet. It's making huge strides and I'm super happy about it, but claiming that most or all games run on Linux is waaaaay overstating the reality.
    - bravetraveler 4y ago
      Thank you for the confirmation and support! I love where Linux gaming is going, but we really should be honest about it. The easiest way to leave a bad impression on newcomers is... telling them one thing and having them see the exact opposite. I recommend giving Linux a try, it's way more capable than it used to be. With that said, pack a parachute - you probably will find yourself needing to switch at times.

gumboza 4y ago

This doesn't surprise me. One thing I know about Windows is you can't rely on it when you need it. So in the middle of an outage yesterday, the Windows 10 start menu stopped working. You can open it but when you click on stuff nothing happens. Reboot doesn't fix it. Fortunately I had cmd pinned to the taskbar so I am literally starting programs from that.

c0nsumer 4y ago
FYI, this is frequently caused by an Explorer shell extension. I'd look at if you have any of those installed. Unfortunately a lot of random utilities will add them, even old stuff that might not work right anymore. Any why? Because lots of app developers seem to think that you'll want to quickly open anything with their app via a right-click popout...

fomine3 4y ago

Non-transparent proxy is the root of pain

patientplatypus 4y ago

[dead]

josteink 4y ago

> Beyond DNS there is also a Dynamic Host Configuration Protocol (DHCP) method where, along with the typical network address settings, the client receives the URL for downloading the PAC file. This is done via option 252, but isn’t widely supported, it’s normally not used, and we don’t use it it either. Consider me the odd one out but I’ve actually heard about (and used) DHCP option 252, but never once heard about WPAD. To me, this statement seems a bit presumptuous.

justsomehnguy 4y ago
Yes, you are the odd one. Or never administered enterprise network with a lot of Windows machines and services. Especially with ISA Server. Though for some time there is a push to just disable WPAD discovery, because it can be used for redirecting the traffic for nefarious purposes and most people don't use a classic web-proxies anyway.
- c0nsumer 4y ago
  OP here. That's where WPAD conversations get interesting... I imagine that for a whole bunch of folks something like default-path-to-internet is sufficient, or having everything hit the proxy and then hairpin some things back internally. But when you start getting into really big private environments there are all sorts of edge cases that Proxy Auto-Config (PAC) files really help with. For those who aren't aware, PAC files are a single JavaScript function that implement tests on the requested URL (and a few other things) and return either DIRECT or PROXY for where that request should go. And this all runs on the client. It's like a super-robust exception list that's centrally hosted and the OS handles caching it, checking for changes, etc. Changes to it can be made centrally and the clients will generally use the new one within a maximum of 20 minutes. Trying to manage a proxy exception list on each client is... hard. Think situations like having numerous proxy servers all throughout the world, users that move around the world, and you don't want to change the client config to access different proxy servers. Or some sites that needs to go through some proxy servers, some through others (eg: for compliance or technical reasons), some direct even though they have a public-ish URL, or not wanting to bounce high-traffic internal sites off the proxies at all. It's basically client-side app layer selective routing to proxies and it's great. WPAD is just a way to automatically discover the PAC file. While it can be hijacked on a malicious network to point devices through nefarious proxies, if the company stuff is set up right, there's nothing any of those URLs could grab. Or better, tack on some always-on VPN or whatnot when not on the trusted network and all the better. IMO the DHCP way of doing it is kinda cool, but DNS is so much simplier and that's why it's widely supported. DNS is just DNS... Anything that can query DNS can implement WPAD logic, which is why it's supported on pretty much every browser platform and macOS and Windows and such. DHCP requires the dhcpcd to get the setting, somehow pass it to each app that needs it, etc. It requires a lot more integration. Windows does it, but only for things that use the Windows HTTP libraries (WinHTTP). macOS could, but I understand why they don't bother... Linux... Could... But proxy support on most Linuxes sucks. HTTP_PROXY/HTTPS_PROXY is awful if authenticating proxies are used (plaintext creds in an environment variable?!?), basically no PAC file support at all, no transparent Kerberos auth... Blah.

ytygg775 4y ago

Clickbaity title. It makes it sound like remote work for everybody everywhere almost broke because of this-or-that thing involving Microsoft and Asus. Not the case. The article is about some obscure issue for a particular company who trusted their IT to be handled by Active Directory in a Microsoft Azure cloud environment, involving Asus home routers. Hardly a general insight. The root cause for this is the inability of our industry to properly define standards and then enforcing/sticking to them. Everybody just hacks something that kinda-sorta works and whoever has the larger market share is right and everybody else has to suffer, even if they themselves want to do the right thing and do it properly. Let's face it, we all suck at this and have to pay for it with this kind of meaningless waste of time "troubleshooting".

stingraycharles 4y ago
I agree that the title is clickbaity -- I expected something much worse and more intentional than some obscure integration bug between Asus routers' DHCP server and proxy-auto-discovery. Why our industry typically fails to agree to formal standards is that it takes time and effort to agree upon standards. The W3C is a good example: browser vendors move faster than the W3C could keep up with, so they decided to bypass them. I don't fully share your negativism towards this: the ability to innovate quickly is important, and even when things get fully standardized, there is no guarantee that every vendor implements them correctly (again, see HTML and just how different things can behave with different browser vendors that all use the same standard).
- xnickb 4y ago
  > the ability to innovate quickly is important That's a superstition.
  - ytygg775 4y ago
    No, it's a fact of life. If you don't, then somebody else will and you'll go under. If you first spend 5 years crafting the perfect invitation to a date for your crush, then meanwhile another guy will have not just asked her out, but by that time they will have rings on their fingers, a house and two kids.

irusensei 4y ago

After a fresh install of Windows and allowing the updates finish I was surprised by MyAsus nagging prompts asking for login. It never asked me if I want that absolute ad riddled crapware disguised as a support tool.

cesarb 4y ago

> Retrospectively I suspect confusion around what it means to shut down or restart a computer led to many of the reports of reboots/shutdowns/driving into the office fixing the problem or not [...] I read somewhere (I think it was a post on /r/sysadmin) that, on modern Windows, "shut down" actually means hibernate, so old-school people who learned that it's always better to power down and power back up instead of just rebooting (since it also restarts the hardware, not only the software) are led astray: on modern Windows, it's the opposite, "restart" is the option which does a clean start-up, while "shut down" just fakes it. Edit: it was this post and its comments: https://old.reddit.com/r/sysadmin/comments/qiqigu/when_i_ask_did_you_turn_it_off_and_on_again_and/ https://old.reddit.com/r/sysadmin/comments/qiqigu/when_i_ask...

WirelessGigabit 4y ago
Shutdown still shuts down the user session, but not the kernel. Shift+Shutdown to get an actual shutdown, or disable Fast Startup in powercfg.cpl. And that Fast Startup is not to be confused with Fast Boot, which skips the POST.

jrochkind1 4y ago

What occurs to me is how consumers have this home internet equipment -- of varying quality, although here it may be "lack of spec" rather than "out of spec" that's a problem -- without the capacity to troubleshoot it if it goes wrong. What if the problem had been triggered by specific routers, but did not bring down the whole server until reboot, but just caused a denial of service to the users with those routers? It would perhaps never have been figured out at all, certain users would just find themselves unable to connect, and have no idea what to do to fix it, with really no feasible way to find out. Router, modem, broadband provider, workstation, OS upgrade, who knows? With most Enterprise IT just saying "I don't know, works for everyone else, must be something wrong on your end, good luck with that." I wonder how often this happens.

c0nsumer 4y ago
> With most Enterprise IT just saying "I don't know, works for everyone else, must be something wrong on your end, good luck with that." This was something new and interesting with COVID WFH. Whereas previously WFH was an optional thing, a bit of a perk, and usually not long-term, we could fault someone's home internet and let them contact the ISP/whatever to figure it out, 2020 changed things. Since we were telling people they needed to work from home, and if they couldn't they'd need to contact HR, it really was best if we helped them out a bit more. Not full-on support, but talking through the problem and making a recommendation of how to make things better basically became the norm.