4 ms·

This is an amazing discovery, article, and fix proposal. Fantastic work, very impressive and also very instructive on how things work on modern PCs and how far

by unwind 1y ago

This is an amazing discovery, article, and fix proposal. Fantastic work, very impressive and also very instructive on how things work on modern PCs and how far you can actually dig to get at stuff that is "supposed" to be hidden. As someone who has written embedded firmware for many years (not for PCs), I can only dream of an end user being this capable to discover a bug. I want to live in the world where Asus immediately send an e-mail offering some kind of short-term contracting work to fly in and talk to their firmware people for a few days and get $FIVE_FIGURES or something, and leave with an updated laptop running their new production BIOS. Obviously this bug has gone un-fixed for four years so that is not the world we're in. That makes me sad. :| Edit: s/fix/fix proposal/.

_zoltan_ 1y ago
sorry, what fix? the linked github page ends in "here is everything, ASUS, please fix it", right?
- unwind 1y ago
  Yeah, sorry, that was a bit unclear. I just meant that the article went as far as propose rather clearly what is needed to fix the issue ("don't sleep() in an interrupt service routine").
- f311a 1y ago
  He knows how to fix it, and he fixed his system. It's not universal, you need a custom patch for each model and fw version.
matheusmoreira 1y ago
As a user and programmer, I can only dream of being this knowledgeable about things. There's a ton of domain knowledge embedded in the article, it's pretty amazing. I managed to reverse engineer a lot of my laptop's features but hit a wall when it came to this ACPI stuff. I dumped the tables and decompiled the code but all I got was stub code. I wanted to be the guy who wrote the Linux drivers for his own laptop but I just didn't manage it. Massive respect for anyone who can do this.
- trelane 1y ago
  Yeah the best way to go is to buy Linux preinstalled and supported. Though, as with Windows in this case, that still won't save you if the system integration and firmware teams don't do their job.
kryogen1c 1y ago
The technical RCA is fascinating, but im also interested in the business processes RCA. - this sounds ubiquitous and reproducible. How did this not get fed back through tech support/RMA channels? Was there so little evidence that it wasn't correlateable, or did ASUS look and arrive at an incorrect conclusion, eg batch of bad silicon? Could it be that they had plentiful evidence and were negligent or incompetent? - it sounds like this is plainly evident when using the machine. What is the QA process? This should not have been possible to miss? - now that they know, what will they do? Imo, the ceo calculus here is clear. If you're a luxury good with elastic demand, you fix the issue and fix the perception (two separate things). Multi-year, multifaceted issues like this have the potential to ruin a brand. I've bought ROG in the past, and I'm inclined to never do so again. EDIT: on further reflection, the firmware bug itself is pretty troubling. the other bugs i get - hardware assumptions were changed, or good code was reused that didnt know or support the gpu mux, i see how those errors comes about. the method sleeping an interrupt... is awful? how did that get reviewed? what is that firmware test suite?
- formerly_proven 1y ago
  > What is the QA process? This should not have been possible to miss? Have you used consumer goods [or virtually anything] from the last couple decades? By and large, nobody cares. Look at the timeline here; clearly nobody cares.
  - sim7c00 1y ago
    the QA process is shipping to customers and sticking their head in the pile of money they amass with a broken piece of junk.
- nikanj 1y ago
  It doesn’t matter that the consumer/gamer laptop is a piece of shit, because all of the competitors are too. Consumer hardware is a volume business, and the actual end-user experience matters very little compared to endorsement deals and marketing strategies. Every one of the affected ASUS laptops probably got a glowing 5/5 review from the usual suspects, and consumers have little hope of getting a fair deal
ajross 1y ago
Yeah. ACPI's AML bytecode is sort of a mixed blessing. It allows for reverse engineering and end user analasys/fixing of bugs like this. It's also just a terrible disaster of a programming environment, with a very large (terrifyingly so, given the limited capability) interpreter that needs to live at the highest privilege level of the kernel. And it's generally used like a hatchet by system integrators for tricks like this, with pretty much exactly the code quality you'd expect. Almost always the path to writing a Linux driver for some oddball laptop subsystem starts with "throw away the ACPI stuff".
- somat 1y ago
  As far as I know there are three ACPI AML stacks, the reference intel one, linux uses this, miscrosoft has one, and those crazy hackers over at the openbsd project decided to make their own.
  - ajross 1y ago
    I think that's right. Though my understanding is that the Windows code is derived from the original Intel one too and has evolved in tandem with the Intel-maintained driver. And... yeah, acpica (drivers/acpica/acpica) is just huge; I checked again and it's at 2.5MB of source code. All for a DSDT table parser and a virtual machine with about the capability of a 6502.
  - bityard 1y ago
    and off by one errors
    - butlike 1y ago
      and off by one errors
- rangestransform 1y ago
  This is why I prefer having my software vendor write native code to interface with the machine instead of stupid bodges like acpi Windows laptops are dead on arrival for me, all windows laptops are physical shovelware
meandmycode 1y ago
Truly awesome analysis, it's great that Asus spends this effort to quality check their hot garbage.. oh wait..

by unwind 1y ago

_zoltan_ 1y ago

sorry, what fix? the linked github page ends in "here is everything, ASUS, please fix it", right?

unwind 1y ago
Yeah, sorry, that was a bit unclear. I just meant that the article went as far as propose rather clearly what is needed to fix the issue ("don't sleep() in an interrupt service routine").
f311a 1y ago
He knows how to fix it, and he fixed his system. It's not universal, you need a custom patch for each model and fw version.

matheusmoreira 1y ago

As a user and programmer, I can only dream of being this knowledgeable about things. There's a ton of domain knowledge embedded in the article, it's pretty amazing. I managed to reverse engineer a lot of my laptop's features but hit a wall when it came to this ACPI stuff. I dumped the tables and decompiled the code but all I got was stub code. I wanted to be the guy who wrote the Linux drivers for his own laptop but I just didn't manage it. Massive respect for anyone who can do this.

trelane 1y ago
Yeah the best way to go is to buy Linux preinstalled and supported. Though, as with Windows in this case, that still won't save you if the system integration and firmware teams don't do their job.

kryogen1c 1y ago

The technical RCA is fascinating, but im also interested in the business processes RCA. - this sounds ubiquitous and reproducible. How did this not get fed back through tech support/RMA channels? Was there so little evidence that it wasn't correlateable, or did ASUS look and arrive at an incorrect conclusion, eg batch of bad silicon? Could it be that they had plentiful evidence and were negligent or incompetent? - it sounds like this is plainly evident when using the machine. What is the QA process? This should not have been possible to miss? - now that they know, what will they do? Imo, the ceo calculus here is clear. If you're a luxury good with elastic demand, you fix the issue and fix the perception (two separate things). Multi-year, multifaceted issues like this have the potential to ruin a brand. I've bought ROG in the past, and I'm inclined to never do so again. EDIT: on further reflection, the firmware bug itself is pretty troubling. the other bugs i get - hardware assumptions were changed, or good code was reused that didnt know or support the gpu mux, i see how those errors comes about. the method sleeping an interrupt... is awful? how did that get reviewed? what is that firmware test suite?

formerly_proven 1y ago
> What is the QA process? This should not have been possible to miss? Have you used consumer goods [or virtually anything] from the last couple decades? By and large, nobody cares. Look at the timeline here; clearly nobody cares.
- sim7c00 1y ago
  the QA process is shipping to customers and sticking their head in the pile of money they amass with a broken piece of junk.
nikanj 1y ago
It doesn’t matter that the consumer/gamer laptop is a piece of shit, because all of the competitors are too. Consumer hardware is a volume business, and the actual end-user experience matters very little compared to endorsement deals and marketing strategies. Every one of the affected ASUS laptops probably got a glowing 5/5 review from the usual suspects, and consumers have little hope of getting a fair deal

ajross 1y ago

Yeah. ACPI's AML bytecode is sort of a mixed blessing. It allows for reverse engineering and end user analasys/fixing of bugs like this. It's also just a terrible disaster of a programming environment, with a very large (terrifyingly so, given the limited capability) interpreter that needs to live at the highest privilege level of the kernel. And it's generally used like a hatchet by system integrators for tricks like this, with pretty much exactly the code quality you'd expect. Almost always the path to writing a Linux driver for some oddball laptop subsystem starts with "throw away the ACPI stuff".

somat 1y ago
As far as I know there are three ACPI AML stacks, the reference intel one, linux uses this, miscrosoft has one, and those crazy hackers over at the openbsd project decided to make their own.
- ajross 1y ago
  I think that's right. Though my understanding is that the Windows code is derived from the original Intel one too and has evolved in tandem with the Intel-maintained driver. And... yeah, acpica (drivers/acpica/acpica) is just huge; I checked again and it's at 2.5MB of source code. All for a DSDT table parser and a virtual machine with about the capability of a 6502.
- bityard 1y ago
  and off by one errors
  - butlike 1y ago
    and off by one errors
rangestransform 1y ago
This is why I prefer having my software vendor write native code to interface with the machine instead of stupid bodges like acpi Windows laptops are dead on arrival for me, all windows laptops are physical shovelware

meandmycode 1y ago

Truly awesome analysis, it's great that Asus spends this effort to quality check their hot garbage.. oh wait..