3 ms·

My team (Microsoft Band) discovered the reason why the surface's keyboard sometimes wouldn't work when connected. There was a hardware bug in the cortex MCU the

by com2kid 4mo ago

My team (Microsoft Band) discovered the reason why the surface's keyboard sometimes wouldn't work when connected. There was a hardware bug in the cortex MCU the keyboard used involving waking from deep sleep. One of our FW engineers spent several months figuring it out and eventually reported it to the manufacturer, and to the Surface team. IIRC it was something about wake on interrupt in a specific deep sleep mode and also something around timing. It was a rather nasty bug. Firmware is full of nightmare scenarios like that.

z3ratul163071 4mo ago
[flagged]
- wilted-iris 4mo ago
  Respectfully, no, not even a little bit. That’s normal everyday microcontrollers for you.
  - exmadscientist 4mo ago
    Yeah, wake from deep sleep is always one of the weirder bits. It's just hard to do.
    - deleted 4mo ago
      [deleted]
- vitally3643 4mo ago
  You clearly have never tried to implement sleep on a microcontroller. It has nothing at all to do with ACPI. And this kind of eldritch bug is par for the course. It has nothing at all to do with Microsoft or PCs at all. Microcontroller sleep just sucks in a lot of incredibly weird ways.
  - com2kid 4mo ago
    On Band we had one of the top firmware engineers in the world who got our power situation setup and figured out that bug. We had another bug involving a single cycle timing tolerance of memory on a bus, we were one cycle off in our config. We dual sourced memory chips and one of our suppliers had chips that worked with 1 fewer wait states, the other chip didn't. Once we started blending the 2nd chip into our production line we had weird issues. One of my guys spent ~2 months thinking it was a software bug. We had a null ptr deref in our code, he went line by line and by the end of 2 months we had literally bug free calendar code. Turns out the timing bug manifested in the exact same bit being corrupted every now and then in memory, which happened to be where our pointer lived at. We figured it out when a new production build caused memory to be moved around enough that someone else's pointer was getting corrupted. :) So then the HW investigation began. The principal engineer on Band's firmware refused to allow us to have any open crash reports, all crash dumps had to be fully investigated, root caused, and fixed. In a few cases we found customers who had faulty hardware and we reached out to individuals to replace their units, and brought the broken units back to our labs to figure out what went wrong on the assembly line. (Although on occasion stuff just gets assembled wrong, even with slews of automated tests on every unit that leaves the factory floor!) That same principal engineer reviewed every single line of code that went into the Band's firmware. That code base had a consistency of code, style, and thoughtfulness that I'll likely never see again in my career.
  - deleted 4mo ago
    [deleted]
  - z3ratul163071 4mo ago
    did not, you are right. however, i don't think it should be an issue: - get trigger for sleep - save state to nvram - ack sleep - wake up the same way as normal power-up, state being already in nvram if this simple protocol is not working in your scenario, you have a uc larping as cpu doing way too much.
fmajid 4mo ago
For an amusing take on the life of a systems programmer, James Mickens' "The Night Watch": https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf

by com2kid 4mo ago

z3ratul163071 4mo ago

[flagged]

wilted-iris 4mo ago
Respectfully, no, not even a little bit. That’s normal everyday microcontrollers for you.
- exmadscientist 4mo ago
  Yeah, wake from deep sleep is always one of the weirder bits. It's just hard to do.
  - deleted 4mo ago
    [deleted]
vitally3643 4mo ago
You clearly have never tried to implement sleep on a microcontroller. It has nothing at all to do with ACPI. And this kind of eldritch bug is par for the course. It has nothing at all to do with Microsoft or PCs at all. Microcontroller sleep just sucks in a lot of incredibly weird ways.
- com2kid 4mo ago
  On Band we had one of the top firmware engineers in the world who got our power situation setup and figured out that bug. We had another bug involving a single cycle timing tolerance of memory on a bus, we were one cycle off in our config. We dual sourced memory chips and one of our suppliers had chips that worked with 1 fewer wait states, the other chip didn't. Once we started blending the 2nd chip into our production line we had weird issues. One of my guys spent ~2 months thinking it was a software bug. We had a null ptr deref in our code, he went line by line and by the end of 2 months we had literally bug free calendar code. Turns out the timing bug manifested in the exact same bit being corrupted every now and then in memory, which happened to be where our pointer lived at. We figured it out when a new production build caused memory to be moved around enough that someone else's pointer was getting corrupted. :) So then the HW investigation began. The principal engineer on Band's firmware refused to allow us to have any open crash reports, all crash dumps had to be fully investigated, root caused, and fixed. In a few cases we found customers who had faulty hardware and we reached out to individuals to replace their units, and brought the broken units back to our labs to figure out what went wrong on the assembly line. (Although on occasion stuff just gets assembled wrong, even with slews of automated tests on every unit that leaves the factory floor!) That same principal engineer reviewed every single line of code that went into the Band's firmware. That code base had a consistency of code, style, and thoughtfulness that I'll likely never see again in my career.
- deleted 4mo ago
  [deleted]
- z3ratul163071 4mo ago
  did not, you are right. however, i don't think it should be an issue: - get trigger for sleep - save state to nvram - ack sleep - wake up the same way as normal power-up, state being already in nvram if this simple protocol is not working in your scenario, you have a uc larping as cpu doing way too much.

fmajid 4mo ago

For an amusing take on the life of a systems programmer, James Mickens' "The Night Watch": https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf