5 ms·

I worked on that as a QA Engineer. My favorite accomplishment was discovering that one version of a Crusoe-powered system would irrevocably hang while playing

I worked on that as a QA Engineer. My favorite accomplishment was discovering that one version of a Crusoe-powered system would irrevocably hang while playing Tomb Raider 2 for between 15 and 45 minutes. It took an entire VLSI team over a week to debug that one. The problem had something to do with a voltage drop on a certain chipset. I also enjoyed discovering that HTTPS didn’t work on Internet Explorer 5.5, jump jets recharged at 10% the expected rate in MechWarrior 2, and a certain modem worked on PCI slots 1 and 3 & didn’t work in PCI slots 2 and 4.

plerpin 6y ago
With strange failures like that, I'm surprised the system even booted at all.
- deleted 6y ago
  [deleted]
- FullyFunctional 6y ago
  Pope was a QA engineer - that means he worked on systems that weren't qualified for shipping. Actual CMS bugs in a shipped device was extremely rare, but the big promise of Transmeta was that would _could_ ship fixes (and did, in at least one case). The perf impact of a CMS fix was typically trivial as it was fixing a translator, unlike an interpreter like the microcode in an Intel core.
  - PopeDotNinja 6y ago
    I mostly worked on development boards that were the same size that you'd put in a regular AT/ATX case. I did some work on notebooks, too, both on prototypes & production models, too.
    - FullyFunctional 6y ago
      I think I know you (Hi!), but my point is that by definition, QA's work is to find the bugs before the customers, so they don't see them. Talking about bugs you find without that qualifier might make people get the wrong impression.
      - PopeDotNinja 6y ago
        Hola!
- ethbro 6y ago
  One thing to remember is that this dropped circa-2001. The dominant Windows versions were ME and XP. Both of which featured driver models of... not too much system protection. So applications (especially games) looked a lot more like their console cousins, in that they took advantage of all kinds of weird hacks and did things 1,000 different ways. As opposed to now, when you have more strict intermediary layers, and software sits on top of more defined interfaces. So things like "the MechWarrior developer in charge of jump jets decided to base timing off the PCI bus speed" or god knows what, isn't that surprising.
VectorLock 6y ago
Thats why I was always skeptical of the Transmeta CPUs. They could 'run' the x86 ISA as its on paper, but they'd never be able to completely match all the nuances of the Intel CPUs to get 100% compatibility. The devils in the details as they say.
- saagarjha 6y ago
  Reimplementing an ISA is not actually an impossible feat; many companies have done so successfully before.
  - VectorLock 6y ago
    Nothing is impossible, but has anybody done it since 2000 starting from scratch? AMD obviously has been at it from the start, Cyrix, VIA, NEC all did it but any since Intel CPUs started being in the tens of millions of transistors?
    - weare138 6y ago
      Hitachi SH-4 | 10M transistors | 1997 SPARC64 V | 191M transistors | 2001 DEC Alpha 21364 (EV7) | 152M transistors | 2003 SPARC64 V+ | 400M transistors | 2004 Cell | 250M transistors | 2005 POWER6 | 789M transistors | 2007 SPARC64 VI | 540M transistors | 2007 SPARC64 VIIIfx | 760M transistors | 2009 16-core SPARC T3 | 1B transistors | 2010 8-core POWER7 | 1.2B transistors | 2010 Quad-core z196 | 1.4B transistors | 2010 SPARC64 IXfx | 1.87B transistors | 2011 SPARC64 X | 2.99B transistors | 2012 8-core POWER7+ | 2.1B transistors | 2012 Apple A7 | 1B transistors | 2013 12-core POWER8 | 4.2B transistors | 2013 Apple A8 | 2B transistors | 2014 Apple A8X | 3B transistors | 2014 32-core SPARC M7 | 10B transistors | 2015 [insert a crapton of ARM CPUs here] AWS Graviton2 | 30B transistors | 2019
      - VectorLock 6y ago
        No x86 ISAs in here. All licensed designs or first party manufacturer. I wasn't saying nobody could make anything as complex as current Intel CPUs. Cloning other ISAs either from clean room or from licensed designs like ARM seems to be much easier than matching Intel CPUs with a massive breadth of software and less hard specifics at the edges.
api 6y ago
Hah... and I thought this was weird: https://youtrack.jetbrains.com/issue/JBR-2310#focus=streamItem-27-4151843.0-0 https://youtrack.jetbrains.com/issue/JBR-2310#focus=streamIt...
- an_opabinia 6y ago
  Oh my god! This is why my $4,000 laptop has been crashing!?
  - api 6y ago
    Only if you have a 10th generation Ice Lake, which is the 2020 Air and 2020 13" Pro. The 16" has a 9th generation Coffee Lake and does not seem to be affected, so if you have this or an earlier laptop it would have to be something else. Read the whole thing... it's a ride. Serious WTFBBQ with a side of potato salad. Once Apple and/or Intel notices this there will probably be a microcode update. Until then if AWS or some other host fields Ice Lake based server cores you may be able to crash the VM host by starting up a Jetbrains IDE in a VM. LOL The parent post on Transmeta QA shows that obscure bugs causing wacky edge cases are not that rare. It's a function of the complexity of modern chips, especially the X86 lineage with its crazy variable instruction widths and legacy cruft. You can, I think, still boot 16-bit DOS and Windows 3.1 on them! ARM is intrinsically less prone to this due to simpler pipelines and less cruft, but ARM64 chips are still pretty complex. Any real world chip that has been around for a while is going to grow some hair on it. ARM has some shag but X86 is hairier than a wizened yak. RISC-V is clean... for now. But there are no really high performance RV chips. I wonder though... do they not fuzz chips? How do these things escape QA at a serious experienced house like Intel? I can understand obscure security issues slipping by but an a hard crash?
    - garaetjjte 6y ago
      >Read the whole thing... it's a ride. Serious WTFBBQ with a side of potato salad. Did they rule out GPU driver crashes? They said it also occurs in VM, but didn't specify whether they had any guest integrations enabled.
userbinator 6y ago
The problem had something to do with a voltage drop on a certain chipset. Did you use any specific stress-testing programs, or just tested with regular applications? Something like the LINPACK benchmark is extremely stressful on the CPU and memory system, and is commonly used today for system stability testing.
- PopeDotNinja 6y ago
  I personally just ran whatever I felt like running. We had testing scripts to follow, but I largely ignored them XD My super power was testing things that other people wouldn't normally test. I was good enough at breaking things in unexpected ways that I mostly got away with it.
dvdkhlng 6y ago
As somebody who owned both a Crusoe-based and an Efficeon-based machine, let me thank you and your colleagues for the great work you did. I never suffered any stability issues with these machines running various Linux flavours (mostly Debian and Ubuntu).
tyingq 6y ago
That sort of detail is fabulous, and reflects really how little we've progressed when it comes down to "brass tacks". Thanks for sharing.
dfcarney 6y ago
I was on the "silicon level debugging" team, which took test cases like this and boiled them down to x86, then through the CMS to p95 assembler, then down to the circuit and device level to figure out the speed paths. There were definitely some crazy issues we found. Honestly, I couldn't have done that job without relying on 80%+ of my undergrad courses (Computer Engineering @ Waterloo). The issue I remember the most clearly was one that an engineer Simon Barrett discovered: a specific sequence of MMX instructions would cross-talk (and cause the occasional bit to flip) on two perpendicular busses in the pipeline when run at a specific voltage/frequency. It sounds simple enough, but imagine trying to figure that one out.

plerpin 6y ago

With strange failures like that, I'm surprised the system even booted at all.

deleted 6y ago
[deleted]
FullyFunctional 6y ago
Pope was a QA engineer - that means he worked on systems that weren't qualified for shipping. Actual CMS bugs in a shipped device was extremely rare, but the big promise of Transmeta was that would _could_ ship fixes (and did, in at least one case). The perf impact of a CMS fix was typically trivial as it was fixing a translator, unlike an interpreter like the microcode in an Intel core.
- PopeDotNinja 6y ago
  I mostly worked on development boards that were the same size that you'd put in a regular AT/ATX case. I did some work on notebooks, too, both on prototypes & production models, too.
  - FullyFunctional 6y ago
    I think I know you (Hi!), but my point is that by definition, QA's work is to find the bugs before the customers, so they don't see them. Talking about bugs you find without that qualifier might make people get the wrong impression.
    - PopeDotNinja 6y ago
      Hola!
ethbro 6y ago
One thing to remember is that this dropped circa-2001. The dominant Windows versions were ME and XP. Both of which featured driver models of... not too much system protection. So applications (especially games) looked a lot more like their console cousins, in that they took advantage of all kinds of weird hacks and did things 1,000 different ways. As opposed to now, when you have more strict intermediary layers, and software sits on top of more defined interfaces. So things like "the MechWarrior developer in charge of jump jets decided to base timing off the PCI bus speed" or god knows what, isn't that surprising.

VectorLock 6y ago

Thats why I was always skeptical of the Transmeta CPUs. They could 'run' the x86 ISA as its on paper, but they'd never be able to completely match all the nuances of the Intel CPUs to get 100% compatibility. The devils in the details as they say.

saagarjha 6y ago
Reimplementing an ISA is not actually an impossible feat; many companies have done so successfully before.
- VectorLock 6y ago
  Nothing is impossible, but has anybody done it since 2000 starting from scratch? AMD obviously has been at it from the start, Cyrix, VIA, NEC all did it but any since Intel CPUs started being in the tens of millions of transistors?
  - weare138 6y ago
    Hitachi SH-4 | 10M transistors | 1997 SPARC64 V | 191M transistors | 2001 DEC Alpha 21364 (EV7) | 152M transistors | 2003 SPARC64 V+ | 400M transistors | 2004 Cell | 250M transistors | 2005 POWER6 | 789M transistors | 2007 SPARC64 VI | 540M transistors | 2007 SPARC64 VIIIfx | 760M transistors | 2009 16-core SPARC T3 | 1B transistors | 2010 8-core POWER7 | 1.2B transistors | 2010 Quad-core z196 | 1.4B transistors | 2010 SPARC64 IXfx | 1.87B transistors | 2011 SPARC64 X | 2.99B transistors | 2012 8-core POWER7+ | 2.1B transistors | 2012 Apple A7 | 1B transistors | 2013 12-core POWER8 | 4.2B transistors | 2013 Apple A8 | 2B transistors | 2014 Apple A8X | 3B transistors | 2014 32-core SPARC M7 | 10B transistors | 2015 [insert a crapton of ARM CPUs here] AWS Graviton2 | 30B transistors | 2019
    - VectorLock 6y ago
      No x86 ISAs in here. All licensed designs or first party manufacturer. I wasn't saying nobody could make anything as complex as current Intel CPUs. Cloning other ISAs either from clean room or from licensed designs like ARM seems to be much easier than matching Intel CPUs with a massive breadth of software and less hard specifics at the edges.

api 6y ago

Hah... and I thought this was weird: https://youtrack.jetbrains.com/issue/JBR-2310#focus=streamItem-27-4151843.0-0 https://youtrack.jetbrains.com/issue/JBR-2310#focus=streamIt...

an_opabinia 6y ago
Oh my god! This is why my $4,000 laptop has been crashing!?
- api 6y ago
  Only if you have a 10th generation Ice Lake, which is the 2020 Air and 2020 13" Pro. The 16" has a 9th generation Coffee Lake and does not seem to be affected, so if you have this or an earlier laptop it would have to be something else. Read the whole thing... it's a ride. Serious WTFBBQ with a side of potato salad. Once Apple and/or Intel notices this there will probably be a microcode update. Until then if AWS or some other host fields Ice Lake based server cores you may be able to crash the VM host by starting up a Jetbrains IDE in a VM. LOL The parent post on Transmeta QA shows that obscure bugs causing wacky edge cases are not that rare. It's a function of the complexity of modern chips, especially the X86 lineage with its crazy variable instruction widths and legacy cruft. You can, I think, still boot 16-bit DOS and Windows 3.1 on them! ARM is intrinsically less prone to this due to simpler pipelines and less cruft, but ARM64 chips are still pretty complex. Any real world chip that has been around for a while is going to grow some hair on it. ARM has some shag but X86 is hairier than a wizened yak. RISC-V is clean... for now. But there are no really high performance RV chips. I wonder though... do they not fuzz chips? How do these things escape QA at a serious experienced house like Intel? I can understand obscure security issues slipping by but an a hard crash?
  - garaetjjte 6y ago
    >Read the whole thing... it's a ride. Serious WTFBBQ with a side of potato salad. Did they rule out GPU driver crashes? They said it also occurs in VM, but didn't specify whether they had any guest integrations enabled.

userbinator 6y ago

The problem had something to do with a voltage drop on a certain chipset. Did you use any specific stress-testing programs, or just tested with regular applications? Something like the LINPACK benchmark is extremely stressful on the CPU and memory system, and is commonly used today for system stability testing.

PopeDotNinja 6y ago
I personally just ran whatever I felt like running. We had testing scripts to follow, but I largely ignored them XD My super power was testing things that other people wouldn't normally test. I was good enough at breaking things in unexpected ways that I mostly got away with it.

dvdkhlng 6y ago

As somebody who owned both a Crusoe-based and an Efficeon-based machine, let me thank you and your colleagues for the great work you did. I never suffered any stability issues with these machines running various Linux flavours (mostly Debian and Ubuntu).

tyingq 6y ago

That sort of detail is fabulous, and reflects really how little we've progressed when it comes down to "brass tacks". Thanks for sharing.

dfcarney 6y ago

I was on the "silicon level debugging" team, which took test cases like this and boiled them down to x86, then through the CMS to p95 assembler, then down to the circuit and device level to figure out the speed paths. There were definitely some crazy issues we found. Honestly, I couldn't have done that job without relying on 80%+ of my undergrad courses (Computer Engineering @ Waterloo). The issue I remember the most clearly was one that an engineer Simon Barrett discovered: a specific sequence of MMX instructions would cross-talk (and cause the occasional bit to flip) on two perpendicular busses in the pipeline when run at a specific voltage/frequency. It sounds simple enough, but imagine trying to figure that one out.