3 ms·

>In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how the

>In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips. I hope you're not in charge of hiring QA. Bugs are often found by people who AREN'T thinking like the developers whichstart wearing blinders on how they're stuff should work and stop trying stupid things.

BeeOnRope 3y ago
I'm not directly in charge of hiring QA, no! I think this sort of excuses the initial blindness to Spectre style attacks in the first place, but once the basic pattern was clear it doesn't excuse the subsequent lack of discovering any of the subsequent issues. It is as if someone found a bug by examining the assembly language of your process which was caused by unsafe inlined `strcpy` calls (though they could not see the source, so they had no idea strcpy was the problem), and then over a the subsequent 6 years other people slowly found more strcpy issues in your application using brute-force black-box engineering and meanwhile you never just grepped your (closed) source for strcpy uses and audited them or used many of the compiler or runtime mitigations against this issue.
- justinator 3y ago
  That still seems like a job for QA and you do have a good point. If there's an attack using one technique, there should be an audit to make sure that technique can't be modified to be used in other nasty ways, then have those tests part of the test suite.
margalabargala 3y ago
> I hope you're not in charge of hiring QA. This seems unnecessarily harsh. The whole post would be improved by removing that sentence IMO.
- justinator 3y ago
  I'm in the camp that developers shouldn't be responsible for having a transcendental ability to predict future (example) security holes. If one appears, it's really the QA's job to document it and experiment with other similar vectors of attack. The developers are, you know: developing. Bring them back in when the security issues needs to be fixed, once it's found just how big the problem is.
  - BeeOnRope 3y ago
    Shouldn't the functions of "development" and "QA" both reside under the umbrella of the chip-maker though? In fact, chip-makers famously invest an insane amount of money into "QA" (aka "validation") and many features or lack thereof are often put down to the cost of QA rather than the cost of development.
    - IIsi50MHz 3y ago
      Possibly. When I worked at a place that designed a "simple" chip that was just a more energy efficient and very parallel version of its FPGA, the fab that was contracted to make it insisted on validating it themselves. It consisted primarily of a lot of copy-paste of the primary logic to build as many paths as would fit on the die. They described it as "unusually dense" and, I heard they later said that they'd never seen a design that dense. The validation process was partly described as people manually driving a car through a 3D model, making sure there were no unexpected junctions or other divergences. This process took months longer than their initial estimate, allegedly due to the density. I guess you could say this was "under the umbrella of the chip-maker", though we had little say in it aside from pressing them for progress as our final product's shipdates came and left. When we finally got the first samples, power consumption was, I think, an order of magnitude higher than expected. Our lead engineer struggled to get it down without going to a smaller process that we could barely afford (and given the delays already, could probably not have afforded to wait for). We thus thought we had working logic, but our case designs were scuttled. After enlarging the cases to accommodate extra cooling, our base unit was more than ten times taller, and our next size up, while the same height, was three times longer. Highest units had a water cooling system[1]. Our QA was able to find other sorts of flaws, like misprinted unpopulated circuit boards, software faults on the host, or when we received shoddy interlink cables that either melted under test[2] or other cables that scrambled communications[3]. All of which is to say, that many other pressing issues can interfere with doing what you feel you ought to be doing. At least at our scale. I can't speak for the likes of big guys like Intel or AMD, but it's possible that unfound faults or known unpatched flaws can ship because resources were committed elsewhere or fabrication leadtimes preclude waiting. This is not to say that shipping a security flaw is okay, but rather that sometimes you think you've done you're due diligence, or sometimes your choices seem to be "Ship, ship late, or never ship.". The answer you pick can be existential, so you hope you've picked the least bad option. [1] Misbegotten, because "beauty of the promo images". [2] Conductors much thinner than spec, not initially observed because both ends were fitted with moulded plugs. [3] Longer than spec, initially recieved with enthusiasm by assembly staff, before an engineer investigating a difficulty saw them and exclaimed, "No-no-no! That's longer than I am tall! Stray capacitance alone will kill the communication.". (He uncharacteristic'ly exagerated here. While they were three times longer than expected, this was at most .40 times his height.)
netheril96 3y ago
No QA is able to find out a Spectre vulnerability. QA is irrelevant in this conversation.
mekoka 3y ago
> I hope you're not in charge of hiring QA. Why the personal attack? The logic is sound. Isn't QA typically part of the same organization?
specialist 3y ago
Spot on. Generalizing: QA/Test and dev people just think differently. I served as QA manager for a while. I was fortunate to have worked with some really, really good QA/Test people. They're more rare than good devs. Some individuals can do well in both worlds. I've had great devs who were pretty good at test. It seems to me like these devs came from outside of software and CS. Like from aerospace or ballet or history. I've mentored QA/Test people so they could better automate and manage their work. Then they could pick up tasks like CI/CD, testing scripts, scrub data, etc. But, in my experience, devs are bad at testing, and just terrible at QA. Getting enough QA/Test support on a team in the 90s was a tough sell, even though everyone gave lip service to quality. It's been a long time since I've worked with an actual Test person, dedicated to that role. And that was just 1 person vs 8 devs. So ridiculous. These days, any kind of "test" is done by "business analysts", whatever that means. And I can't recall the last QA person I've worked with (since my stint as QA Manager in the '90s). FWIW, I wholly agree with Dan Luu's observations about today's QA/Test standard practice in software vs hardware.

BeeOnRope 3y ago

I'm not directly in charge of hiring QA, no! I think this sort of excuses the initial blindness to Spectre style attacks in the first place, but once the basic pattern was clear it doesn't excuse the subsequent lack of discovering any of the subsequent issues. It is as if someone found a bug by examining the assembly language of your process which was caused by unsafe inlined `strcpy` calls (though they could not see the source, so they had no idea strcpy was the problem), and then over a the subsequent 6 years other people slowly found more strcpy issues in your application using brute-force black-box engineering and meanwhile you never just grepped your (closed) source for strcpy uses and audited them or used many of the compiler or runtime mitigations against this issue.

justinator 3y ago
That still seems like a job for QA and you do have a good point. If there's an attack using one technique, there should be an audit to make sure that technique can't be modified to be used in other nasty ways, then have those tests part of the test suite.

margalabargala 3y ago

> I hope you're not in charge of hiring QA. This seems unnecessarily harsh. The whole post would be improved by removing that sentence IMO.

justinator 3y ago
I'm in the camp that developers shouldn't be responsible for having a transcendental ability to predict future (example) security holes. If one appears, it's really the QA's job to document it and experiment with other similar vectors of attack. The developers are, you know: developing. Bring them back in when the security issues needs to be fixed, once it's found just how big the problem is.
- BeeOnRope 3y ago
  Shouldn't the functions of "development" and "QA" both reside under the umbrella of the chip-maker though? In fact, chip-makers famously invest an insane amount of money into "QA" (aka "validation") and many features or lack thereof are often put down to the cost of QA rather than the cost of development.
  - IIsi50MHz 3y ago
    Possibly. When I worked at a place that designed a "simple" chip that was just a more energy efficient and very parallel version of its FPGA, the fab that was contracted to make it insisted on validating it themselves. It consisted primarily of a lot of copy-paste of the primary logic to build as many paths as would fit on the die. They described it as "unusually dense" and, I heard they later said that they'd never seen a design that dense. The validation process was partly described as people manually driving a car through a 3D model, making sure there were no unexpected junctions or other divergences. This process took months longer than their initial estimate, allegedly due to the density. I guess you could say this was "under the umbrella of the chip-maker", though we had little say in it aside from pressing them for progress as our final product's shipdates came and left. When we finally got the first samples, power consumption was, I think, an order of magnitude higher than expected. Our lead engineer struggled to get it down without going to a smaller process that we could barely afford (and given the delays already, could probably not have afforded to wait for). We thus thought we had working logic, but our case designs were scuttled. After enlarging the cases to accommodate extra cooling, our base unit was more than ten times taller, and our next size up, while the same height, was three times longer. Highest units had a water cooling system[1]. Our QA was able to find other sorts of flaws, like misprinted unpopulated circuit boards, software faults on the host, or when we received shoddy interlink cables that either melted under test[2] or other cables that scrambled communications[3]. All of which is to say, that many other pressing issues can interfere with doing what you feel you ought to be doing. At least at our scale. I can't speak for the likes of big guys like Intel or AMD, but it's possible that unfound faults or known unpatched flaws can ship because resources were committed elsewhere or fabrication leadtimes preclude waiting. This is not to say that shipping a security flaw is okay, but rather that sometimes you think you've done you're due diligence, or sometimes your choices seem to be "Ship, ship late, or never ship.". The answer you pick can be existential, so you hope you've picked the least bad option. [1] Misbegotten, because "beauty of the promo images". [2] Conductors much thinner than spec, not initially observed because both ends were fitted with moulded plugs. [3] Longer than spec, initially recieved with enthusiasm by assembly staff, before an engineer investigating a difficulty saw them and exclaimed, "No-no-no! That's longer than I am tall! Stray capacitance alone will kill the communication.". (He uncharacteristic'ly exagerated here. While they were three times longer than expected, this was at most .40 times his height.)

netheril96 3y ago

No QA is able to find out a Spectre vulnerability. QA is irrelevant in this conversation.

mekoka 3y ago

> I hope you're not in charge of hiring QA. Why the personal attack? The logic is sound. Isn't QA typically part of the same organization?

specialist 3y ago

Spot on. Generalizing: QA/Test and dev people just think differently. I served as QA manager for a while. I was fortunate to have worked with some really, really good QA/Test people. They're more rare than good devs. Some individuals can do well in both worlds. I've had great devs who were pretty good at test. It seems to me like these devs came from outside of software and CS. Like from aerospace or ballet or history. I've mentored QA/Test people so they could better automate and manage their work. Then they could pick up tasks like CI/CD, testing scripts, scrub data, etc. But, in my experience, devs are bad at testing, and just terrible at QA. Getting enough QA/Test support on a team in the 90s was a tough sell, even though everyone gave lip service to quality. It's been a long time since I've worked with an actual Test person, dedicated to that role. And that was just 1 person vs 8 devs. So ridiculous. These days, any kind of "test" is done by "business analysts", whatever that means. And I can't recall the last QA person I've worked with (since my stint as QA Manager in the '90s). FWIW, I wholly agree with Dan Luu's observations about today's QA/Test standard practice in software vs hardware.