4 ms·

If you have to pass some standardized tests, the engineers will do what they're supposed to: Meet the goals. I wouldn't call it 'rigged tests', it's more like

by nom 9y ago

If you have to pass some standardized tests, the engineers will do what they're supposed to: Meet the goals. I wouldn't call it 'rigged tests', it's more like they optimize the system under test to meet the spec. It probably happens everywhere, not only in cars. The behavior of your fridge, AC or any other hardware is probably optimized in weird ways just to meet Energy Star etc.

electrograv 9y ago
Exactly -- what we need are better tests that aren't as easy to game. The idea of installing a real-world pollution measurement device to each car is still valid though -- that would just become the new test.
hwillis 9y ago
>I wouldn't call it 'rigged tests', it's more like they optimize the system under test to meet the spec. You would do so wrongly. In these cases, like VW, the car is programmed specifically to detect when it is being tested, and lower emissions. They aren't optimizing to test spec, they are cheating. There's no other word for it.
- greglindahl 9y ago
  In the auto industry, it's called a "defeat device", and has a long history of causing woe to car companies who use such devices.
  - nom 9y ago
    Since when? And which long history? Seriously, I don't know, where can I read up on this?
    - hwillis 9y ago
      Wikipedia has a list of a bunch of incidents going back to the 70s, in which the phrase "the largest fine to date" appears often: https://en.wikipedia.org/wiki/Defeat_device https://en.wikipedia.org/wiki/Defeat_device
- DannyBee 9y ago
  No, he's right. That's "optimize for the test spec". Detect an idiom and optimize it is very common in this situation. For example, commercial compilers are often written to specifically detect and rewrite idioms/code in oft-used benchmarks (that aren't common to the real world), etc Your browser's javascript engine almost certainly does it. They aren't cheating, they are just making the test worthless by optimizing it. They still shouldn't do it, but it's super-common.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]

by nom 9y ago

electrograv 9y ago

Exactly -- what we need are better tests that aren't as easy to game. The idea of installing a real-world pollution measurement device to each car is still valid though -- that would just become the new test.

hwillis 9y ago

>I wouldn't call it 'rigged tests', it's more like they optimize the system under test to meet the spec. You would do so wrongly. In these cases, like VW, the car is programmed specifically to detect when it is being tested, and lower emissions. They aren't optimizing to test spec, they are cheating. There's no other word for it.

greglindahl 9y ago
In the auto industry, it's called a "defeat device", and has a long history of causing woe to car companies who use such devices.
- nom 9y ago
  Since when? And which long history? Seriously, I don't know, where can I read up on this?
  - hwillis 9y ago
    Wikipedia has a list of a bunch of incidents going back to the 70s, in which the phrase "the largest fine to date" appears often: https://en.wikipedia.org/wiki/Defeat_device https://en.wikipedia.org/wiki/Defeat_device
DannyBee 9y ago
No, he's right. That's "optimize for the test spec". Detect an idiom and optimize it is very common in this situation. For example, commercial compilers are often written to specifically detect and rewrite idioms/code in oft-used benchmarks (that aren't common to the real world), etc Your browser's javascript engine almost certainly does it. They aren't cheating, they are just making the test worthless by optimizing it. They still shouldn't do it, but it's super-common.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]