5 ms·

Serious question, how is this different from what VW did? There was a lot of talk about complicit engineers vs. shady management in that discussion. Did this ev

Serious question, how is this different from what VW did? There was a lot of talk about complicit engineers vs. shady management in that discussion. Did this ever make it to production?

henryfjordan 6y ago
VW actively detected if the car was on a treadmill testing setup and electronically changed the engine parameters to make it run cleaner. The water heater was designed with the test in mind but functions the same regardless. VW was clearly cheating whereas the water heater is taking advantage of known deficiencies in the test (is that cheating?).
- ivanbakel 6y ago
  What was more heinous about the VW case IIRC is that the cleanliness gains the cars made on the treadmill setup were enough to push them into compliance with regulations, which the engines normally didn't meet. "Cheating" a better efficiency rating is significantly less reprehensible than cheating a legal obligation which was legislated for public health and environmental reasons.
  - ebg13 6y ago
    But the efficiency ratings also exist for public health and environmental reasons. Why does it matter to you that one is a mandate? They're both intentionally deceiving regulators and consumers (vs accidentally acing the test without cheating). Both are fraud.
    - ReactiveJelly 6y ago
      One is polluting at the tailpipe, the other is only polluting at a power plant.
    - Bartweiss 6y ago
      I do think that manipulating a purely instructive measure is less extreme than manipulating a compliance test; consumers can seek alternate tests and reviews, but the state emissions test has special status even if a dozen other tests give a different result. That said, I believe Energy Star ratings affect tax rebates and electric bills, and they're required to be printed on products - so that's not really an arbitrary test. There are other differences here too, I think. The water heater trick is passive manipulation that stays in place at all times, which limits how far from "real" performance it can get. And per the story, it seems more like "teaching to the test" than "cheating". That is, Volkswagen consciously moved away from the mandate outside of testing. The water heater was (potentially) as energy-efficient as they could design, with the test score manipulated on top of that. None of that makes it harmless - if "as good as you can make" doesn't hit standards without manipulating them, that's still a problem. But I do find it less galling than "intentionally worsens emissions outside the test bench".
      - Gibbon1 6y ago
        The flip side of the water heater test is, you could game the test the other way too. Making your water heater look worse than it is. Would you do that? No. The difference between the water heater and VW is the water heater manufacturer is providing a representative sample. And VW was not. It'd also be dubious to say that the water heater company is acting in bad faith. Where VW's bad faith rose to the level of criminal. On the other hand Volvo appears to be acting in strictly good faith. Bad faith for a crash test would be crafting a silver plate model for testing. Reminds me that's what my uncle said the power supply manufacturer he worked for did.
    - kelnos 6y ago
      The difference is that the water heater test itself was flawed in that it depended on arbitrary design decisions that have nothing to do with efficiency. Two completely innocent manufacturers could build water heaters with the exact same real-world efficiency, but score fairly differently on the efficiency ratings just due to how they're designed. While I agree that this particular water heater manufacturer was doing something shady in order to get the best score, at least they weren't selling a product that did something differently while under test conditions vs. in real-world usage. They merely realized that the test itself had wide error bars, and designed their heater to "err" in the positive side of those. VW, in contrast, sold a product that lied to the testers about its emissions in order to pass certifications, while in real-world driving would behave in a way that would not pass muster. And to me I think that's the key: VW's cars intentionally behaved differently depending on if they were being tested or if they were being driven in normal real-world usage. This water heater behaved the same regardless of whether it was being tested or was heating water in someone's home. In a way I think of this in academic terms. The water heater manufacturer studied the SAT to learn what kind of questions were going to be asked. VW stole the answer key to the test and memorized it.
  - deleted 6y ago
    [deleted]
- alistairSH 6y ago
  I'm not sure that's an accurate description of VW's cheat. VW knew the parameters of the test (stationary car, no steering input, prescribed throttle inputs, etc). And they configured the car to pass the test. Water-heater engineer knew the parameters of the test (number and location of probes). And he configure the water-heater to pass the test. Same-same. In both cases, an engineering team willfully committed fraud to improve their sales figures. The only differences are a bit pedantic. VW's "configuration" was more elaborate. But, both groups gamed the test.
  - amscanne 6y ago
    You might misunderstand what VW did. Apparently it is common for cars to have a "test mode" bit, because they run on a dyno (only one set of wheels spin), and the car may disable certain traction control systems, etc. VW changed the way other systems (the engine itself) function in this mode. So even if everything is exactly the same on the road (e.g. 30mph in a straight line for hours), the car will perform differently and have different emissions. You don't drive around on a dyno. Optimizing for the test may go up to the line, but VW crossed it.
  - Nasrudith 6y ago
    One added difference is that the water heater. The intent may be deceptive but they provided what was asked for and what they claimed. It is almost malicious compliance. Dickish gaming and probably worth suing over but not sure if it is criminal fraud per se. It is like the joke about Soviet factory metrics for some item like nails. They set the quota on numbers and got useless ones better described as needles. They wised up a bit and set the quota by weight next month and got one useless massive nail instead. It isn't even too far from the truth given actual management involved things like a train line circularly shipping coal between depots instead of retrieving from source or distributing to end users to boost their "metric tons transported kilometers" metric. VW meanwhile had a covert illegal configuration while claiming mileage and lack of maintenancd urea. The claims are outright false - that it provides all three benefits instead of two of three.
    - viraptor 6y ago
kube-system 6y ago
The big difference with VW is that they put in a mechanism that detects the test and completely changed the behavior of the car for the test.
- ebg13 6y ago
  So they _intentionally_ set themselves up to lie to regulatory agencies and consumers about real world efficiency. That honestly sounds basically the same to me. In both cases the tests are poor approximations, and in both cases someone could accidentally optimize the test, and in both cases someone did it intentionally to deceive people.
  - kube-system 6y ago
    Morally, they both seem to fall in the same category. Legally, it might be a complicated question.
  - henryfjordan 6y ago
    Imagine the water heater company didn't know they were gaming the test. They first design a water heater and it gets a B- on the test. Being overachievers, they work hard and submit a second design that gets an A+. They might not realize that both heaters are basically the same with the only difference being some heating element spacing that works better for the test. Both times they submitted a legit design that was the same they'd provide to consumers. Sure, we know the engineers knew what was happening, but we can see how one might innocently arrive in the same scenario. I think it's safe to say the test is flawed. The VW test is not like that. There's no way to innocently arrive in the scenario they did. They did not game a bad test, they literally lied to the test administrators. The car ran in "clean mode" only if it was in the test environment. If the car ran like it did on the road, they'd have failed (which is how they were caught, with a mobile testing setup). One of the points in the article is that regulating for safety based on known testing conditions is going to result in over-fitting for the test. The water heater company is guilty of intentionally over-fitting. VW just straight up lied. I don't think those 2 actions are equal, VW is worse, but I agree that both are dishonest to a degree.
duxup 6y ago
VW's vehicles did not meet the standards required when used every day. They only met the standard during testing. The water heaters don't sound like they'd fail any given test.
im3w1l 6y ago
Many important differences. 1. Actively detecting test and behaving differently. It's like stealing a test vs teaching to the test. 2. Lower stakes. Health issues are much more serious than inefficiency. 3. It affects the buyer. It's more acceptable for the buyer to be cheated than everyone around them. 4. People could have created these layers by accident. Favouring those who got lucky is unfair. Honestly I think basically all my gadgets exaggerate how energy efficient they are, by tuning parameters for tests that don't correspond to the real world. My dishwasher has an energy efficient mode, the manual literally says it's just for compliance and recommends other modes. It's just a fact of life.
- Bartweiss 6y ago
  This is omnipresent even where regulators aren't involved: every graphics card benchmark out there is 'manipulated' relative to real world performance. At this point it's so universal that I don't think anyone is even fighting it - as long as everyone games benchmarks roughly the same amount, the relative scores stay usable. Your point about fairness and passive design is the one that makes me view these cases differently also. In the anecdote, the product being tested was the same one being sold, and there's no sign the heater was worsened to improve test performance. The designers just picked the best-scoring option among some reasonable configurations. (Frankly, once they noticed that issue, what were they supposed to do? Pick the worst-scoring, or pick the spec out of a hat?) In the VW story, the test-bench vehicle was fundamentally different from the market vehicle, and the road version was designed to behave worse on the metrics to get other gains. I happen to know someone who bought a diesel Jetta specifically because it was more eco-friendly than other options, and I think he'd draw a clear line between tuning for test metrics and VW consciously lying to their buyers.
  - orclev 6y ago
    It's interesting you mention graphics cards because that very behavior has lead to the gaming community favoring benchmarks derived from a handful of current gen games min/max/avg FPS over so called synthetic benchmarks. It only took a handful of instances of companies baking in "benchmark" modes that get triggered when certain benchmarks are detected for people to start discounting those benchmarks in favor of more organic measurements.
malandrew 6y ago
With the following question, I'm not absolving VW from criticism. With that in mind: Why are we not holding those doing the measurement accountable as well? If you produce a test that can be gamed and your job is to test things to meet consumer expectations, you've failed at your job. After all is said and done, what is a better outcome: a) VW is punished for gaming the test b) the test is significantly harder to game With (a), we have only one less manufacturer gaming the tests, VW. With (b) we have tests that none of the manufacturers can game any longer or at least will take time to game. The testers should be expected to always be two steps ahead. This is not unlike whitehat/blackhat security engineering. We should pay bug bounties to teams that successfully exploit the tests and we should be actively running red team drills. https://en.wikipedia.org/wiki/Red_team https://en.wikipedia.org/wiki/Red_team
dndvr 6y ago
What VW did is similar to uber's greyball system in that they give a different experience to the regulator rather than giving everyone an experience that is tuned to what a regulator might hope to see

Serious question, how is this different from what VW did? There was a lot of talk about complicit engineers vs. shady management in that discussion. Did this ever make it to production?

henryfjordan 6y ago

VW actively detected if the car was on a treadmill testing setup and electronically changed the engine parameters to make it run cleaner. The water heater was designed with the test in mind but functions the same regardless. VW was clearly cheating whereas the water heater is taking advantage of known deficiencies in the test (is that cheating?).

ivanbakel 6y ago
What was more heinous about the VW case IIRC is that the cleanliness gains the cars made on the treadmill setup were enough to push them into compliance with regulations, which the engines normally didn't meet. "Cheating" a better efficiency rating is significantly less reprehensible than cheating a legal obligation which was legislated for public health and environmental reasons.
- ebg13 6y ago
  But the efficiency ratings also exist for public health and environmental reasons. Why does it matter to you that one is a mandate? They're both intentionally deceiving regulators and consumers (vs accidentally acing the test without cheating). Both are fraud.
  - ReactiveJelly 6y ago
    One is polluting at the tailpipe, the other is only polluting at a power plant.
  - Bartweiss 6y ago
    I do think that manipulating a purely instructive measure is less extreme than manipulating a compliance test; consumers can seek alternate tests and reviews, but the state emissions test has special status even if a dozen other tests give a different result. That said, I believe Energy Star ratings affect tax rebates and electric bills, and they're required to be printed on products - so that's not really an arbitrary test. There are other differences here too, I think. The water heater trick is passive manipulation that stays in place at all times, which limits how far from "real" performance it can get. And per the story, it seems more like "teaching to the test" than "cheating". That is, Volkswagen consciously moved away from the mandate outside of testing. The water heater was (potentially) as energy-efficient as they could design, with the test score manipulated on top of that. None of that makes it harmless - if "as good as you can make" doesn't hit standards without manipulating them, that's still a problem. But I do find it less galling than "intentionally worsens emissions outside the test bench".
    - Gibbon1 6y ago
      The flip side of the water heater test is, you could game the test the other way too. Making your water heater look worse than it is. Would you do that? No. The difference between the water heater and VW is the water heater manufacturer is providing a representative sample. And VW was not. It'd also be dubious to say that the water heater company is acting in bad faith. Where VW's bad faith rose to the level of criminal. On the other hand Volvo appears to be acting in strictly good faith. Bad faith for a crash test would be crafting a silver plate model for testing. Reminds me that's what my uncle said the power supply manufacturer he worked for did.
  - kelnos 6y ago
    The difference is that the water heater test itself was flawed in that it depended on arbitrary design decisions that have nothing to do with efficiency. Two completely innocent manufacturers could build water heaters with the exact same real-world efficiency, but score fairly differently on the efficiency ratings just due to how they're designed. While I agree that this particular water heater manufacturer was doing something shady in order to get the best score, at least they weren't selling a product that did something differently while under test conditions vs. in real-world usage. They merely realized that the test itself had wide error bars, and designed their heater to "err" in the positive side of those. VW, in contrast, sold a product that lied to the testers about its emissions in order to pass certifications, while in real-world driving would behave in a way that would not pass muster. And to me I think that's the key: VW's cars intentionally behaved differently depending on if they were being tested or if they were being driven in normal real-world usage. This water heater behaved the same regardless of whether it was being tested or was heating water in someone's home. In a way I think of this in academic terms. The water heater manufacturer studied the SAT to learn what kind of questions were going to be asked. VW stole the answer key to the test and memorized it.
- deleted 6y ago
  [deleted]
alistairSH 6y ago
I'm not sure that's an accurate description of VW's cheat. VW knew the parameters of the test (stationary car, no steering input, prescribed throttle inputs, etc). And they configured the car to pass the test. Water-heater engineer knew the parameters of the test (number and location of probes). And he configure the water-heater to pass the test. Same-same. In both cases, an engineering team willfully committed fraud to improve their sales figures. The only differences are a bit pedantic. VW's "configuration" was more elaborate. But, both groups gamed the test.
- amscanne 6y ago
  You might misunderstand what VW did. Apparently it is common for cars to have a "test mode" bit, because they run on a dyno (only one set of wheels spin), and the car may disable certain traction control systems, etc. VW changed the way other systems (the engine itself) function in this mode. So even if everything is exactly the same on the road (e.g. 30mph in a straight line for hours), the car will perform differently and have different emissions. You don't drive around on a dyno. Optimizing for the test may go up to the line, but VW crossed it.
- Nasrudith 6y ago
  One added difference is that the water heater. The intent may be deceptive but they provided what was asked for and what they claimed. It is almost malicious compliance. Dickish gaming and probably worth suing over but not sure if it is criminal fraud per se. It is like the joke about Soviet factory metrics for some item like nails. They set the quota on numbers and got useless ones better described as needles. They wised up a bit and set the quota by weight next month and got one useless massive nail instead. It isn't even too far from the truth given actual management involved things like a train line circularly shipping coal between depots instead of retrieving from source or distributing to end users to boost their "metric tons transported kilometers" metric. VW meanwhile had a covert illegal configuration while claiming mileage and lack of maintenancd urea. The claims are outright false - that it provides all three benefits instead of two of three.
  - viraptor 6y ago

kube-system 6y ago

The big difference with VW is that they put in a mechanism that detects the test and completely changed the behavior of the car for the test.

ebg13 6y ago
So they _intentionally_ set themselves up to lie to regulatory agencies and consumers about real world efficiency. That honestly sounds basically the same to me. In both cases the tests are poor approximations, and in both cases someone could accidentally optimize the test, and in both cases someone did it intentionally to deceive people.
- kube-system 6y ago
  Morally, they both seem to fall in the same category. Legally, it might be a complicated question.
- henryfjordan 6y ago
  Imagine the water heater company didn't know they were gaming the test. They first design a water heater and it gets a B- on the test. Being overachievers, they work hard and submit a second design that gets an A+. They might not realize that both heaters are basically the same with the only difference being some heating element spacing that works better for the test. Both times they submitted a legit design that was the same they'd provide to consumers. Sure, we know the engineers knew what was happening, but we can see how one might innocently arrive in the same scenario. I think it's safe to say the test is flawed. The VW test is not like that. There's no way to innocently arrive in the scenario they did. They did not game a bad test, they literally lied to the test administrators. The car ran in "clean mode" only if it was in the test environment. If the car ran like it did on the road, they'd have failed (which is how they were caught, with a mobile testing setup). One of the points in the article is that regulating for safety based on known testing conditions is going to result in over-fitting for the test. The water heater company is guilty of intentionally over-fitting. VW just straight up lied. I don't think those 2 actions are equal, VW is worse, but I agree that both are dishonest to a degree.

duxup 6y ago

VW's vehicles did not meet the standards required when used every day. They only met the standard during testing. The water heaters don't sound like they'd fail any given test.

im3w1l 6y ago

Many important differences. 1. Actively detecting test and behaving differently. It's like stealing a test vs teaching to the test. 2. Lower stakes. Health issues are much more serious than inefficiency. 3. It affects the buyer. It's more acceptable for the buyer to be cheated than everyone around them. 4. People could have created these layers by accident. Favouring those who got lucky is unfair. Honestly I think basically all my gadgets exaggerate how energy efficient they are, by tuning parameters for tests that don't correspond to the real world. My dishwasher has an energy efficient mode, the manual literally says it's just for compliance and recommends other modes. It's just a fact of life.

Bartweiss 6y ago
This is omnipresent even where regulators aren't involved: every graphics card benchmark out there is 'manipulated' relative to real world performance. At this point it's so universal that I don't think anyone is even fighting it - as long as everyone games benchmarks roughly the same amount, the relative scores stay usable. Your point about fairness and passive design is the one that makes me view these cases differently also. In the anecdote, the product being tested was the same one being sold, and there's no sign the heater was worsened to improve test performance. The designers just picked the best-scoring option among some reasonable configurations. (Frankly, once they noticed that issue, what were they supposed to do? Pick the worst-scoring, or pick the spec out of a hat?) In the VW story, the test-bench vehicle was fundamentally different from the market vehicle, and the road version was designed to behave worse on the metrics to get other gains. I happen to know someone who bought a diesel Jetta specifically because it was more eco-friendly than other options, and I think he'd draw a clear line between tuning for test metrics and VW consciously lying to their buyers.
- orclev 6y ago
  It's interesting you mention graphics cards because that very behavior has lead to the gaming community favoring benchmarks derived from a handful of current gen games min/max/avg FPS over so called synthetic benchmarks. It only took a handful of instances of companies baking in "benchmark" modes that get triggered when certain benchmarks are detected for people to start discounting those benchmarks in favor of more organic measurements.

malandrew 6y ago

With the following question, I'm not absolving VW from criticism. With that in mind: Why are we not holding those doing the measurement accountable as well? If you produce a test that can be gamed and your job is to test things to meet consumer expectations, you've failed at your job. After all is said and done, what is a better outcome: a) VW is punished for gaming the test b) the test is significantly harder to game With (a), we have only one less manufacturer gaming the tests, VW. With (b) we have tests that none of the manufacturers can game any longer or at least will take time to game. The testers should be expected to always be two steps ahead. This is not unlike whitehat/blackhat security engineering. We should pay bug bounties to teams that successfully exploit the tests and we should be actively running red team drills. https://en.wikipedia.org/wiki/Red_team https://en.wikipedia.org/wiki/Red_team

dndvr 6y ago

What VW did is similar to uber's greyball system in that they give a different experience to the regulator rather than giving everyone an experience that is tuned to what a regulator might hope to see