6 ms·

Two things that are surprising to me about this story: 1) The secret was kept for so long. How many programmers were involved with the relevant code? How many

by harryh 11y ago

Two things that are surprising to me about this story: 1) The secret was kept for so long. How many programmers were involved with the relevant code? How many project managers signed off on it? Surely VW does its own emissions testing internally so some of them must have known. How high up in management did this go? It seems like it must have been quite a few. Amazing that none of them got mad and told someone outside the company. 2) I would never have guessed that emissions from a car engine could vary so widely. 20%? Sure. 50%? Sure. But news outlets are reporting that these cars are emitting at least 10x and possibly as much as 40x NOx as they should. This is clearly because of my ignorance of the details of the engineering here, but I was shocked that such a difference could happen.

HNaTTY 11y ago
From what I've read so far, this has to do with heat necessary for the catalytic converter to absorb NOx -- without enough heat, NOx isn't absorbed, so in emissions test mode, they're just running the engine in such a way that more heat is generated to make sure the cat works properly.
- harryh 11y ago
  Interesting! How much heat are we talking about here? When an engine is just running in normal operation it generates a fair amount of heat naturally. I guess that's not enough?
  - ams6110 11y ago
    Diesel engines run with a surplus of air so they only burn the fuel they need to. They don't really generate a lot of heat unless they are under load (relative to gas engines). Just cruising down the highway a diesel will not need much fuel and will run fairly cool.
    - durin42 11y ago
      The owners manual for the VW TDIs actually warns you to not try and preheat the engine by letting it idle in the winter, because it won't work.
- mzs 11y ago
  Fuel economy improves at higher temperatures, but NOx production increases dramatically then. Conventional catalytic systems cannot keep-up so either a selective catalyst (expensive) or improved egr system (performance sapping) must be used.
teddyh 11y ago
Programmers are susceptible to venality just like anybody else. http://www.commitstrip.com/en/2015/09/22/software-worth-billions/ http://www.commitstrip.com/en/2015/09/22/software-worth-bill...
- harryh 11y ago
  Of course. My surprise has nothing to do with my opinion of programmers vs other professions. Instead it has to do with the ability of large groups to keep big secrets. That's a pretty hard thing to do.
  - nitrogen 11y ago
    Instead it has to do with the ability of large groups to keep big secrets. That's a pretty hard thing to do. Honestly, I wonder if this is a myth started by a large group that was keeping a lot of secrets. It seems like it would be pretty easy to keep secrets under wraps by compartmentalizing knowledge, emphasizing and rewarding loyalty, and intimidating or marginalizing potential whistleblowers.
  - PhasmaFelis 11y ago
    Part of it is also that announcing this sort of thing online doesn't do much unless it goes viral, and going viral seems to have a lot to do with random luck. There's so many horrible things happening in the world that it's difficult to pay attention to more than a few of them.
goatforce5 11y ago
Agreed. I haven't been following the story at all closely, but if VW was able to get (wildly?) better figures than competing companies, I would have thought those other companies would be buying VWs and testing them and/or pulling them apart to figure out how they were doing it... And at some point much earlier than today those other companies would have realized the numbers didn't add up.
- mzs 11y ago
  Mazda engineers for example thought that there was something in the atomization they could still improve cause that is what they guessed VW did better which now proves to have been a snipe hunt sadly.
- CydeWeys 11y ago
  To be clear, it's not that VW was able to get wildly better figures than competing companies, it's that, in one engine map, they get NOx emissions levels that are around the same as other companies (i.e. it meets emissions criteria). Then, in another engine map, when it only uses when it suspects it isn't being tested for emissions, it runs a different fuel/air mixture that allows increased economy (which might only be a 10% improvement or whatever), but at the expense of severely increased emissions. Other companies running the same sort of emission map that optimizes for fuel efficiency at the expense of all emissions would get similar figures of 40X for emitting way too much emission. VW is the main one we're aware of that's cheating in this manner.
tkinom 11y ago
I too am curious on how it really happen. If one burn 1 gallon of diesel get x amount of NOx, wouldn't 40x NOx need to burn 40x amount of diesel? What does the SW do that? Dial down the power of the engine to 1/40th of "normal operations" during the testing?
- tkinom 11y ago
  Nevermind, got the answer here: http://spectrum.ieee.org/cars-that-think/transportation/advanced-cars/how-professors-caught-vw-cheating http://spectrum.ieee.org/cars-that-think/transportation/adva...
  - x0x0 11y ago
    from the article [Dan] Carder [director of West Virginia University’s Center for Alternative Fuels, Engines and Emissions (CAFEE)] did, however, shed light on diesel exhaust. The problem, he says, is one of thermal management. Catalytic converters break down NOX only when they’re hot, and keeping them hot may sometimes require that you burn more fuel than you need for the performance you want. You might want to run the engine on a lean fuel-air mixture, but you have to run it rich. “There’s also a trap for NOX that acts like a sponge,” Carder points out. “But if you saturate it with NOX, then you have to go to a rich-burning condition to use the catalyst to reduce the NOX” and thus wring out the sponge.
- mzs 11y ago
  No, but good question. NOx production varies wildly on factors such as atomization of fuel, ratio vs stochiaometric in mix, temperature of combustion...
- ubercore 11y ago
  I don't know if it directly answers your question, but NOx emissions are dependent on combustion temperature I believe, so the ECU could potentially adjust fuel and timings to create conditions that don't spew NOx, but don't perform as well either. Then when on the road, it opens back up to increase performance but liberate more of the NOx that otherwise might be bound up in less harmful substances or caught by the particulate filter/catalytic converter.
x0x0 11y ago
My understanding is that diesel cars had to be manufactured with either a NOx storage catalyst or injectable urea, both designed to convert the NOx emissions into something less harmful. What VM did seems to be the equivalent of just removing the catalytic converter. It beggars belief that the ceo and many executives were unaware. Considering that, in order to make emissions standards, competitors have to add hardware to the emissions system or inject urea, which increases costs and hassle to the driver (you have to fill the AdBlue, every piece of hardware is a maintenance cost waiting to happen) and to VW (plumbing, harder to convince drivers to buy diesel.) VW figured out a way to avoid both, a distinct advantage to their diesel products. And nobody in the executive team inquired at all how this engineering miracle they pulled off worked? I don't believe it. If nothing else, someone in the executive team asked how long it would take their competitors to copy VW's innovation. Or suggested patenting it.
- mzs 11y ago
  It is possible that CEO was unaware. The line was that VW engineers had solved it for small engines with better CDI and engine/transmission management. Theoretically it could be enough and they did patent that stuff.
hackuser 11y ago
> The secret was kept for so long. How many programmers were involved with the relevant code? How many project managers signed off on it? Surely VW does its own emissions testing internally so some of them must have known. How high up in management did this go? It seems like it must have been quite a few. Amazing that none of them got mad and told someone outside the company. We should not be surprised; we need to learn to expect it and manage that risk. Institutions of all types have long histories of such conspiracies to cover illicit activity. Think of the many recent financial industry scandals, the Catholic Church (and Penn State University) covering widespread rape of children, various military scandals, US government spying on citizens, IT companies spying on their users, performance enhancing drugs in baseball ... in fact we should be surprised when someone steps forward. The #1 rule of any institution is loyalty. Rape will be covered for; blowing the whistle is a mortal sin. It shouldn't be that way but we can't deal with these problems until we accept that it's human nature, at least in certain situations. EDIT: I don't mean to preach. A certain part of me is always surprised too.
- harryh 11y ago
  You are clearly correct and you're probably right that I underestimate the power of loyalty. Just something in my lizard brain remains surprised. Defect in my circuitry I suppose.
- notvladputin 11y ago
  The used of PEDs in baseball was a well known issue dating back to the 1980s. Players, front office types, writers, and anyone in the public paying attention knew about it. I can't comment on the other items, but it's pretty clear that everyone knew about PEDs but just didn't care enough to ban them. (That said, my personal opinion is that steroid testing done in baseball is much more about using the issue as a wedge to weaken the MLB Players Association, rather than "fixing" the game.)
  - hackuser 11y ago
    I don't agree: Back then I was a very knowledgable baseball fan (I probably could have named almost every player on every team) and I didn't know. I heard occasional rumors but nothing more persuasive than that, and nothing at all which indicated, in any convincing way, how widespread it was. When Jose Conseco talked about it, he was ridiculed as a nutcase and shunned (as expected; see my comment above about loyalty). I remember when home runs and offense increased dramatically, to record levels, and everyone was saying that the ball was juiced - nobody suspected it was the players who were juiced.
Uhhrrr 11y ago
For 1), it could be that one set of engineers was tasked with gathering emissions data and finding which parameters got the legal result, another set wrote the software which gathered the data to determine the run profile, and another was tasked with setting the parameters in software for a given run profile. And, the internal emissions testers are probably told to replicate the various government trials. It could be probably done with just one or two directors knowing the whole story. Reminds me of the denouement of "Real Genius": https://www.youtube.com/watch?v=HoT-h0S1gkE https://www.youtube.com/watch?v=HoT-h0S1gkE
- harryh 11y ago
  Excellent movie reference! And ya, that's a reasonable hypothesis. I wonder if we'll ever find out what really happened. Probably not. :(
  - Uhhrrr 11y ago
    I guarantee* that multiple books will be written about this. We'll probably find out some pieces of what happened. *insofar as a pseudonymous Internet commenter can guarantee anything
- greggman 11y ago
  What was actually built http://fas.org/spp/starwars/program/news99/991001-news-madden.htm http://fas.org/spp/starwars/program/news99/991001-news-madde... Sorry there are other references. Search for TRW + laser + San Clemente
  - Uhhrrr 11y ago
    Interesting that I can't easily find articles about it after 2001.
usrusr 11y ago
Organizations keep those secrets by rationalizing that their behavior is not illegal at all. "This might be against the spirit of the law, but hey, here is this clever loophole that we found." This way, what could have been a perfectly whistle-blowable secret is just a dirty little implementation detail that won't be shared because the people involved are not particularly proud of it. What makes this pattern so powerful is that its persuasiveness does not correlate much with the legal quality of the rationalization: as long as the rationalization is not routinely tested in court, people will happily believe just about anything if it makes their job easier. Doubly so in a corporate environment where optimism is rewarded and pessimism is punished. Nonetheless it has been very much knowable, for years if not decades, that some level of benchmark-gaming is happening in the car industry. But that's an inconvenient truth to anyone who is living a car centric life, so even people outside of the car industry proper were likely to not make much of it when a car engineer had a spell of loose lips. I am convinced that it is neither a coincidence nor a conspiracy that only a study initiated by a quite car-critical organization could break that wall of motivated disinterest. Re 40x: a clean combustion combines all the inputs into mostly harmless outputs. Inert gases that have shed all chemically bound energy and won't react much with the molecules that make us alive (unfortunately, we don't know how to perfection that chemical efficiency without sacrificing thermal efficiency). Those mostly harmless gases might still be bad (think CO2), but are not terribly bad as in directly poisonous. In an imperfect combustion, all that stuff can combine into much more reactive and thus dangerous gases. If you set your regulatory threshold sufficiently close to perfection, even a very slight imbalance in the combustion can cause "orders of magnitude beyond threshold". It's the same mathematical principle that makes "many nines" uptime guarantees such a beast: if you fail, you will most probably fail quite hard.

by harryh 11y ago

HNaTTY 11y ago

From what I've read so far, this has to do with heat necessary for the catalytic converter to absorb NOx -- without enough heat, NOx isn't absorbed, so in emissions test mode, they're just running the engine in such a way that more heat is generated to make sure the cat works properly.

harryh 11y ago
Interesting! How much heat are we talking about here? When an engine is just running in normal operation it generates a fair amount of heat naturally. I guess that's not enough?
- ams6110 11y ago
  Diesel engines run with a surplus of air so they only burn the fuel they need to. They don't really generate a lot of heat unless they are under load (relative to gas engines). Just cruising down the highway a diesel will not need much fuel and will run fairly cool.
  - durin42 11y ago
    The owners manual for the VW TDIs actually warns you to not try and preheat the engine by letting it idle in the winter, because it won't work.
mzs 11y ago
Fuel economy improves at higher temperatures, but NOx production increases dramatically then. Conventional catalytic systems cannot keep-up so either a selective catalyst (expensive) or improved egr system (performance sapping) must be used.

teddyh 11y ago

Programmers are susceptible to venality just like anybody else. http://www.commitstrip.com/en/2015/09/22/software-worth-billions/ http://www.commitstrip.com/en/2015/09/22/software-worth-bill...

harryh 11y ago
Of course. My surprise has nothing to do with my opinion of programmers vs other professions. Instead it has to do with the ability of large groups to keep big secrets. That's a pretty hard thing to do.
- nitrogen 11y ago
  Instead it has to do with the ability of large groups to keep big secrets. That's a pretty hard thing to do. Honestly, I wonder if this is a myth started by a large group that was keeping a lot of secrets. It seems like it would be pretty easy to keep secrets under wraps by compartmentalizing knowledge, emphasizing and rewarding loyalty, and intimidating or marginalizing potential whistleblowers.
- PhasmaFelis 11y ago
  Part of it is also that announcing this sort of thing online doesn't do much unless it goes viral, and going viral seems to have a lot to do with random luck. There's so many horrible things happening in the world that it's difficult to pay attention to more than a few of them.

goatforce5 11y ago

Agreed. I haven't been following the story at all closely, but if VW was able to get (wildly?) better figures than competing companies, I would have thought those other companies would be buying VWs and testing them and/or pulling them apart to figure out how they were doing it... And at some point much earlier than today those other companies would have realized the numbers didn't add up.

mzs 11y ago
Mazda engineers for example thought that there was something in the atomization they could still improve cause that is what they guessed VW did better which now proves to have been a snipe hunt sadly.
CydeWeys 11y ago
To be clear, it's not that VW was able to get wildly better figures than competing companies, it's that, in one engine map, they get NOx emissions levels that are around the same as other companies (i.e. it meets emissions criteria). Then, in another engine map, when it only uses when it suspects it isn't being tested for emissions, it runs a different fuel/air mixture that allows increased economy (which might only be a 10% improvement or whatever), but at the expense of severely increased emissions. Other companies running the same sort of emission map that optimizes for fuel efficiency at the expense of all emissions would get similar figures of 40X for emitting way too much emission. VW is the main one we're aware of that's cheating in this manner.

tkinom 11y ago

I too am curious on how it really happen. If one burn 1 gallon of diesel get x amount of NOx, wouldn't 40x NOx need to burn 40x amount of diesel? What does the SW do that? Dial down the power of the engine to 1/40th of "normal operations" during the testing?

tkinom 11y ago
Nevermind, got the answer here: http://spectrum.ieee.org/cars-that-think/transportation/advanced-cars/how-professors-caught-vw-cheating http://spectrum.ieee.org/cars-that-think/transportation/adva...
- x0x0 11y ago
  from the article [Dan] Carder [director of West Virginia University’s Center for Alternative Fuels, Engines and Emissions (CAFEE)] did, however, shed light on diesel exhaust. The problem, he says, is one of thermal management. Catalytic converters break down NOX only when they’re hot, and keeping them hot may sometimes require that you burn more fuel than you need for the performance you want. You might want to run the engine on a lean fuel-air mixture, but you have to run it rich. “There’s also a trap for NOX that acts like a sponge,” Carder points out. “But if you saturate it with NOX, then you have to go to a rich-burning condition to use the catalyst to reduce the NOX” and thus wring out the sponge.
mzs 11y ago
No, but good question. NOx production varies wildly on factors such as atomization of fuel, ratio vs stochiaometric in mix, temperature of combustion...
ubercore 11y ago
I don't know if it directly answers your question, but NOx emissions are dependent on combustion temperature I believe, so the ECU could potentially adjust fuel and timings to create conditions that don't spew NOx, but don't perform as well either. Then when on the road, it opens back up to increase performance but liberate more of the NOx that otherwise might be bound up in less harmful substances or caught by the particulate filter/catalytic converter.

x0x0 11y ago

My understanding is that diesel cars had to be manufactured with either a NOx storage catalyst or injectable urea, both designed to convert the NOx emissions into something less harmful. What VM did seems to be the equivalent of just removing the catalytic converter. It beggars belief that the ceo and many executives were unaware. Considering that, in order to make emissions standards, competitors have to add hardware to the emissions system or inject urea, which increases costs and hassle to the driver (you have to fill the AdBlue, every piece of hardware is a maintenance cost waiting to happen) and to VW (plumbing, harder to convince drivers to buy diesel.) VW figured out a way to avoid both, a distinct advantage to their diesel products. And nobody in the executive team inquired at all how this engineering miracle they pulled off worked? I don't believe it. If nothing else, someone in the executive team asked how long it would take their competitors to copy VW's innovation. Or suggested patenting it.

mzs 11y ago
It is possible that CEO was unaware. The line was that VW engineers had solved it for small engines with better CDI and engine/transmission management. Theoretically it could be enough and they did patent that stuff.

hackuser 11y ago

> The secret was kept for so long. How many programmers were involved with the relevant code? How many project managers signed off on it? Surely VW does its own emissions testing internally so some of them must have known. How high up in management did this go? It seems like it must have been quite a few. Amazing that none of them got mad and told someone outside the company. We should not be surprised; we need to learn to expect it and manage that risk. Institutions of all types have long histories of such conspiracies to cover illicit activity. Think of the many recent financial industry scandals, the Catholic Church (and Penn State University) covering widespread rape of children, various military scandals, US government spying on citizens, IT companies spying on their users, performance enhancing drugs in baseball ... in fact we should be surprised when someone steps forward. The #1 rule of any institution is loyalty. Rape will be covered for; blowing the whistle is a mortal sin. It shouldn't be that way but we can't deal with these problems until we accept that it's human nature, at least in certain situations. EDIT: I don't mean to preach. A certain part of me is always surprised too.

harryh 11y ago
You are clearly correct and you're probably right that I underestimate the power of loyalty. Just something in my lizard brain remains surprised. Defect in my circuitry I suppose.
notvladputin 11y ago
The used of PEDs in baseball was a well known issue dating back to the 1980s. Players, front office types, writers, and anyone in the public paying attention knew about it. I can't comment on the other items, but it's pretty clear that everyone knew about PEDs but just didn't care enough to ban them. (That said, my personal opinion is that steroid testing done in baseball is much more about using the issue as a wedge to weaken the MLB Players Association, rather than "fixing" the game.)
- hackuser 11y ago
  I don't agree: Back then I was a very knowledgable baseball fan (I probably could have named almost every player on every team) and I didn't know. I heard occasional rumors but nothing more persuasive than that, and nothing at all which indicated, in any convincing way, how widespread it was. When Jose Conseco talked about it, he was ridiculed as a nutcase and shunned (as expected; see my comment above about loyalty). I remember when home runs and offense increased dramatically, to record levels, and everyone was saying that the ball was juiced - nobody suspected it was the players who were juiced.

Uhhrrr 11y ago

For 1), it could be that one set of engineers was tasked with gathering emissions data and finding which parameters got the legal result, another set wrote the software which gathered the data to determine the run profile, and another was tasked with setting the parameters in software for a given run profile. And, the internal emissions testers are probably told to replicate the various government trials. It could be probably done with just one or two directors knowing the whole story. Reminds me of the denouement of "Real Genius": https://www.youtube.com/watch?v=HoT-h0S1gkE https://www.youtube.com/watch?v=HoT-h0S1gkE

harryh 11y ago
Excellent movie reference! And ya, that's a reasonable hypothesis. I wonder if we'll ever find out what really happened. Probably not. :(
- Uhhrrr 11y ago
  I guarantee* that multiple books will be written about this. We'll probably find out some pieces of what happened. *insofar as a pseudonymous Internet commenter can guarantee anything
greggman 11y ago
What was actually built http://fas.org/spp/starwars/program/news99/991001-news-madden.htm http://fas.org/spp/starwars/program/news99/991001-news-madde... Sorry there are other references. Search for TRW + laser + San Clemente
- Uhhrrr 11y ago
  Interesting that I can't easily find articles about it after 2001.

usrusr 11y ago

Organizations keep those secrets by rationalizing that their behavior is not illegal at all. "This might be against the spirit of the law, but hey, here is this clever loophole that we found." This way, what could have been a perfectly whistle-blowable secret is just a dirty little implementation detail that won't be shared because the people involved are not particularly proud of it. What makes this pattern so powerful is that its persuasiveness does not correlate much with the legal quality of the rationalization: as long as the rationalization is not routinely tested in court, people will happily believe just about anything if it makes their job easier. Doubly so in a corporate environment where optimism is rewarded and pessimism is punished. Nonetheless it has been very much knowable, for years if not decades, that some level of benchmark-gaming is happening in the car industry. But that's an inconvenient truth to anyone who is living a car centric life, so even people outside of the car industry proper were likely to not make much of it when a car engineer had a spell of loose lips. I am convinced that it is neither a coincidence nor a conspiracy that only a study initiated by a quite car-critical organization could break that wall of motivated disinterest. Re 40x: a clean combustion combines all the inputs into mostly harmless outputs. Inert gases that have shed all chemically bound energy and won't react much with the molecules that make us alive (unfortunately, we don't know how to perfection that chemical efficiency without sacrificing thermal efficiency). Those mostly harmless gases might still be bad (think CO2), but are not terribly bad as in directly poisonous. In an imperfect combustion, all that stuff can combine into much more reactive and thus dangerous gases. If you set your regulatory threshold sufficiently close to perfection, even a very slight imbalance in the combustion can cause "orders of magnitude beyond threshold". It's the same mathematical principle that makes "many nines" uptime guarantees such a beast: if you fail, you will most probably fail quite hard.