3 ms·

Disclaimer: I work for a competitor. Bottom line: real world use should be tested, but this is expensive and neither side wants to pay for it. Regarding compl

by csours 11y ago

Disclaimer: I work for a competitor. Bottom line: real world use should be tested, but this is expensive and neither side wants to pay for it. Regarding compliance department complicity, the biggest factor in my mind is two-fold: You test for compliance with regulations and you have limited time. You ensure that the internal test parameters match the external regulatory test. You don't have funding to do real world tests. You don't have funding to do static code analysis. You believe the documents you are given, as they have been signed by responsible adults. You use the tools you are given, because they have been developed by professionals. Regarding what competitors knew and when: I haven't seen a discussion of exactly how much better these engines were than competitors. Regarding the scandal in general: The public has been demanding better and better automotive safety and quality represented by government regulations, voting with dollars, and lawsuits. It takes time for these lessons to be internalized, and different corporations and internal groups do this at different speeds. If you are the largest automaker in the world it is kind of natural to think that everything must be ok! (See also: GM, Toyota)

jakejake 11y ago
I can imagine this as not totally different from software dev and QA. Many times the development team puts in some feature that doesn't have any visible output for the testers to test - rather it is some internal change or improvement. The QA department, then responsible for testing this new "feature" will most likely go off of a script that was sent to them by the developers to verify the old and new behavior, going on some metric that perhaps they don't really understand. In some cases, the QA may actually have to rely on a tool actually built by the developers to test the feature. So in some cases the QA, not being software developers and not responsible for actually reviewing the code, will have to somewhat go on faith that the scripts and tools provided by the developers are accurate. I'm not saying that means the engineers were responsible at VW, but it does explain how something could slip through the checks-and-balances that have been set up.
- jacquesm 11y ago
  > The QA department, then responsible for testing this new "feature" will most likely go off of a script that was sent to them by the developers to verify the old and new behavior, going on some metric that perhaps they don't really understand. That's wrong right there. If you do this properly the people writing the QA test script work from the same spec the programmers used to implement the feature but they are actually different people and the test gets written first. Obviously in smaller shops that won't work but anything the size of newsworthy should be set up like that.
  - jakejake 11y ago
    I definitely agree that's the way it should be. I've seen the reality inside a few large corporations and basically nothing would surprise me.
jacquesm 11y ago
Thank you for this comment (one of the more valuable ones in the whole thread). > Bottom line: real world use should be tested, but this is expensive and neither side wants to pay for it. Agreed, but it would be good for a company the size of VW to at least test one car with a new engine platform in real life conditions especially after major upgrades to the emission platform. If a magazine can afford to test 9 vehicles then you'd think that VW could afford to test at least one. > Regarding compliance department complicity, the biggest factor in my mind is two-fold: You test for compliance with regulations and you have limited time. You ensure that the internal test parameters match the external regulatory test. Ok for the during-the-production-run case but also for the very first time a new platform comes online? Wouldn't you want to know what the engine does in real life rather than just on the test stand? In other words, does the actual real life impact matter so little to companies that have 'green' written all over their brochures that they'd rather not know if they were cheating? Would the compliance department at your employer take this as a wake-up call and at least run a single sample through an actual road test to see how their theories hold up in practice? > You don't have funding to do real world tests. How expensive would real world tests be for those engines that are 'the most suspect' based on the measured values during the simulated cycle? > You believe the documents you are given, as they have been signed by responsible adults. The word 'believe' is not in my vocabulary when I do an audit. I tend to take a very dim view of people that ask me to believe stuff and with the kind of money riding on your average deal I'd be a very sore loser indeed if I ever had to use the defense 'I believed what they said' when reality is found out to be contrary to what I was told. Trust but verify. > Regarding what competitors knew and when: I haven't seen a discussion of exactly how much better these engines were than competitors. Depending on the make and model, there are even some that are worse but they pass the tests as well and apparently - what we know to date - do not contain software that outright detects the test cycle and then runs the engine under a different regime. > Regarding the scandal in general: The public has been demanding better and better automotive safety and quality represented by government regulations, voting with dollars, and lawsuits. It takes time for these lessons to be internalized, and different corporations and internal groups do this at different speeds. If you are the largest automaker in the world it is kind of natural to think that everything must be ok! (See also: GM, Toyota) Yes, that's the kind of thinking that got other manufacturers in trouble already. Surely such a thing would not happen at our company... I've seen this mentality in practice and it always puzzles me. There is no such thing as 'too big' when it comes to stuff like this, in fact, when you're big the multiplier is larger so the incentives to color outside the lines are larger.

by csours 11y ago

jakejake 11y ago

I can imagine this as not totally different from software dev and QA. Many times the development team puts in some feature that doesn't have any visible output for the testers to test - rather it is some internal change or improvement. The QA department, then responsible for testing this new "feature" will most likely go off of a script that was sent to them by the developers to verify the old and new behavior, going on some metric that perhaps they don't really understand. In some cases, the QA may actually have to rely on a tool actually built by the developers to test the feature. So in some cases the QA, not being software developers and not responsible for actually reviewing the code, will have to somewhat go on faith that the scripts and tools provided by the developers are accurate. I'm not saying that means the engineers were responsible at VW, but it does explain how something could slip through the checks-and-balances that have been set up.

jacquesm 11y ago
> The QA department, then responsible for testing this new "feature" will most likely go off of a script that was sent to them by the developers to verify the old and new behavior, going on some metric that perhaps they don't really understand. That's wrong right there. If you do this properly the people writing the QA test script work from the same spec the programmers used to implement the feature but they are actually different people and the test gets written first. Obviously in smaller shops that won't work but anything the size of newsworthy should be set up like that.
- jakejake 11y ago
  I definitely agree that's the way it should be. I've seen the reality inside a few large corporations and basically nothing would surprise me.

jacquesm 11y ago

Thank you for this comment (one of the more valuable ones in the whole thread). > Bottom line: real world use should be tested, but this is expensive and neither side wants to pay for it. Agreed, but it would be good for a company the size of VW to at least test one car with a new engine platform in real life conditions especially after major upgrades to the emission platform. If a magazine can afford to test 9 vehicles then you'd think that VW could afford to test at least one. > Regarding compliance department complicity, the biggest factor in my mind is two-fold: You test for compliance with regulations and you have limited time. You ensure that the internal test parameters match the external regulatory test. Ok for the during-the-production-run case but also for the very first time a new platform comes online? Wouldn't you want to know what the engine does in real life rather than just on the test stand? In other words, does the actual real life impact matter so little to companies that have 'green' written all over their brochures that they'd rather not know if they were cheating? Would the compliance department at your employer take this as a wake-up call and at least run a single sample through an actual road test to see how their theories hold up in practice? > You don't have funding to do real world tests. How expensive would real world tests be for those engines that are 'the most suspect' based on the measured values during the simulated cycle? > You believe the documents you are given, as they have been signed by responsible adults. The word 'believe' is not in my vocabulary when I do an audit. I tend to take a very dim view of people that ask me to believe stuff and with the kind of money riding on your average deal I'd be a very sore loser indeed if I ever had to use the defense 'I believed what they said' when reality is found out to be contrary to what I was told. Trust but verify. > Regarding what competitors knew and when: I haven't seen a discussion of exactly how much better these engines were than competitors. Depending on the make and model, there are even some that are worse but they pass the tests as well and apparently - what we know to date - do not contain software that outright detects the test cycle and then runs the engine under a different regime. > Regarding the scandal in general: The public has been demanding better and better automotive safety and quality represented by government regulations, voting with dollars, and lawsuits. It takes time for these lessons to be internalized, and different corporations and internal groups do this at different speeds. If you are the largest automaker in the world it is kind of natural to think that everything must be ok! (See also: GM, Toyota) Yes, that's the kind of thinking that got other manufacturers in trouble already. Surely such a thing would not happen at our company... I've seen this mentality in practice and it always puzzles me. There is no such thing as 'too big' when it comes to stuff like this, in fact, when you're big the multiplier is larger so the incentives to color outside the lines are larger.