4 ms·
Many of the comments in this thread follow a similar pattern. They are all essentially derivatives of Goodhart's law. [0] >> Once a metric becomes a target, it
by daniel-levin 11y ago
Many of the comments in this thread follow a similar pattern. They are all essentially derivatives of Goodhart's law. [0]
>> Once a metric becomes a target, it ceases to be a good measure.
It's quite useful to have a name - a label - for this sort of behaviour. In this case, the means by which VW optimised for this metric was crafty computer software! A metric is only as good as the cost of the unintended consequences that come with imposing it on people, in particular large corporations.
I am not surprised that this is done by many corporations as mentioned elsewhere in the comments (ATI/Nvidia etc). This is because regulations and benchmarks that can be gamed, such as engine efficiency, will be gamed because the incentives of corporations work out that way:
(I hope this isn't perceived as an anti-corporate rant because it's not. Just a way of framing facts).
For regulations imposed on large corporations in general (think EPA admissions standards):
1) Regulations are mandated by a government to effect some purpose (stop destroying Earth)
2) Executives - the people running the corporations - are bound by legal responsibility - their fiduciary duty [1] - to create value for their shareholders (selling lots of cars, and profitably).
3) Regulations can obstruct the activities of a corporation by making them do things they don't want to (making efficient cars is hard and costly and affects the bottom line - its cheaper to manufacture a car with less platinum [2] in it)
4) Companies are incentivized to minimize the time and money spent on adhering to regulations
5) Once a regulation is standardised and codified, it becomes possible to follow the letter of it, while ignoring the spirit of it. All a corporation has to do is meet the metrics in the specification to abide by the regulation - in other words - game the system.
Given these incentives, we have seen that corporations prefer to abide by regulations in the maximally efficient way: gaming them. Thus reducing a well-meaning government initiative to gamesmanship.
Kudos to the EPA for calling VW out on this. This kind of incident reinforces (in me at least) the idea that environmental legislation is an extraordinarily difficult thing to get right.
[0] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
[1] https://www.law.cornell.edu/wex/fiduciary_duty https://www.law.cornell.edu/wex/fiduciary_duty
[2] Catalytic convertors
- zardo 11y ago>This kind of incident reinforces (in me at least) the idea that environmental legislation is an extraordinarily difficult thing to get right. And the more detailed you make the regulation to prevent gaming, the harder it is for new competitors to enter the industry. Also good god having read and interpet their frequently self contradicting regs is a nightmare. Hell, Tesla got hit with an EPA penalty early on because they didn't certify the roadster's non-existant emissions.
- maxmax 11y agoIn this case, VW didn't game the regulation. They gamed the test conditions. They deliberately violated the regulation. Violations should be expected on occasion. Violators should expect to be punished when caught.
- brc 11y agoTo me the tests have to be real measurements rather than a proxy. Having a special test makes the test easy to pass. If the test was instead to have sampled data from x,000 cars, gathered at service intervals fit within certain bands, then that would require actually engineering most cars to return within the set range across a wide range of use types. Smaller niche manufacturers could just pass a simpler test to encourage market entrants. 200 slightly worse cars is nothing against 200,000 intentionally bad cars. The regulations need to target what is trying to be achieved rather than simplified proxies.