9 ms·
The article reminds me of a story from a friend that was an engineer at a major water heater company. Modern electric water heaters are already fairly efficient
by wscott 6y ago
The article reminds me of a story from a friend that was an engineer at a major water heater company. Modern electric water heaters are already fairly efficient, but every model gets tested by the government to be given an efficiency rating.
The test consists of replacing the anode with a rod that has a series of temperature probes to measure the water at different levels in the water heater. Since water would form layers of different temperatures you needed measure at multiple locations to know the overall energy in the water. Then a series of water draws are performed with pauses between them and the total energy used is compared to the energy delivered.
Our engineer realized that if he could put the temperature layers at just the right locations then the calculations with the limited number of probes could be skewed. And this can be done by tweaking the height of the 2 heating elements. Just moving them a few inches.
The first time he tried this, an internal test came back at 101% efficiency. At little more tweaking and the test returns a consistent 99.x% that was respectable.
- ryanianian 6y agoIsn't this a symptom of the exact testing procedure being a white box that rarely changes? Many students would score 100% on the ACT if the entire question and answer set were widely published. Also that seems like a silly test. Couldn't you just measure the differential after mixing? TFA also mentions it, but noise is also a factor. For the water-heater example, the limited number of probes introduces a known amount of potential variance. Rather than a straight % efficiency, a more useful number to report would include the confidence interval - Its 99% ± 3%. Consumers can't do stats but at least it's more honest. TFA touches on this but doesn't draw any conclusions other than "we need to look at the source data and do our own analysis".
- chickenpotpie 6y agoAlso https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- bananamerica 6y agoHow did I know this linked to some kind of general principle with a cool short name even before reading or opening the link?
- dredmorbius 6y agoWell, there are several such: https://en.wikipedia.org/wiki/List_of_eponymous_laws https://en.wikipedia.org/wiki/List_of_eponymous_laws Though I'm not aware of a law stating that for any given principle there is an existing eponymous law.
- esja 6y agoHaven’t you heard of DredMorbius’s Law of Eponymity?
- dredmorbius 6y agoBut bananamerica really discovered it. ... oh, wait: https://en.wikipedia.org/wiki/Stigler%27s_law_of_eponymy https://en.wikipedia.org/wiki/Stigler%27s_law_of_eponymy
- AnthonyMouse 6y agoThe patent system in a nutshell, ladies and gentlemen.
- bananamerica 6y agoSomething tells me there’s a XKCD for that, or at least it should be.
- colechristensen 6y agoIt means the test procedure has vulnerabilities which need to be patched.
- dharmab 6y agoThis phenomenon also appears in the testing of crash helmets. The DOT (US government) helmet standard is easy to game since it's very strict on how and where the helmets are tested. The Snell Foundation standard is a bit better since the humans testing the helmets are allowed to look for weak points to target for anvil drops. The new FIM standard adds more variability to the tests to better simulate the varying angles of crashes. https://youtu.be/uLj9WfoWPSQ?t=413 https://youtu.be/uLj9WfoWPSQ?t=413
- Bartweiss 6y agoCrash test dummies have basically this problem also. They're designed for realism in certain very narrow ways, and then the very small number of approved dummies are used for testing car safety. The industry has made a bit of progress, surprisingly unprompted by regulations - female and child dummies came into circulation before they were required in tests. But overall, testing is still run against a tiny handful of body types which move 'realistically' in only a few regulation-guided respects.
- craftinator 6y agoI think some of this falls into the simulation paradox: the more accurate the simulation, the closer the simulation is to the thing being modelled. But it's a quadratic relationship in most cases, so at some point meaningful increases in simulation accuracy cease to be economically viable.
- dharmab 6y agoYeah, but in the words of RyanF9, "The US government can afford a BB gun", so there's no reason that DOT can't test helmet visors. The main reason the DOT standard is so bad is because its mired in bureaucracy and managed by a severely underfunded organization.
- cybrjoe 6y agoSerious question, how is this different from what VW did? There was a lot of talk about complicit engineers vs. shady management in that discussion. Did this ever make it to production?
- henryfjordan 6y agoVW actively detected if the car was on a treadmill testing setup and electronically changed the engine parameters to make it run cleaner. The water heater was designed with the test in mind but functions the same regardless. VW was clearly cheating whereas the water heater is taking advantage of known deficiencies in the test (is that cheating?).
- ivanbakel 6y agoWhat was more heinous about the VW case IIRC is that the cleanliness gains the cars made on the treadmill setup were enough to push them into compliance with regulations, which the engines normally didn't meet. "Cheating" a better efficiency rating is significantly less reprehensible than cheating a legal obligation which was legislated for public health and environmental reasons.
- ebg13 6y agoBut the efficiency ratings also exist for public health and environmental reasons. Why does it matter to you that one is a mandate? They're both intentionally deceiving regulators and consumers (vs accidentally acing the test without cheating). Both are fraud.
- ReactiveJelly 6y agoOne is polluting at the tailpipe, the other is only polluting at a power plant.
- Bartweiss 6y agoI do think that manipulating a purely instructive measure is less extreme than manipulating a compliance test; consumers can seek alternate tests and reviews, but the state emissions test has special status even if a dozen other tests give a different result. That said, I believe Energy Star ratings affect tax rebates and electric bills, and they're required to be printed on products - so that's not really an arbitrary test. There are other differences here too, I think. The water heater trick is passive manipulation that stays in place at all times, which limits how far from "real" performance it can get. And per the story, it seems more like "teaching to the test" than "cheating". That is, Volkswagen consciously moved away from the mandate outside of testing. The water heater was (potentially) as energy-efficient as they could design, with the test score manipulated on top of that. None of that makes it harmless - if "as good as you can make" doesn't hit standards without manipulating them, that's still a problem. But I do find it less galling than "intentionally worsens emissions outside the test bench".
- apcragg 6y agoIs that not outright fraud? Just because you can hack something doesn't make it right or legal.
- bravoetch 6y agoThe system incentives are based on outcomes, not inputs. People are not benefiting from pointing out flaws in the system. They are benefiting from exploiting the flaws.
- balfirevic 6y agoHow far out of optimal (for testing) position would you require their water temperature layers to be?
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- sangfroid_bio 6y agoThere was a scandal a few years back when a Kaggle team did something simar with ML. They were treating the competition as a black box and optimising for the unknown dataset instead of actually building better AI algorithms.
- vngzs 6y agoThis? https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436 https://www.kaggle.com/c/petfinder-adoption-prediction/discu...
- sangfroid_bio 6y agoIt goes back way further than that, the incidents that I am recalling involved academic teams and I think ImageNet. It happened quite a few times.
- t_serpico 6y agoOut of curiosity, why would the water form layers of different temperatures?
- Gibbon1 6y agoI think because with a modern water heater especially electric models the heat loss is so low convection often stops completely. So the water in the tank doesn't mix. I think you can get situations too where the water at the level of the heating elements gets hotter and hotter and the water away from it gets cold. My water heater strongly suggests adding a mixing valve for that reason.
- jodrellblank 6y ago> "water at the level of the heating elements gets hotter and hotter and the water away from it gets cold." Eh? How come that temperature differential doesn't cause convection? The colder water being more dense and sinking down, the warmer water being less dense and rising up. If the heating elements aren't at the bottom, specifically for this reason, how come they aren't?
- dmurray 6y ago> If the heating elements aren't at the bottom, specifically for this reason, how come they aren't? There are typically two heating elements, one near the bottom and one further up. You can use the higher one if you only need a half-tank of hot water. I'm not sure if this is the complete answer to your other question, or if sometimes conditions are such that convection occurs extremely slowly despite a substantial temperature difference.
- Gibbon1 6y agoI think perversely it's not the density profile in the vertical dimension that drives convection but the horizontal dimension. A hotter thus lower density but perfectly flat homogeneous layer can't displace the colder denser layer above it because the pressure between the two layers is uniform.
- AmericanChopper 6y agoReminds me of the Ferrari fuel injection controversy in F1 last season. The gist of it is the theory that Ferrari were cheating the maximum fuel injection rates, by tuning their injection system to inject more fuel when the injection sensor wasn’t taking a reading. https://www.motorsport.com/f1/news/analysis-fia-settlement-ferrari-engine/4703298/ https://www.motorsport.com/f1/news/analysis-fia-settlement-f...
- iamgopal 6y agoI've maintained that water heater is the only equipment that is 100 percent efficient. Glad to know that is not the case.