3 ms·

The simpler, cheaper, and faster you make the test... the more quickly it will be defeated by the adulterators. Much like spam filters and SEO, leaking anything

by kurthr 4y ago

The simpler, cheaper, and faster you make the test... the more quickly it will be defeated by the adulterators. Much like spam filters and SEO, leaking anything about how you defend against an attack just makes the attackers marginally improve their game. Simple amine/nitrogen tests for protein is how we got poisonous melamine in fake milk and wheat gluten killing babies, dogs, and cats. It comes down to Goodhart's law.

canadiantim 4y ago
That’s not entirely true. The domain within which the cheating can occur shrinks by eg checking certain properties, tho the problem you’re talking about happens when only a few properties are checked because then of course only those properties need to be gamed/manipulated. It’s about how much coverage your tests have over the properties of what you’re testing
lostlogin 4y ago
I agree that the test becomes the target in many things, but how do you suggest applying that to food safety? If easy tests are out, you go to a very complex test immediately?
- kurthr 4y ago
  You test at least moderately close to the actual target and you punish those who are breaking the rules brutally (it is actually food safety!). Instead we allow Olive oil that isn't and Fish that aren't because, nobody has died yet? Allergies are possible even at very low levels. My real point wasn't the simplicity or cheapness, but often the biproduct of those is that it's a test you already have that doesn't measure what you want (nitrogen rather than protein). In the case of Olive oil, if what we cared about was it's fluoresced color rather than the material in it, that would be great. If what we care about is components, random GC/IR spectroscopy (perhaps after centrifuging) to see the actual compounds with consequences would be a better choice than the cheapest thing they can just add another weird chemical to defeat.
  - lostlogin 4y ago
    Thanks. A funny version of this is Manuka honey. It's not easy to define and it's slowly turned into an NZ versus Australia thing. There have been examples where bees that only gave Manuka to feed on are making honey that doesn't meet the standard, and bees that weren't thought to be feeding on Manuka have been making Manuka honey. I have seen recipes for making non compliant honey into compliant honey (to be clear, the input honey is Manuka, but the lab test wasn't being being passed). It's about blending various types of honey. It's a bit dumb in my view and needs a better test. As you say, it's meeting the test, not the objective of the test.

by kurthr 4y ago

canadiantim 4y ago

That’s not entirely true. The domain within which the cheating can occur shrinks by eg checking certain properties, tho the problem you’re talking about happens when only a few properties are checked because then of course only those properties need to be gamed/manipulated. It’s about how much coverage your tests have over the properties of what you’re testing

lostlogin 4y ago

I agree that the test becomes the target in many things, but how do you suggest applying that to food safety? If easy tests are out, you go to a very complex test immediately?

kurthr 4y ago
You test at least moderately close to the actual target and you punish those who are breaking the rules brutally (it is actually food safety!). Instead we allow Olive oil that isn't and Fish that aren't because, nobody has died yet? Allergies are possible even at very low levels. My real point wasn't the simplicity or cheapness, but often the biproduct of those is that it's a test you already have that doesn't measure what you want (nitrogen rather than protein). In the case of Olive oil, if what we cared about was it's fluoresced color rather than the material in it, that would be great. If what we care about is components, random GC/IR spectroscopy (perhaps after centrifuging) to see the actual compounds with consequences would be a better choice than the cheapest thing they can just add another weird chemical to defeat.
- lostlogin 4y ago
  Thanks. A funny version of this is Manuka honey. It's not easy to define and it's slowly turned into an NZ versus Australia thing. There have been examples where bees that only gave Manuka to feed on are making honey that doesn't meet the standard, and bees that weren't thought to be feeding on Manuka have been making Manuka honey. I have seen recipes for making non compliant honey into compliant honey (to be clear, the input honey is Manuka, but the lab test wasn't being being passed). It's about blending various types of honey. It's a bit dumb in my view and needs a better test. As you say, it's meeting the test, not the objective of the test.