11 ms·

There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial. This should be an async non blo

There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial. This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust. There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc). Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures). There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming. There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem. And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group. Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences. Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.

alphazard 3y ago
The success of the Nutrition Facts labeling does not get enough publicity. Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep. Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously. Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.
- taeric 3y ago
  This isn't without downsides, of course. The case of manufacturers adding allergens to food deliberately is alarming in its own way.
  - hermitdev 3y ago
    Surely, you aren't complaining of peanuts in peanut butter, so can you share an example, and why the allergen presumably shouldn't be there?
    - taeric 3y ago
      Apologies, I should have included a link. I am referring to https://apnews.com/article/sesame-allergies-label-b28f8eb3dc846f2a19d87b03440848f1 https://apnews.com/article/sesame-allergies-label-b28f8eb3dc.... Mayhap that is overblown? I confess I have not followed it too heavily. Very thankful that I am not allergic to anything in my adult life.
      - hermitdev 3y ago
        I don't have any food allergies, either, so I admittedly don't pay much attention to allergens (or even listed ingredients most of the time). I'm genuinely curious, too.
    - swsieber 3y ago
      I had the same questions and found this article: https://snacksafely.com/2016/06/kelloggs-unintended-consequences-and-the-death-of-may-contain/ https://snacksafely.com/2016/06/kelloggs-unintended-conseque... Basically, there were stricter measures put in place called HARPC. From my linked article: > The new directives mandate that the “Top 8 allergens” identified by FALCPA (peanuts, tree nuts, milk, eggs, wheat, soy, fish, and crustacean shellfish) must either be ingredients of the product and identified as such, or the manufacturer must take extra care (and cost) to ensure that there is no cross-contact with them. There is no middle ground or “out” for the manufacturer, which is why we believe “May contain” type label advisories are heading for extinction. And that poses a problem, at least in the short term. > companies, when faced with the added burden of instituting and documenting cross-contact prevention measures as dictated by HARPC, may instead choose to add trace amounts of the allergen to the product, as doing so makes the allergen an ingredient of the product and obviates the need for preventative cross-contact measures for that allergen. > it means that manufacturers will either take stricter measures to prevent cross-contact or add a trace amount of the allergen and list it in the ingredient list, thus eliminating the ambiguity that currently plagues us all. > A compromise that might have avoided the unintended consequences of companies like Kellogg’s adding traces of allergens to their products is to have offered them a third option: A mandatory “May contain” label advisory for any product made on shared equipment or in shared facilities that did not meet the FSMA threshold for cross-contact prevention. Such label advisories are voluntary today, rendering them ambiguous at best, but a definitively worded and located advisory statement included on all such products would have provided a way for manufacturers to meet the requirements of HARPC without resorting to the addition of allergens. So I'd be surprised if it was happening on an ongoing basis, but I can definitely see why people would be irked.
deleted 3y ago
[deleted]
mike_hearn 3y ago
There's the Cochrane Collaboration. They don't tick off every item on your list but it's fairly close to what you're asking for. It's mentioned in the article as they do a lot of meta-studies. Unfortunately they only started trying to spot fraudulent RCTs in 2021. Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID. Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6] The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4] https://en.wikipedia.org/wiki/Cochrane_(organisation) https://en.wikipedia.org/wiki/Cochrane_(organisation)
- camelite 3y ago
  "Many commentators have claimed that a recently-updated Cochrane Review shows that 'masks don't work', which is an inaccurate and misleading interpretation." https://www.cochrane.org/news/statement-physical-interventions-interrupt-or-reduce-spread-respiratory-viruses-review https://www.cochrane.org/news/statement-physical-interventio...
  - mike_hearn 3y ago
    Next sentence: "It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive." ... which is what I just said: some people got mad at them because their review found no reliable evidence that masking worked (or rather, that mask mandates worked, but these are virtually the same thing). The null hypothesis for any medical intervention is that it has no effect. You start from that and then try to prove your hypothesis that it does have an effect, which is what medical studies are for. If you can't prove something works then we fall back to the null and assume it doesn't. So that isn't a misleading or inaccurate interpretation of the results, though it would certainly have been politically convenient for the Cochrane organization if their reviewers could have supported the claims of public health authorities.
p-e-w 3y ago
> There should be Nutrition Facts but for scientific trials. No, there should be prison time for scientists who conduct unethical trials or publish fake results. The public (and policy makers) place such immense trust in those people and what they publish that nothing less is even remotely adequate. When someone puts arsenic in food, they go to prison – labeling the food with "contains arsenic" doesn't cut it. Do this and watch science magically fix itself.
- ekianjo 3y ago
  Revoke their license and titles to start with so they cannot operate anymore in the field
  - Spinnaker_ 3y ago
    I would like to see Universities take the lead here. A bunch of high profile degree revocations would generate some waves at least.
    - tcmart14 3y ago
      As reasonable as that would be. I feel like it would just turn into people getting their degrees revoke claiming 'cancel culture' and becoming gurus with many mindless followers pushing loose weight quick schemes kind of thing. Because today if you face consequences, its no longer your fault, its everyone else trying to cancel you from doing the bad thing you are doing.
- twic 3y ago
  Which prison? As the article says: > Ultimately, a lingering question is — as with paper mills — why so many suspect RCTs are being produced in the first place. Mol, from his experiences investigating the Egyptian studies, blames lack of oversight and superficial assessments that promote academics on the basis of their number of publications, as well as the lack of stringent checks from institutions and journals on bad practices. A substantial part of what's happening here is that first-world countries with generally good cultures of research integrity are basing medical policy on studies done in countries where the system encourages researchers to cheat. British and US authorities can't put Egyptian or Chinese researchers in prison, can they?
jonlucc 3y ago
Why would the statisticians be anonymous? I'm aware of at least a couple cases in which an independent set of statisticians were provided the data from a clinical trial specifically for a re-analysis. In one case, they showed some pretty concerning inconsistencies and the other confirmed no effect on the primary analysis, but suggested some sub-populations that might have shown an effect if a future study was properly powered. That follow-up clinical trial was just published showing pretty remarkable effect in the sub-population. I don't think there's reason to believe either independent analysis was anything other than independent. There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive. Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.
- obblekk 3y ago
  So junior scientists can be hired without them being concerned for future career prospects.
s1artibartfast 3y ago
Im struggling to see the difference between this and the current FDA process, and think it is 90% the same. Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take. This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust. Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods >There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc). Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale. >Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures). Labeling contains drug specific information. There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming. Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem. The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2] And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group. Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold. Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences. This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency. https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20895s039s042lbl.pdf https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20... https://www.fda.gov/media/72253/download https://www.fda.gov/media/72253/download https://climedo.de/en/blog/list-of-mdr-certified-notified-bodies/ https://climedo.de/en/blog/list-of-mdr-certified-notified-bo...
csours 3y ago
Yes, I have said this as "The UX of [medical] study papers is terrible". Some people do not agree, they think that it should not be made easier to understand, that non-experts cannot really understand medical studies, so they should not be more approachable. I think that's dead wrong.

alphazard 3y ago

The success of the Nutrition Facts labeling does not get enough publicity. Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep. Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously. Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.

taeric 3y ago
This isn't without downsides, of course. The case of manufacturers adding allergens to food deliberately is alarming in its own way.
- hermitdev 3y ago
  Surely, you aren't complaining of peanuts in peanut butter, so can you share an example, and why the allergen presumably shouldn't be there?
  - taeric 3y ago
    Apologies, I should have included a link. I am referring to https://apnews.com/article/sesame-allergies-label-b28f8eb3dc846f2a19d87b03440848f1 https://apnews.com/article/sesame-allergies-label-b28f8eb3dc.... Mayhap that is overblown? I confess I have not followed it too heavily. Very thankful that I am not allergic to anything in my adult life.
    - hermitdev 3y ago
      I don't have any food allergies, either, so I admittedly don't pay much attention to allergens (or even listed ingredients most of the time). I'm genuinely curious, too.
  - swsieber 3y ago
    I had the same questions and found this article: https://snacksafely.com/2016/06/kelloggs-unintended-consequences-and-the-death-of-may-contain/ https://snacksafely.com/2016/06/kelloggs-unintended-conseque... Basically, there were stricter measures put in place called HARPC. From my linked article: > The new directives mandate that the “Top 8 allergens” identified by FALCPA (peanuts, tree nuts, milk, eggs, wheat, soy, fish, and crustacean shellfish) must either be ingredients of the product and identified as such, or the manufacturer must take extra care (and cost) to ensure that there is no cross-contact with them. There is no middle ground or “out” for the manufacturer, which is why we believe “May contain” type label advisories are heading for extinction. And that poses a problem, at least in the short term. > companies, when faced with the added burden of instituting and documenting cross-contact prevention measures as dictated by HARPC, may instead choose to add trace amounts of the allergen to the product, as doing so makes the allergen an ingredient of the product and obviates the need for preventative cross-contact measures for that allergen. > it means that manufacturers will either take stricter measures to prevent cross-contact or add a trace amount of the allergen and list it in the ingredient list, thus eliminating the ambiguity that currently plagues us all. > A compromise that might have avoided the unintended consequences of companies like Kellogg’s adding traces of allergens to their products is to have offered them a third option: A mandatory “May contain” label advisory for any product made on shared equipment or in shared facilities that did not meet the FSMA threshold for cross-contact prevention. Such label advisories are voluntary today, rendering them ambiguous at best, but a definitively worded and located advisory statement included on all such products would have provided a way for manufacturers to meet the requirements of HARPC without resorting to the addition of allergens. So I'd be surprised if it was happening on an ongoing basis, but I can definitely see why people would be irked.

deleted 3y ago

[deleted]

mike_hearn 3y ago

There's the Cochrane Collaboration. They don't tick off every item on your list but it's fairly close to what you're asking for. It's mentioned in the article as they do a lot of meta-studies. Unfortunately they only started trying to spot fraudulent RCTs in 2021. Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID. Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6] The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4] https://en.wikipedia.org/wiki/Cochrane_(organisation) https://en.wikipedia.org/wiki/Cochrane_(organisation)

camelite 3y ago
"Many commentators have claimed that a recently-updated Cochrane Review shows that 'masks don't work', which is an inaccurate and misleading interpretation." https://www.cochrane.org/news/statement-physical-interventions-interrupt-or-reduce-spread-respiratory-viruses-review https://www.cochrane.org/news/statement-physical-interventio...
- mike_hearn 3y ago
  Next sentence: "It would be accurate to say that the review examined whether interventions to promote mask wearing help to slow the spread of respiratory viruses, and that the results were inconclusive." ... which is what I just said: some people got mad at them because their review found no reliable evidence that masking worked (or rather, that mask mandates worked, but these are virtually the same thing). The null hypothesis for any medical intervention is that it has no effect. You start from that and then try to prove your hypothesis that it does have an effect, which is what medical studies are for. If you can't prove something works then we fall back to the null and assume it doesn't. So that isn't a misleading or inaccurate interpretation of the results, though it would certainly have been politically convenient for the Cochrane organization if their reviewers could have supported the claims of public health authorities.

p-e-w 3y ago

> There should be Nutrition Facts but for scientific trials. No, there should be prison time for scientists who conduct unethical trials or publish fake results. The public (and policy makers) place such immense trust in those people and what they publish that nothing less is even remotely adequate. When someone puts arsenic in food, they go to prison – labeling the food with "contains arsenic" doesn't cut it. Do this and watch science magically fix itself.

ekianjo 3y ago
Revoke their license and titles to start with so they cannot operate anymore in the field
- Spinnaker_ 3y ago
  I would like to see Universities take the lead here. A bunch of high profile degree revocations would generate some waves at least.
  - tcmart14 3y ago
    As reasonable as that would be. I feel like it would just turn into people getting their degrees revoke claiming 'cancel culture' and becoming gurus with many mindless followers pushing loose weight quick schemes kind of thing. Because today if you face consequences, its no longer your fault, its everyone else trying to cancel you from doing the bad thing you are doing.
twic 3y ago
Which prison? As the article says: > Ultimately, a lingering question is — as with paper mills — why so many suspect RCTs are being produced in the first place. Mol, from his experiences investigating the Egyptian studies, blames lack of oversight and superficial assessments that promote academics on the basis of their number of publications, as well as the lack of stringent checks from institutions and journals on bad practices. A substantial part of what's happening here is that first-world countries with generally good cultures of research integrity are basing medical policy on studies done in countries where the system encourages researchers to cheat. British and US authorities can't put Egyptian or Chinese researchers in prison, can they?

jonlucc 3y ago

Why would the statisticians be anonymous? I'm aware of at least a couple cases in which an independent set of statisticians were provided the data from a clinical trial specifically for a re-analysis. In one case, they showed some pretty concerning inconsistencies and the other confirmed no effect on the primary analysis, but suggested some sub-populations that might have shown an effect if a future study was properly powered. That follow-up clinical trial was just published showing pretty remarkable effect in the sub-population. I don't think there's reason to believe either independent analysis was anything other than independent. There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive. Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.

obblekk 3y ago
So junior scientists can be hired without them being concerned for future career prospects.

s1artibartfast 3y ago

Im struggling to see the difference between this and the current FDA process, and think it is 90% the same. Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take. This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust. Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods >There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc). Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale. >Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures). Labeling contains drug specific information. There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming. Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem. The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2] And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group. Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold. Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences. This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency. https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20895s039s042lbl.pdf https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20... https://www.fda.gov/media/72253/download https://www.fda.gov/media/72253/download https://climedo.de/en/blog/list-of-mdr-certified-notified-bodies/ https://climedo.de/en/blog/list-of-mdr-certified-notified-bo...

csours 3y ago

Yes, I have said this as "The UX of [medical] study papers is terrible". Some people do not agree, they think that it should not be made easier to understand, that non-experts cannot really understand medical studies, so they should not be more approachable. I think that's dead wrong.