5 ms·

80% of current drug applications in China failed to meet analysis requirements

127 points by ting_bu_dung 10y ago

Odenwaelder 10y ago
In chinese culture, losing one's "face" is one of the worst things that can happen to a person. I'm wondering whether the fact that 80% of data has been fabricated is a result of that. As in, "I have to make my findings significant to not lose my face", or, "Presenting negative findings to my superior will make me lose my face". Maybe someone more familiar with this culture could elaborate.
- ravenstine 10y ago
  Very interesting analysis. If this is accurate, they should get rid of that attitude, pronto, as they have the highest potential of being on the scientific forefront(or at least that of medical science).
- finid 10y ago
  Wait a minute! That seems to be backward. Doesn't "losing face" come from being caught doing the wrong thing or being dishonest, so that the incentive to fabricate data for something as important as scientific research should be very, very low?
  - Odenwaelder 10y ago
    You might be right, depending on the culture's definition of "wrong". Let's hope you're right...
  - HillRat 10y ago
    I'm not sure that reputational risk is the same thing; I suspect one way to look at it is that a researcher who returns the null case is disrupting the organization by putting a lot of other peoples' work and assurances at risk. So there would be a significant amount of implicit social pressure to show results and maintain organizational harmony.
    - beeftime 10y ago
      having lived and worked in China for a few years, I can confirm you're dead on the money.
    - 794CD01 10y ago
      That sounds like academia anywhere.
      - finid 10y ago
        Yep, publish or perish applies to academia everywhere.
  - psyc 10y ago
    No, because losing face is not really about morality or honesty in the first place - it's more important than honesty. The primary thing here is the sharp and pervasive divide between facade and reality. The facade is more important. There are a lot of cultures where fronting is the main thing. It's just particularly evident in China.
    - travmatt 10y ago
      Also in Japan, as even a nuclear disaster isn't impetus enough to forego 'saving face': On July 5, an independent investigative commission established by the Japanese Diet issued its final report on the accident at the Fukushima Daiichi Nuclear Power Station. Hailed as the definitive word on the subject thus far, the report points to what it calls the "fundamental causes" of the disaster, all of them cultural. The chairman's message in the report assails "the ingrained conventions of Japanese culture: our reflexive obedience; our reluctance to question authority; our devotion to 'sticking with the program'; our groupism; and our insularity." The nuclear disaster, in short, was "a disaster 'Made in Japan.'" News media around the world characterized the report's damning indictment of Japanese culture as unusually candid for the nation known to do anything to save face.[http://thebulletin.org/japan%E2%80%99s-culture-culprit-nuclear-accident http://thebulletin.org/japan%E2%80%99s-culture-culprit-nucle...]
  - mike_hock 10y ago
    Being caught is the keyword.
- radicsge 10y ago
  It's not about the face. It was clearly written there "local pharmaceutical companies trying to produce Western drugs struggle to turn a profit"... What about Theranos in the US? Same.. Isn't it because the turn everything into profit within two years "success or nothing" attitude? If something ruins the world, it will be two countries that under the surface is way too similar to each other.
brudgers 10y ago
Radio Free Asia (RFA) is a private, nonprofit international broadcasting agency of the United States government[2] that broadcasts and publishes online news, information, and commentary to listeners in East Asia while "advancing the goals of U.S. foreign policy." https://en.wikipedia.org/wiki/Radio_Free_Asia https://en.wikipedia.org/wiki/Radio_Free_Asia The source cited is Radio Free Asia not the report itself. Radio Free Asia does not link to any additional sources. That is not to say that the story is inaccurate, but rather that the claims are not well substantiated and that the source is not apolitical.
- eveningcoffee 10y ago
  This is good catch, but considering the history of such programs (https://en.wikipedia.org/wiki/Radio_Free_Europe/Radio_Liberty https://en.wikipedia.org/wiki/Radio_Free_Europe/Radio_Libert...), it is that they publish rather information that has been suspended by local governments than something that has been made up. Of course you should be cautious with what they publish, but this applies to every published information.
  - brudgers 10y ago
    From the RFA story: data gathered during clinical trials were incomplete, failed to meet analysis requirements or were untraceable, the paper cited a source in the agency as saying. Outside of China, clinical data is often not shared, suppressed when it does not support the claims, and p-value hacking is used to justify 'off-label' use. This is not usually spun as fabricated, and in commercial drug development my intuition is that 80% of proposed drugs having at least one of these features would be a plausible hypothesis
tlb 10y ago
It mentions 1622 current clinical trials. Of those, how many would be expected to be approved in a rational system? I wonder if the huge %age is due to a lot of low-quality applications mixed in with the serious ones. If the barrier to apply is sufficiently low, you would expect a Sturgeon's law dynamic.
- Alex3917 10y ago
  > I wonder if the huge %age is due to a lot of low-quality applications mixed in with the serious ones. I doubt it. Most clinical trial data in the U.S. is similarly fraudulent, but there isn't really any concept of high-quality vs low-quality applications like you might see with YC or whatever. Instead, pharma companies tend to rely on the same set of 20 or 30 tricks to take chemical compounds that might be no better than placebos and make them look like wonder drugs. I've been keeping a tally of these tricks on my website: http://www.alexkrupp.com/Citevault.html#pharmaceuticals http://www.alexkrupp.com/Citevault.html#pharmaceuticals
eveningcoffee 10y ago
Considering past occurrences https://en.wikipedia.org/wiki/2008_Chinese_milk_scandal https://en.wikipedia.org/wiki/2008_Chinese_milk_scandal and general behaviour in online market places (my last experience was that the product was advertised to be shipped from Germany (expect fast delivery), but arrived a month later from China) then it should not come as very big surprise (I am not saying that this is normal or acceptable, but that it is not very surprising). My question is that what is the cause of this general fraudulent behaviour? Does it have some historical roots? Does it come from lack of actual functioning oversight? Another anecdotal case: try asking for directions in China. Even when person does not know the correct directions, they do not mind to direct you, and when confronted, they refuse to admit that they do not know. Is this mindset result of constant fear of repressions, or it runs deeper into the history of the society (say it is also present in Taiwan?). I am not accusing anybody in anything, I just would like to understand.
- _yvjs 10y ago
  There was an interesting study looking at measures of honesty and economic growth: https://www.uea.ac.uk/documents/3154295/7054672/Honesty+paper/41fecf09-235e-45c1-afc2-b872ea0ac882 https://www.uea.ac.uk/documents/3154295/7054672/Honesty+pape.... Chinese nationals were in the top 2 of both measures of dishonesty (which didn't correlate all that well in general, so I'm not sure "honesty" is all that easy to measure), while British nationals were in the bottom 2 of both. One of the correlations they found was that rates of honesty were correlated with rates of Protestant Christianity.
  - rootkit55 10y ago
    It would be interesting to see results of that game played for a million dollars. You go into a room alone and flip a coin. If you flip heads you win the million dollars. If you flip tails you win nothing. It's also interesting that in situations like the Wells Fargo scandal, honesty didn't even matter. If you were honest, you were simply fired.
- asddddd 10y ago
  > general behaviour in online market places (my last experience was that the product was advertised to be shipped from Germany (expect fast delivery), but arrived a month later from China) Sounds more like a drop shipper reselling stuff from Aliexpress or similar, not necessarily anything to do with a Chinese seller. They add a hefty markup and ship directly to you. It's particularly popular with counterfeits ("replicas").
DanielBMarkham 10y ago
Science is having an astounding crisis -- I don't think there's been anything like it in history before. If this is true about Chinese clinical data? More generally, if there is any simple tag that covers fraud at this scale? Other people working under that tag will experience deleterious effects for years to come. The only way I see out of this is a rigorous, self-imposed body of standards. Even then I'm not sure that this will be sufficient. Think of all those people, all of that money, all of the hundreds of thousands of hours of research. For nothing. Actually, it's worse than nothing because it's been flooding the system with fake data.
- Odenwaelder 10y ago
  It's not only about fake data, it's also, maybe even most importantly, the low quality data. Publications with low data quality decrease the signal-to-noise ratio, leading to a slower scientific progress. Purely from a scientific standpoint: If you identify fake data, you just take them out of your model training set. But identifying low quality data is much harder.
  - DanielBMarkham 10y ago
    Agreed. Poor quality data is worse. A couple of years ago, as this story started to break, I would mention it to scientist friends. "Don't worry," they'd say, "We always have the meta analysis to fall back on." I don't hear that anymore. ADD: A critical issue here, which you touch on tangentially, is the mix of motivation and milieu. If I'm making stuff up -- bad data -- other people can identify it and delete it. But if my goal is to appear to be doing good science instead of actually doing it, then it can become extremely problematic both to identify and remedy what I've done. A lot of "normal" science is done around the margins, with not-so-incredible hypotheses and pedestrian-looking data. Bad science with really poor data fits seamlessly into that model without having any distinguishing characteristics.
  - AlexCoventry 10y ago
    It's reminiscent of the role played by the Sophons in Cixin Liu's The Three-Body Problem, except in this case it's self-inflicted. https://en.wikipedia.org/wiki/The_Three-Body_Problem#First_Contacts https://en.wikipedia.org/wiki/The_Three-Body_Problem#First_C...
ksk 10y ago
Title should make it clear that this article is about currently pending applications, not about all clinical trials.
- sctb 10y ago
  Thanks, we've updated the title.
projektfu 10y ago
In the couple years I spent doing biology research, about 8 years ago, I was skeptical of most research from China because it already had a reputation already of being untrustworthy. But why stop at China? I spent a good part of a year trying and failing to replicate an English study, and looking around for materials referenced in another study that simply don't exist, finally being told by the principal on that study that they mischaracterized the materials in the paper. Sometimes when you get the results you are hoping for, you publish, and you don't care if they were wrong, right? Perhaps it would be better if scientists generally took a more skeptical view of others' publications and tried to replicate them in house. At conferences, instead of asking, "Where do we go from here", with the first study, ask, "What do I need to do to replicate this?"
- nonbel 10y ago
  I had a very similar experience. And you are correct that these problems are not at all limited to "China", although those authors tend to be a little less sophisticated in their (what may generously be called) shortcuts. For example, if you do a real literature review on a medical topic (not just credulously take their word for it) you are almost sure to find that one of the primary assumptions/claims was actually never studied. One paper refs a second which refs a third that speculated about it in the introduction, but now "everyone knows it".
  - projektfu 10y ago
    So true. This is especially important as people study ways of improving metrics of markers of disease instead of actually improving disease endpoints. For example, there is a supplement marketed in the veterinary space that may lower blood urea nitrogen in uremic animals, according to a study. Whoop-de-do! I don't care about urea, it doesn't cause the disease of uremia. It is just a marker. You could convert all the urea to ammonia and now have a patient with two diseases and a BUN of 0.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
msie 10y ago
Ting bu dung...
- jza00425 10y ago
  lol, after checking comments and stories of this guy, what a deep love to China
bombless 10y ago
fake news
canada_dry 10y ago
No wonder wealthy Chinese go out of their way to buy not-made-in-China food and medicine.
nytimesn 10y ago
Saints vs Chargers live stream free https://www.squadup.com/events/free-saints-vs-chargers-live-stream-nfl-gamecast-on-any-device https://www.squadup.com/events/free-saints-vs-chargers-live-...
- ljf 10y ago
  More spam?

127 points by ting_bu_dung 10y ago

In chinese culture, losing one's "face" is one of the worst things that can happen to a person. I'm wondering whether the fact that 80% of data has been fabricated is a result of that. As in, "I have to make my findings significant to not lose my face", or, "Presenting negative findings to my superior will make me lose my face". Maybe someone more familiar with this culture could elaborate.

ravenstine 10y ago
Very interesting analysis. If this is accurate, they should get rid of that attitude, pronto, as they have the highest potential of being on the scientific forefront(or at least that of medical science).
finid 10y ago
Wait a minute! That seems to be backward. Doesn't "losing face" come from being caught doing the wrong thing or being dishonest, so that the incentive to fabricate data for something as important as scientific research should be very, very low?
- Odenwaelder 10y ago
  You might be right, depending on the culture's definition of "wrong". Let's hope you're right...
- HillRat 10y ago
  I'm not sure that reputational risk is the same thing; I suspect one way to look at it is that a researcher who returns the null case is disrupting the organization by putting a lot of other peoples' work and assurances at risk. So there would be a significant amount of implicit social pressure to show results and maintain organizational harmony.
  - beeftime 10y ago
    having lived and worked in China for a few years, I can confirm you're dead on the money.
  - 794CD01 10y ago
    That sounds like academia anywhere.
    - finid 10y ago
      Yep, publish or perish applies to academia everywhere.
- psyc 10y ago
  No, because losing face is not really about morality or honesty in the first place - it's more important than honesty. The primary thing here is the sharp and pervasive divide between facade and reality. The facade is more important. There are a lot of cultures where fronting is the main thing. It's just particularly evident in China.
  - travmatt 10y ago
    Also in Japan, as even a nuclear disaster isn't impetus enough to forego 'saving face': On July 5, an independent investigative commission established by the Japanese Diet issued its final report on the accident at the Fukushima Daiichi Nuclear Power Station. Hailed as the definitive word on the subject thus far, the report points to what it calls the "fundamental causes" of the disaster, all of them cultural. The chairman's message in the report assails "the ingrained conventions of Japanese culture: our reflexive obedience; our reluctance to question authority; our devotion to 'sticking with the program'; our groupism; and our insularity." The nuclear disaster, in short, was "a disaster 'Made in Japan.'" News media around the world characterized the report's damning indictment of Japanese culture as unusually candid for the nation known to do anything to save face.[http://thebulletin.org/japan%E2%80%99s-culture-culprit-nuclear-accident http://thebulletin.org/japan%E2%80%99s-culture-culprit-nucle...]
- mike_hock 10y ago
  Being caught is the keyword.
radicsge 10y ago
It's not about the face. It was clearly written there "local pharmaceutical companies trying to produce Western drugs struggle to turn a profit"... What about Theranos in the US? Same.. Isn't it because the turn everything into profit within two years "success or nothing" attitude? If something ruins the world, it will be two countries that under the surface is way too similar to each other.

brudgers 10y ago

Radio Free Asia (RFA) is a private, nonprofit international broadcasting agency of the United States government[2] that broadcasts and publishes online news, information, and commentary to listeners in East Asia while "advancing the goals of U.S. foreign policy." https://en.wikipedia.org/wiki/Radio_Free_Asia https://en.wikipedia.org/wiki/Radio_Free_Asia The source cited is Radio Free Asia not the report itself. Radio Free Asia does not link to any additional sources. That is not to say that the story is inaccurate, but rather that the claims are not well substantiated and that the source is not apolitical.

eveningcoffee 10y ago
This is good catch, but considering the history of such programs (https://en.wikipedia.org/wiki/Radio_Free_Europe/Radio_Liberty https://en.wikipedia.org/wiki/Radio_Free_Europe/Radio_Libert...), it is that they publish rather information that has been suspended by local governments than something that has been made up. Of course you should be cautious with what they publish, but this applies to every published information.
- brudgers 10y ago
  From the RFA story: data gathered during clinical trials were incomplete, failed to meet analysis requirements or were untraceable, the paper cited a source in the agency as saying. Outside of China, clinical data is often not shared, suppressed when it does not support the claims, and p-value hacking is used to justify 'off-label' use. This is not usually spun as fabricated, and in commercial drug development my intuition is that 80% of proposed drugs having at least one of these features would be a plausible hypothesis

tlb 10y ago

It mentions 1622 current clinical trials. Of those, how many would be expected to be approved in a rational system? I wonder if the huge %age is due to a lot of low-quality applications mixed in with the serious ones. If the barrier to apply is sufficiently low, you would expect a Sturgeon's law dynamic.

Alex3917 10y ago
> I wonder if the huge %age is due to a lot of low-quality applications mixed in with the serious ones. I doubt it. Most clinical trial data in the U.S. is similarly fraudulent, but there isn't really any concept of high-quality vs low-quality applications like you might see with YC or whatever. Instead, pharma companies tend to rely on the same set of 20 or 30 tricks to take chemical compounds that might be no better than placebos and make them look like wonder drugs. I've been keeping a tally of these tricks on my website: http://www.alexkrupp.com/Citevault.html#pharmaceuticals http://www.alexkrupp.com/Citevault.html#pharmaceuticals

eveningcoffee 10y ago

Considering past occurrences https://en.wikipedia.org/wiki/2008_Chinese_milk_scandal https://en.wikipedia.org/wiki/2008_Chinese_milk_scandal and general behaviour in online market places (my last experience was that the product was advertised to be shipped from Germany (expect fast delivery), but arrived a month later from China) then it should not come as very big surprise (I am not saying that this is normal or acceptable, but that it is not very surprising). My question is that what is the cause of this general fraudulent behaviour? Does it have some historical roots? Does it come from lack of actual functioning oversight? Another anecdotal case: try asking for directions in China. Even when person does not know the correct directions, they do not mind to direct you, and when confronted, they refuse to admit that they do not know. Is this mindset result of constant fear of repressions, or it runs deeper into the history of the society (say it is also present in Taiwan?). I am not accusing anybody in anything, I just would like to understand.

_yvjs 10y ago
There was an interesting study looking at measures of honesty and economic growth: https://www.uea.ac.uk/documents/3154295/7054672/Honesty+paper/41fecf09-235e-45c1-afc2-b872ea0ac882 https://www.uea.ac.uk/documents/3154295/7054672/Honesty+pape.... Chinese nationals were in the top 2 of both measures of dishonesty (which didn't correlate all that well in general, so I'm not sure "honesty" is all that easy to measure), while British nationals were in the bottom 2 of both. One of the correlations they found was that rates of honesty were correlated with rates of Protestant Christianity.
- rootkit55 10y ago
  It would be interesting to see results of that game played for a million dollars. You go into a room alone and flip a coin. If you flip heads you win the million dollars. If you flip tails you win nothing. It's also interesting that in situations like the Wells Fargo scandal, honesty didn't even matter. If you were honest, you were simply fired.
asddddd 10y ago
> general behaviour in online market places (my last experience was that the product was advertised to be shipped from Germany (expect fast delivery), but arrived a month later from China) Sounds more like a drop shipper reselling stuff from Aliexpress or similar, not necessarily anything to do with a Chinese seller. They add a hefty markup and ship directly to you. It's particularly popular with counterfeits ("replicas").

DanielBMarkham 10y ago

Science is having an astounding crisis -- I don't think there's been anything like it in history before. If this is true about Chinese clinical data? More generally, if there is any simple tag that covers fraud at this scale? Other people working under that tag will experience deleterious effects for years to come. The only way I see out of this is a rigorous, self-imposed body of standards. Even then I'm not sure that this will be sufficient. Think of all those people, all of that money, all of the hundreds of thousands of hours of research. For nothing. Actually, it's worse than nothing because it's been flooding the system with fake data.

Odenwaelder 10y ago
It's not only about fake data, it's also, maybe even most importantly, the low quality data. Publications with low data quality decrease the signal-to-noise ratio, leading to a slower scientific progress. Purely from a scientific standpoint: If you identify fake data, you just take them out of your model training set. But identifying low quality data is much harder.
- DanielBMarkham 10y ago
  Agreed. Poor quality data is worse. A couple of years ago, as this story started to break, I would mention it to scientist friends. "Don't worry," they'd say, "We always have the meta analysis to fall back on." I don't hear that anymore. ADD: A critical issue here, which you touch on tangentially, is the mix of motivation and milieu. If I'm making stuff up -- bad data -- other people can identify it and delete it. But if my goal is to appear to be doing good science instead of actually doing it, then it can become extremely problematic both to identify and remedy what I've done. A lot of "normal" science is done around the margins, with not-so-incredible hypotheses and pedestrian-looking data. Bad science with really poor data fits seamlessly into that model without having any distinguishing characteristics.
- AlexCoventry 10y ago
  It's reminiscent of the role played by the Sophons in Cixin Liu's The Three-Body Problem, except in this case it's self-inflicted. https://en.wikipedia.org/wiki/The_Three-Body_Problem#First_Contacts https://en.wikipedia.org/wiki/The_Three-Body_Problem#First_C...

ksk 10y ago

Title should make it clear that this article is about currently pending applications, not about all clinical trials.

sctb 10y ago
Thanks, we've updated the title.

projektfu 10y ago

In the couple years I spent doing biology research, about 8 years ago, I was skeptical of most research from China because it already had a reputation already of being untrustworthy. But why stop at China? I spent a good part of a year trying and failing to replicate an English study, and looking around for materials referenced in another study that simply don't exist, finally being told by the principal on that study that they mischaracterized the materials in the paper. Sometimes when you get the results you are hoping for, you publish, and you don't care if they were wrong, right? Perhaps it would be better if scientists generally took a more skeptical view of others' publications and tried to replicate them in house. At conferences, instead of asking, "Where do we go from here", with the first study, ask, "What do I need to do to replicate this?"

nonbel 10y ago
I had a very similar experience. And you are correct that these problems are not at all limited to "China", although those authors tend to be a little less sophisticated in their (what may generously be called) shortcuts. For example, if you do a real literature review on a medical topic (not just credulously take their word for it) you are almost sure to find that one of the primary assumptions/claims was actually never studied. One paper refs a second which refs a third that speculated about it in the introduction, but now "everyone knows it".
- projektfu 10y ago
  So true. This is especially important as people study ways of improving metrics of markers of disease instead of actually improving disease endpoints. For example, there is a supplement marketed in the veterinary space that may lower blood urea nitrogen in uremic animals, according to a study. Whoop-de-do! I don't care about urea, it doesn't cause the disease of uremia. It is just a marker. You could convert all the urea to ammonia and now have a patient with two diseases and a BUN of 0.
- deleted 10y ago
  [deleted]
- deleted 10y ago

msie 10y ago

Ting bu dung...

jza00425 10y ago
lol, after checking comments and stories of this guy, what a deep love to China

bombless 10y ago

fake news

canada_dry 10y ago

No wonder wealthy Chinese go out of their way to buy not-made-in-China food and medicine.

nytimesn 10y ago

Saints vs Chargers live stream free https://www.squadup.com/events/free-saints-vs-chargers-live-stream-nfl-gamecast-on-any-device https://www.squadup.com/events/free-saints-vs-chargers-live-...

ljf 10y ago
More spam?