4 ms·

From a sociology point of view, yes you should distrust any signs of p-hacking or selective reporting, especially when there is a low effect size. From a techn

From a sociology point of view, yes you should distrust any signs of p-hacking or selective reporting, especially when there is a low effect size. From a technology point of view, I think we are already well past "no one.. really knowing what is going on." Take the current study. Past researchers had identified the CD19 antigen found (often exclusively) on B cells, and others had found that autologous T-cells could persist in a patient for many years, potentially providing a durable response. The current researchers built on this knowledge to design and engineer T-cells that could wipe out a patient's B cells. So it's a very planned, targeted approach, more akin to engineering than the old approach of testing thousands of random molecules as potential treatments. When the targeted approach works, and especially when it has a large and consistent effect on patients as in this case, then you can be very confident at an early stage. This needs to be recognized and medical practice needs to change as a result. We're entering an era of incredibly detailed knowledge and precision tools for research and development. The pace of development will continue to increase and treatments will become more patient-specific as individual targets are identified. You could consider the large, slow, blunt clinical trials as being like optimization for a particular treatment. When new drugs are rare or not well understood it makes sense to spend time to optimize their use. But when new drugs come frequently and are well targeted this premature optimization costs lives.

nonbel 11y ago
>"Past researchers had identified the CD19 antigen found (often exclusively) on B cells, and others had found that autologous T-cells could persist in a patient for many years, potentially providing a durable response." I didn't watch the video, so I'm not sure where this info is coming from, but providing such references is one reason why they should write a paper up before talking to the media. Also, I am fairly certain if we do follow up on these claims (I know nothing about them specifically) we will find the actual evidence is much more ambiguous, there has been no direct replication attempted, the references do not actually contain the evidence claimed, etc.
- nonbel 11y ago
  See the problem? I can claim anything I want about this, be it optimistic or pessimistic, and no one can check. There is no paper (or equivalent source of information), so no way to assess the evidence.
  - tominous 11y ago
    One way to approach an announcement like this is to wait until everything is wrapped up nicely in a neat paper. Another approach is to investigate further using the tools available: look up the authors' past work on Google Scholar, check clinicaltrials.gov, look for third party opinions, even ask the authors directly. I'd argue that the scientists and journalists at the AAAS conference where this was presented are capable of this second approach, and so are many participants on HN.
    - nonbel 11y ago
      I think if you try it out, after a few you'll find this is naive. Further, you are doing the job of the person making the claim! It is too time consuming to research a claim indirectly like that. That is what you do after reading the paper which contains (usually most of) the required information and (usually a biased sample of) citations to the other research that were found relevant. Reading the paper is only a first filtering step to evaluating a claim, a conference presentation simply doesn't merit that effort because it contains far too little information. A conference presentation is more like "hey, check out what I'm working on". Now, that doesn't mean people don't try to do this, it is one of the primary uses for misinterpretation of p-values.

nonbel 11y ago

>"Past researchers had identified the CD19 antigen found (often exclusively) on B cells, and others had found that autologous T-cells could persist in a patient for many years, potentially providing a durable response." I didn't watch the video, so I'm not sure where this info is coming from, but providing such references is one reason why they should write a paper up before talking to the media. Also, I am fairly certain if we do follow up on these claims (I know nothing about them specifically) we will find the actual evidence is much more ambiguous, there has been no direct replication attempted, the references do not actually contain the evidence claimed, etc.

nonbel 11y ago
See the problem? I can claim anything I want about this, be it optimistic or pessimistic, and no one can check. There is no paper (or equivalent source of information), so no way to assess the evidence.
- tominous 11y ago
  One way to approach an announcement like this is to wait until everything is wrapped up nicely in a neat paper. Another approach is to investigate further using the tools available: look up the authors' past work on Google Scholar, check clinicaltrials.gov, look for third party opinions, even ask the authors directly. I'd argue that the scientists and journalists at the AAAS conference where this was presented are capable of this second approach, and so are many participants on HN.
  - nonbel 11y ago
    I think if you try it out, after a few you'll find this is naive. Further, you are doing the job of the person making the claim! It is too time consuming to research a claim indirectly like that. That is what you do after reading the paper which contains (usually most of) the required information and (usually a biased sample of) citations to the other research that were found relevant. Reading the paper is only a first filtering step to evaluating a claim, a conference presentation simply doesn't merit that effort because it contains far too little information. A conference presentation is more like "hey, check out what I'm working on". Now, that doesn't mean people don't try to do this, it is one of the primary uses for misinterpretation of p-values.