4 ms·

Complete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather se

Complete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather see less data? Or are you volunteering to be test subject #2? Show us how to do it right Dr. M.! In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements. Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator. Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint. https://pubmed.ncbi.nlm.nih.gov/36702864/ https://pubmed.ncbi.nlm.nih.gov/36702864/

grafmax 2y ago
I think social media discussions of science would be better informed by the concept of a ‘hierarchy of evidence’. Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.
- abirch 2y ago
  Here’s a relevant xkcd https://xkcd.com/882/ https://xkcd.com/882/ For the small sample
deleted 2y ago
[deleted]
jrootabega 2y ago
If he said unblinded at some point, it could have been because the study author looked into the cup to determine which substance had been taken too soon. The subject should have had no knowledge of what was taken until the entire 16-month trial was over. We should avoid extreme polarization of our judgments in general. The study deserves some amount of praise for things it did somewhat well (like the method of blinding which is clever, but not applicable to everyone), and criticism for things it did not do well, such as designing your own study methodology for your own mood. That alone will affect the results. Simply RUNNING an experiment can affect your mood because it's interesting (or even maybe frustrating). The subject probably felt pride and satisfaction whenever they used their pill selection technique, which could improve mood on its own. Neither accolades nor complete derision are appropriate, although trying to claim too strong a result from this study is kinda deserving of derision if you claim to be science-minded. The study was well-meaning and displayed cleverness.
- robwwilliams 2y ago
  And that is exactly the point made in the target post by the author. He explicitly raised that criticism himself. Double kudos for self-criticism. You will not find many conventional science publications pointing out: “Shucks, we could have done this a better”.
  - jrootabega 2y ago
    The ancestor post is neither a "Complete injustice" nor "derision" nor an "insult", and it doesn't warrant a hostile mocking reply. Its tone could have been gentler, but it wasn't that bad. And the study doesn't really deserve "accolades", it deserves to be recognized for whatever it does well. Such polarization of tone and vocabulary doesn't accomplish much, and I'll even propose that it actually prevents good things from happening. It is good that the author is aware of, and acknowledges, the problems in the study. What other studies and journals have done wrong doesn't make the author or study more deserving of praise. Also, you asked why he said "unblinded", and I think you now have the answer to that.
    - 2y ago
levocardia 2y ago
Because.... >While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken That is not blinding
- sgc 2y ago
  I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial. Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless. As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.
  - kqr 2y ago
    > As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? It gives a hint of how well the blinding worked.
    - sgc 2y ago
      Yes, that is a good point. But it also causes him to continuously think about indicators to determine what he has taken. He is constantly trying to punch the veil on this "blind" study instead of doing everything he can to avoid that. Are they really the exact same weight? Do they really feel the same in his hand, mouth and throat? Does one have a slightly different taste - after all, no filling process will leave the outside of the capsule 100% free of powder, etc? Very subtle differences could manifest themselves over 16 months.
  - fc417fc802 2y ago
throwup238 2y ago
> Why do you say unblinded? It’s unblinded because the subject is preparing the concoction under study. There is no way they can create a blind experiment if they’re the ones preparing the control. The placebo effect is nothing if not pernicious and cunning, able to exploit even the most subtle psychological signal - like minuscule differences in the amount of powder in a capsule. Blinded studies have independent doctors prepare and dispense the candidate drug so they know whether its the real thing or a placebo, but their patients dont. In double blinded studies, neither the doctor nor the patient have any idea about what they’re getting because a third party prepares the drugs.
eth0up 2y ago
If you ever wonder why some folks with a fair amount of potential and something to offer keep to themselves, this isn't the worst example. I think most people could criticize the carbon out of a corpse if they themselves weren't being criticized into one. If we devolved from apes, maybe apes devolved from piranhas.
- robwwilliams 2y ago
  hilarious

grafmax 2y ago

I think social media discussions of science would be better informed by the concept of a ‘hierarchy of evidence’. Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.

abirch 2y ago
Here’s a relevant xkcd https://xkcd.com/882/ https://xkcd.com/882/ For the small sample

deleted 2y ago

[deleted]

jrootabega 2y ago

If he said unblinded at some point, it could have been because the study author looked into the cup to determine which substance had been taken too soon. The subject should have had no knowledge of what was taken until the entire 16-month trial was over. We should avoid extreme polarization of our judgments in general. The study deserves some amount of praise for things it did somewhat well (like the method of blinding which is clever, but not applicable to everyone), and criticism for things it did not do well, such as designing your own study methodology for your own mood. That alone will affect the results. Simply RUNNING an experiment can affect your mood because it's interesting (or even maybe frustrating). The subject probably felt pride and satisfaction whenever they used their pill selection technique, which could improve mood on its own. Neither accolades nor complete derision are appropriate, although trying to claim too strong a result from this study is kinda deserving of derision if you claim to be science-minded. The study was well-meaning and displayed cleverness.

robwwilliams 2y ago
And that is exactly the point made in the target post by the author. He explicitly raised that criticism himself. Double kudos for self-criticism. You will not find many conventional science publications pointing out: “Shucks, we could have done this a better”.
- jrootabega 2y ago
  The ancestor post is neither a "Complete injustice" nor "derision" nor an "insult", and it doesn't warrant a hostile mocking reply. Its tone could have been gentler, but it wasn't that bad. And the study doesn't really deserve "accolades", it deserves to be recognized for whatever it does well. Such polarization of tone and vocabulary doesn't accomplish much, and I'll even propose that it actually prevents good things from happening. It is good that the author is aware of, and acknowledges, the problems in the study. What other studies and journals have done wrong doesn't make the author or study more deserving of praise. Also, you asked why he said "unblinded", and I think you now have the answer to that.
  - 2y ago

levocardia 2y ago

Because.... >While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken That is not blinding

sgc 2y ago
I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial. Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless. As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.
- kqr 2y ago
  > As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? It gives a hint of how well the blinding worked.
  - sgc 2y ago
    Yes, that is a good point. But it also causes him to continuously think about indicators to determine what he has taken. He is constantly trying to punch the veil on this "blind" study instead of doing everything he can to avoid that. Are they really the exact same weight? Do they really feel the same in his hand, mouth and throat? Does one have a slightly different taste - after all, no filling process will leave the outside of the capsule 100% free of powder, etc? Very subtle differences could manifest themselves over 16 months.
- fc417fc802 2y ago

throwup238 2y ago

> Why do you say unblinded? It’s unblinded because the subject is preparing the concoction under study. There is no way they can create a blind experiment if they’re the ones preparing the control. The placebo effect is nothing if not pernicious and cunning, able to exploit even the most subtle psychological signal - like minuscule differences in the amount of powder in a capsule. Blinded studies have independent doctors prepare and dispense the candidate drug so they know whether its the real thing or a placebo, but their patients dont. In double blinded studies, neither the doctor nor the patient have any idea about what they’re getting because a third party prepares the drugs.

eth0up 2y ago

If you ever wonder why some folks with a fair amount of potential and something to offer keep to themselves, this isn't the worst example. I think most people could criticize the carbon out of a corpse if they themselves weren't being criticized into one. If we devolved from apes, maybe apes devolved from piranhas.

robwwilliams 2y ago
hilarious