5 ms·

This is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting

This is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting this much effort into any medication trial makes it much more likely that you’re going to be incentivized to find effects that don’t exist. I think it’s nice that the author admits that they found nothing, but statistically, worthless drugs show effects in much better-designed trials than this one: it’s basically a coin toss.

episteme 2y ago
You could argue that this N is the only N that matters though.
- malfist 2y ago
  Not too me though. Especially not to me if I'm trying to decide to supplement l-theanine
  - deleted 2y ago
    [deleted]
  - ryandrake 2y ago
    Right. The only N that you can draw any conclusions about is the author himself. So, why even publish it? The "results" are not applicable to anyone reading it. This is the health version of the software industry's "It works on my system!"
    - episteme 2y ago
      Publish it is quite a loaded word here. Almost every blog or post you read is from the point of view of a single person. There's nothing wrong with him putting his experience out into the world.
robwwilliams 2y ago
Complete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather see less data? Or are you volunteering to be test subject #2? Show us how to do it right Dr. M.! In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements. Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator. Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint. https://pubmed.ncbi.nlm.nih.gov/36702864/ https://pubmed.ncbi.nlm.nih.gov/36702864/
- grafmax 2y ago
  I think social media discussions of science would be better informed by the concept of a ‘hierarchy of evidence’. Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.
  - abirch 2y ago
    Here’s a relevant xkcd https://xkcd.com/882/ https://xkcd.com/882/ For the small sample
- deleted 2y ago
  [deleted]
- jrootabega 2y ago
wslh 2y ago
In science, an n=1 experiment isn’t discarded; instead, it adds information that can guide future experiments.
derlvative 2y ago
Noooooo you can’t just run independent experiments you need institutions and phds and bureaucracy and gold plating nooooo
- smohare 2y ago
  It’s not about elitism. It’s that there are so many confounding factors that even a well-informed approach makes such a study comtain very little of value Comments like yours expose a particularly distasteful amount of hubris.
  - derlvative 2y ago
    You don’t like it you don’t have to read the blog article. I assure you are not the intended audience. For the rest of us it provided valuable insight.
- CamperBob2 2y ago
  These smug pilots have lost touch with the down-to-earth lives and concerns of ordinary passengers like us. Let's see a show of hands: who thinks I should fly the plane?
  - eru 2y ago
    You should link to the source of your quote.
    - CamperBob2 2y ago
      Arguably so, but all I can find are other plagiarized copies. :-P The New Yorker's paywall has successfully obscured the origin of the joke, so that's on them, as far as I'm concerned.
      - eru 2y ago
        Yes. Well, instead of an actual web link, you could just mention that it's from the New Yorker.
altcognito 2y ago
Why do you say pseudo blinding, it seems like it is blind in that the author doesn’t know if he is taking the test or not. Now you can argue that there isn’t enough time between samples, or he needs more subjects but he was blind to whether he was taking it that day or not.
- jrootabega 2y ago
  If the author felt good on a particular day for whatever reason, and then learned they had taken the active substance, their reports are contaminated forever. It works the other way, too. It works any way you slice it.
brothrock 2y ago
N=1 is addressed, see outcome predictions. N=1 comes with caveats, of course, but a study like this, with a proven harmless supplement, should be welcomed and praised. It is clearly a step forward from what you can watch about theanine on YouTube or TikTok. I consider this a work of citizen science. While it should not be taken for more than it is, it’s a great example of how someone can experiment without a high burden.
- ryandrake 2y ago
  That's a pretty low bar though. OK, it's one step up from a monetized YouTube video that boils down to "It works--Trust me, bro." I still wouldn't really call it citizen science.
- matthewdgreen 2y ago
  N=1 studies aren’t evil. They’re just pretty close to the entire history of pre-modern medicine that led us to bad evidence. My concern here is not that someone is sharing their opinions, it’s the fact that the person doing this explicitly heaps derision on the “placebo people” (or some other phrasing) and then heaps praise on other people doing N=1 studies and proceeds to do one. This stuff all needs to be treated with humor, good faith, and then extreme skepticism about any result it produces.
jrootabega 2y ago
Hell, I'd say it's an 0<=N<1 because it involves subjective mood reporting, and there was no participant who was not contaminated by flaws in the methodology.
tomrod 2y ago
Aye. Completely agree. Path dependence matters, which is why you can't just look at pre/post action.
azeirah 2y ago
Any experiment you perform on yourself has N=1. Self-science isn't as robust as gold standard double blind blablabla PhD trials, but come on. How else are you going to find out whether a particular diet or medication works for you specifically? It's ALWAYS N=1.
- hartator 2y ago
  And a honest n=1 is better than a double blind n=1000 but outcome-interested study. It’s so easy to make data tell the story you want.
cortesoft 2y ago
N=1 trial is great if you are trying to figure out what works for you as an individual
bonestamp2 2y ago
> This is an N=1 trial Sure, OP addressed that and said it would be especially useful more people did it and "if we find a way to collectively interpret the data".
Enginerrrd 2y ago
>Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. This is only an appropriate criticism in so far as you want to make conclusions about theanine as an intervention in the broader population. It is however, perhaps much BETTER than large N trials if the author wishes to draw conclusions about how theanine affects THEM.

episteme 2y ago

You could argue that this N is the only N that matters though.

malfist 2y ago
Not too me though. Especially not to me if I'm trying to decide to supplement l-theanine
- deleted 2y ago
  [deleted]
- ryandrake 2y ago
  Right. The only N that you can draw any conclusions about is the author himself. So, why even publish it? The "results" are not applicable to anyone reading it. This is the health version of the software industry's "It works on my system!"
  - episteme 2y ago
    Publish it is quite a loaded word here. Almost every blog or post you read is from the point of view of a single person. There's nothing wrong with him putting his experience out into the world.

robwwilliams 2y ago

Complete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather see less data? Or are you volunteering to be test subject #2? Show us how to do it right Dr. M.! In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements. Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator. Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint. https://pubmed.ncbi.nlm.nih.gov/36702864/ https://pubmed.ncbi.nlm.nih.gov/36702864/

grafmax 2y ago
I think social media discussions of science would be better informed by the concept of a ‘hierarchy of evidence’. Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.
- abirch 2y ago
  Here’s a relevant xkcd https://xkcd.com/882/ https://xkcd.com/882/ For the small sample
deleted 2y ago
[deleted]
jrootabega 2y ago

wslh 2y ago

In science, an n=1 experiment isn’t discarded; instead, it adds information that can guide future experiments.

derlvative 2y ago

Noooooo you can’t just run independent experiments you need institutions and phds and bureaucracy and gold plating nooooo

smohare 2y ago
It’s not about elitism. It’s that there are so many confounding factors that even a well-informed approach makes such a study comtain very little of value Comments like yours expose a particularly distasteful amount of hubris.
- derlvative 2y ago
  You don’t like it you don’t have to read the blog article. I assure you are not the intended audience. For the rest of us it provided valuable insight.
CamperBob2 2y ago
These smug pilots have lost touch with the down-to-earth lives and concerns of ordinary passengers like us. Let's see a show of hands: who thinks I should fly the plane?
- eru 2y ago
  You should link to the source of your quote.
  - CamperBob2 2y ago
    Arguably so, but all I can find are other plagiarized copies. :-P The New Yorker's paywall has successfully obscured the origin of the joke, so that's on them, as far as I'm concerned.
    - eru 2y ago
      Yes. Well, instead of an actual web link, you could just mention that it's from the New Yorker.

altcognito 2y ago

Why do you say pseudo blinding, it seems like it is blind in that the author doesn’t know if he is taking the test or not. Now you can argue that there isn’t enough time between samples, or he needs more subjects but he was blind to whether he was taking it that day or not.

jrootabega 2y ago
If the author felt good on a particular day for whatever reason, and then learned they had taken the active substance, their reports are contaminated forever. It works the other way, too. It works any way you slice it.

brothrock 2y ago

N=1 is addressed, see outcome predictions. N=1 comes with caveats, of course, but a study like this, with a proven harmless supplement, should be welcomed and praised. It is clearly a step forward from what you can watch about theanine on YouTube or TikTok. I consider this a work of citizen science. While it should not be taken for more than it is, it’s a great example of how someone can experiment without a high burden.

ryandrake 2y ago
That's a pretty low bar though. OK, it's one step up from a monetized YouTube video that boils down to "It works--Trust me, bro." I still wouldn't really call it citizen science.
matthewdgreen 2y ago
N=1 studies aren’t evil. They’re just pretty close to the entire history of pre-modern medicine that led us to bad evidence. My concern here is not that someone is sharing their opinions, it’s the fact that the person doing this explicitly heaps derision on the “placebo people” (or some other phrasing) and then heaps praise on other people doing N=1 studies and proceeds to do one. This stuff all needs to be treated with humor, good faith, and then extreme skepticism about any result it produces.

jrootabega 2y ago

Hell, I'd say it's an 0<=N<1 because it involves subjective mood reporting, and there was no participant who was not contaminated by flaws in the methodology.

tomrod 2y ago

Aye. Completely agree. Path dependence matters, which is why you can't just look at pre/post action.

azeirah 2y ago

Any experiment you perform on yourself has N=1. Self-science isn't as robust as gold standard double blind blablabla PhD trials, but come on. How else are you going to find out whether a particular diet or medication works for you specifically? It's ALWAYS N=1.

hartator 2y ago
And a honest n=1 is better than a double blind n=1000 but outcome-interested study. It’s so easy to make data tell the story you want.

cortesoft 2y ago

N=1 trial is great if you are trying to figure out what works for you as an individual

bonestamp2 2y ago

> This is an N=1 trial Sure, OP addressed that and said it would be especially useful more people did it and "if we find a way to collectively interpret the data".

Enginerrrd 2y ago

>Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. This is only an appropriate criticism in so far as you want to make conclusions about theanine as an intervention in the broader population. It is however, perhaps much BETTER than large N trials if the author wishes to draw conclusions about how theanine affects THEM.