5 ms·

Because.... >While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropp

Because.... >While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken That is not blinding

sgc 2y ago
I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial. Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless. As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.
- kqr 2y ago
  > As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? It gives a hint of how well the blinding worked.
  - sgc 2y ago
    Yes, that is a good point. But it also causes him to continuously think about indicators to determine what he has taken. He is constantly trying to punch the veil on this "blind" study instead of doing everything he can to avoid that. Are they really the exact same weight? Do they really feel the same in his hand, mouth and throat? Does one have a slightly different taste - after all, no filling process will leave the outside of the capsule 100% free of powder, etc? Very subtle differences could manifest themselves over 16 months.
- fc417fc802 2y ago
  If your sample is blinded yet you consistently guess correctly, then presumably either 1. you failed at blinding or 2. there is a strong and discernible effect, regardless of what your other metrics might say (your other metrics could always be flawed after all). > no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out. But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.
  - sgc 2y ago
    He has 94 data points. Not nearly enough to average out so many potentially confounding variables, and there is no way to know they would. That will be the case in almost all N=1 experiments. Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. This would be a probable case for many people trying to use theanine in this way. I could never accept that we should presume it would average out, and thus I consider logging potentially confounding variables essential to a valid experiment of this type of uncontrolled experiment. Your hypothesis would be much more relevant in a larger experiment of course.
    - fc417fc802 2y ago
      > Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. That would not be a problem regarding the averaging I referred to, although it could well pose a problem for measurement depending on how it interacted with the selected metrics. Note that the averaging I refer to is not regarding all possible values of some metric, but rather any discrepancy in the distribution of metrics which we expected to follow the same distribution between the sample and the control. I think maybe there's a misunderstanding? It seems that we both agree that a variety of additional variable should be logged. I was not suggesting to omit them, but rather to use discrepancies in them to detect fundamental issues with the data or study design. I would also expect larger studies to do the same where possible. At 94 data points it is entirely possible that there would be outliers that would have averaged out for a larger N but did not. In such a scenario the presence of such outliers should then be taken to indicate a problem with the data (ie the more discrepancies you observe, the less you should trust the data).

sgc 2y ago

I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial. Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless. As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.

kqr 2y ago
> As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment? It gives a hint of how well the blinding worked.
- sgc 2y ago
  Yes, that is a good point. But it also causes him to continuously think about indicators to determine what he has taken. He is constantly trying to punch the veil on this "blind" study instead of doing everything he can to avoid that. Are they really the exact same weight? Do they really feel the same in his hand, mouth and throat? Does one have a slightly different taste - after all, no filling process will leave the outside of the capsule 100% free of powder, etc? Very subtle differences could manifest themselves over 16 months.
fc417fc802 2y ago
If your sample is blinded yet you consistently guess correctly, then presumably either 1. you failed at blinding or 2. there is a strong and discernible effect, regardless of what your other metrics might say (your other metrics could always be flawed after all). > no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out. But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.
- sgc 2y ago
  He has 94 data points. Not nearly enough to average out so many potentially confounding variables, and there is no way to know they would. That will be the case in almost all N=1 experiments. Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. This would be a probable case for many people trying to use theanine in this way. I could never accept that we should presume it would average out, and thus I consider logging potentially confounding variables essential to a valid experiment of this type of uncontrolled experiment. Your hypothesis would be much more relevant in a larger experiment of course.
  - fc417fc802 2y ago
    > Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. That would not be a problem regarding the averaging I referred to, although it could well pose a problem for measurement depending on how it interacted with the selected metrics. Note that the averaging I refer to is not regarding all possible values of some metric, but rather any discrepancy in the distribution of metrics which we expected to follow the same distribution between the sample and the control. I think maybe there's a misunderstanding? It seems that we both agree that a variety of additional variable should be logged. I was not suggesting to omit them, but rather to use discrepancies in them to detect fundamental issues with the data or study design. I would also expect larger studies to do the same where possible. At 94 data points it is entirely possible that there would be outliers that would have averaged out for a larger N but did not. In such a scenario the presence of such outliers should then be taken to indicate a problem with the data (ie the more discrepancies you observe, the less you should trust the data).