11 ms·
This is great. The author defines their own metrics, is doing their own A/B tests and publishes their interpretation plus the raw data. Imagine a world where al
by mg 2y ago
This is great. The author defines their own metrics, is doing their own A/B tests and publishes their interpretation plus the raw data. Imagine a world where all health blogging was like that.
Personally, I have not published any results yet, but I have been doing this type of experiments for 4 years now. And collected 48874 data points so far. I built a simple system to do it in Vim:
https://www.gibney.org/a_syntax_for_self-tracking https://www.gibney.org/a_syntax_for_self-tracking
I also built a bunch of tooling to analyze the data.
I think that mankind could greatly benefit from more people doing randomized studies on their own. Especially if we find a way to collectively interpret the data.
So I really applaud the author for conducting this and especially for providing the raw data.
Reading through the article and the comments here on HN, I wish there was more focus on the interpretation of the experiment. Pretty much all comments here seem to be anecdotal.
Let's look at the author's interpretation. Personally, I find that part a bit short.
They calculated 4 p-values and write:
Technically, I did find two significant results.
I wonder what "Technically" means here. Are there "significant results" that are "better" than just "technically significant results"?
Then they continue:
Of course, I don’t think this
means I’ve proven theanine is harmful.
So what does it mean? What was the goal of collecting the data? What would the interpretation have been if the data would show a significant positive effect of Theanine?
It's great that they offer the raw data. I look forward to taking a look at it later today.
- aszantu 2y agoI found, I take Abt. 4 days to react to stuff. Except for those cheap cooking oils, I get migraines within a time-frame of about 3 hrs.
- brodo 2y agoYour self-tracking syntax looks neat. How do you track if you are not in front of a computer?
- mg 2y agoI write into a text file on my phone. And later move those lines into the file on my computer. It's a bit cumbersome, but not too much. But yeah, I look forward to when I finally get around to writing an app for it.
- bosie 2y agodo you use an app and shortcuts (or something similar) to speed this up?
- mg 2y agoOn the phone I don't. In Vim, I use multiple shortcuts that I made for this use case.
- yjftsjthsd-h 2y agoYou can use vim via termux, which is nice when you already have things in version control. Alternatively, may I suggest the DailyLog app in F-Droid?
- kortilla 2y agoWell the issue is that an experiment with 1 person can’t prove much because you can’t have a control group. Too much other stuff is changing in a single persons life that could account for all observed side effects. You also have latent side effect issues. A person could smoke for 10 years, not smoke for another 10, and then conclude that smoking doesn’t cause cancer. Then they get lung cancer 20 years later. Excellent data and statistics is not sufficient for a good experiment
- mg 2y agoAn experiment of 1 person can very well produce useful data. It depends on the setup of the experiment. Imagine an experiment where a person's thumb gets randomly hit with either a hammer or a feather once per day. And they then subjectively rate the experience. After 1000 days of collecting data, I doubt that we would wrongly come to the conclusion that the hammer treatment leads to the nicer outcome. The setup of the Theanine experiment which is the basis of this thread looks good on first sight. I have the feeling that the interpretation could use more thought though.
- deleted 2y ago[deleted]
- tomalbrc 2y agoWhat a weird take, in 99.999% of cases you don’t have such a black/white contrast
- mg 2y agoSure. But even when you add noise to the described experiment, you get useful data. That is the point I am making: Experiments of a single person can be useful. The critics of single person experiments usually come up with examples vastly different than the Theanine experiment described here. With long term experiments which are only conducted once. But the Theanine experiment was looking for a short term effect and can be conducted many times. The hammer experiment I made up would be an extreme example of this type of experiment which leads itself well to be conducted by a single person. What I am trying to point out is that if you are a skeptic, it would be better to try and find weaknesses in the experiment at hand. Not making up completely different experiments.
- kqr 2y ago> So what does it mean? What was the goal of collecting the data? What would the interpretation have been if the data would show a significant positive effect of Theanine? I think what the author is saying is that for them to bother with theanine on a permanent basis, it would have to have shown an effect large enough to be apparent just from plotting. In other words, they mean technical significance as opposed to clinical significance. A small effect can be statistically verifiable without being meaningful in practice.
- mg 2y agoan effect large enough to be apparent just from plotting And how large is that? Without putting a number on it, how do we come to the conclusion that the effect is not large enough? That it didn't show in their sample of data points could have been just random chance. But before we take the measured effects at face value, I think it's important think about them more. They report significant p values of their success in predicting if the capsule holds Theanine and also for the effect of the capsule when it holds Theanine. Both negative correlations. My first thought reading this is that the Placebo tasted more like Theanine and thinking they took Theanine had a positive effect on the outcome.
- stevage 2y agoThey were looking for a clear effect. they did not find one.
- matthewdgreen 2y agoThis is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting this much effort into any medication trial makes it much more likely that you’re going to be incentivized to find effects that don’t exist. I think it’s nice that the author admits that they found nothing, but statistically, worthless drugs show effects in much better-designed trials than this one: it’s basically a coin toss.
- episteme 2y agoYou could argue that this N is the only N that matters though.
- malfist 2y agoNot too me though. Especially not to me if I'm trying to decide to supplement l-theanine
- deleted 2y ago[deleted]
- ryandrake 2y agoRight. The only N that you can draw any conclusions about is the author himself. So, why even publish it? The "results" are not applicable to anyone reading it. This is the health version of the software industry's "It works on my system!"
- episteme 2y agoPublish it is quite a loaded word here. Almost every blog or post you read is from the point of view of a single person. There's nothing wrong with him putting his experience out into the world.
- robwwilliams 2y agoComplete injustice to this lovely study. Why do you say unblinded? Why do you insult a time series study as “dressing up with lots of data”? Would you rather see less data? Or are you volunteering to be test subject #2? Show us how to do it right Dr. M.! In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements. Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator. Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint. https://pubmed.ncbi.nlm.nih.gov/36702864/ https://pubmed.ncbi.nlm.nih.gov/36702864/
- robwwilliams 2y ago“Technically” here could imply “not corrected for multiple tests”. But the typical qualifier I use is “nominally significant” when I don’t apply an correction for multiple tests. Or “not using a one-way t test”. The most appropriate null hypothesis in this lovely study is “does theanine REDUCE anxiety”, not “does theanine change anxiety either up or down”. What impressed me most is the suggestion for an improved experimental design to remove his temporal drift by using 100 pre-loaded envelopes and only decoding the results at the end.
- azalemeth 2y ago> The most appropriate null hypothesis in this lovely study is “does theanine REDUCE anxiety”, not “does theanine change anxiety either up or down”. I disagree with this. You have a prior belief that theanine might reduce anxiety; if you wanted to you could codify that subjective belief and perform some variety of Bayesian hypothesis test [1] and compute a Bayes factor. The main reason that one-sided tests are advocated for is power; that is often the same as having a prior belief in disguise. Why not quantify it? However, scientifically, if the data conclusively show that "theanine increases anxiety" that is a meaningful, non-artefactual result: it is hugely important to be sensitive to the answer 'you are wrong' and may well ironically spur development in a direction to help understand what is going on. I personally think that one sided tests are best avoided except in the case where it is physically impossible to have an effect in the other direction. Examples of this are rare, but they do occasionally exist. [1] https://mspeekenbrink.github.io/sdam-book/ch-Bayes-factors.html https://mspeekenbrink.github.io/sdam-book/ch-Bayes-factors.h...
- robwwilliams 2y agoSure. I can see your point, but the most reasonable posterior probability of the null given the biohacker community’s belief is one-tailed. This also gives more power to reject the null.
- ac29 2y ago> The most appropriate null hypothesis in this lovely study is “does theanine REDUCE anxiety”, not “does theanine change anxiety either up or down”. Neither of those is a hypothesis, which require a prediction not just a question. The null hypothesis for this experiment would be "Theanine has no effect on stress".
- K0balt 2y agoCitizens science used to be much more common and there were several publications initially built around the concept. Unfortunately, the idea has all but been obliterated by inadequate public education and the chronic lack of time that now embattles the middle classes.
- luqtas 2y agofortunately* citzen science was replaced by solid methods... this is a long fancy self reported post on a metric that is easily figured out by biomarkers (stress levels). what's the consequences if the author decided to make some claim about a substance?
- luqtas 2y agocouldn't edit but here it's how science measure stress: https://pmc.ncbi.nlm.nih.gov/articles/PMC8434839/ https://pmc.ncbi.nlm.nih.gov/articles/PMC8434839/
- K0balt 2y agoI think you are ignoring the fact that citizen science was alive and very well up into the 1980s. It was never a replacement for professional science, but rather citizen scientists often observed things that were deemed worth of follow-up by academic researchers. Citizens scientists rarely post “facts” but rather interesting avenues for research or further investigation. Part of being an educated citizen scientist is to understand the limitations of your knowledge, data, and methods. Quacks and cranks, on the other hand, are always making grand new “discoveries” lol. Anyone who has a decent education can make observations apply the scientific method. I say this coming from a family of actual scientists from molecular biology to particle physics, who will tell you the same, and also give credit to the multitude of citizen scientists who have done just that. As for myself, I’ll stick to engineering.
- luqtas 2y agoi think anyone researching whatever in their home is doing 'science' but this isn't the case! i linked in an answer to my post what science knows when measuring stress... it's far away from a self-reported subjective question i think the problem is whatever you post on the internet is politic. it may have a huge reach, and so far so good people trying far from lethal doses of theanine. now, what if an influencer wants to make their stuff look intellectual and go buy pure caffeine? how about the people coming after wards? one thing is a 3D-print project at your garage on keyboards ergonomics, home automation, another is rock-climbing gear and another is substances with reachable lethal doses where the masses can buy (not the case here but again, i cited caffeine but there's much more). the author has a stellar presentation but they seriously researched what science does when measuring stress? they didn't even considered The Perceived Stress Scale (PSS), which is used on professional research. i stand against public posts with badly methods (or the complete lack of research) suppressed by fancy graphics on substance use/abuse. i also bet isn't that hard to buy empty pills with opaque color
- quijoteuniv 2y agoI think is great that folk self observe, and that is a key to a lot, everyone ought to find what works for themselves, however there is a tendency to want to fix things with pills. There is a fundamental error there, if successful you might damp your symptoms but then you start a Whac-A-Mole (whac-a-symptom) but you are not looking at the root. The problem will come in another form. Have you consider that the symptom is a defence mechanism ?What about Taichi, mindfulness ness practices, yoga why not a study on that? It is definitely more work… than taking some pills. No I am not against taking supplements. But ultimately is a workaround you are not fixing the bug
- Forgeties79 2y agoI’m not sure if this is your intention, but I take issue with the implication that “pills” (aka “medication” and “medicine,” “pills” has a generally negative connotation) aren’t often the solution to the root problem. It is great if you can solve things with diet, mindfulness, etc. But sometimes you need medical intervention and yes sometimes that means you need to take medication in the form of pills. There are millions of people who need that regardless of how they change their lifestyle or regulate their emotions/mental health without them. Basically I don’t like the idea that you are implying medication is a bandaid and not ever the actual solution. If I misreading your comment my apologies
- quijoteuniv 2y agoThis is good point, thank you, lot of people needs medicine, and i am glad they are accesible. My point goes more towards this: Imagine someone runs a marathon everyday, after a couple of months, start experiencing pain and inflammation. Do they need painkillers or do they need to ask themselves why they are running a marathon everyday? I am not against taking some painkillers as long as you also ask yourself the question. Do i really need to run a marathon everyday? I do not want to cure disease with yoga, no, but you should give it a go, whatever works for you, self observe, train a bit, walk in the forest, surf, meditate. Wellbeing cannot be achieved only with medicine.
- 2y ago
- jvanderbot 2y agoReally fantastic. I've been using theanine for a long time, but never for any of these purported benefits. And the benefits I do use it for would be near impossible to measure. I just use it to make a over-caffinated monkey brain state tend toward a "lock-in" mental state. That's super hard to measure, and just as likely the theanine is a trigger for a mental deep dive that could just as well be sugar. But the ritual works, and that's what's important to me. It just took the intial "It helps mellow out caffeine for deep focus" idea to establish the ritual. Science? No. Effective? Yeah, I think so.
- ninetyninenine 2y ago>Pretty much all comments here seem to be anecdotal. The entire blog post was anecdotal.
- gus_massa 2y ago>> Technically, I did find two significant results. The problem is that he is comparing two very different things, the level when he took the pill and the level one hour after that. So it's not surprising that they are different. Let's imagine a very very very stupid experiment, where the problem is more obvious. Does Coca Cola or Pepsi improve luck? N=1000000, double blind randomized controlled trial. 1) Each subject flips a coin. tail=0, head=1. 2) They drink a glas of soda, 50% Coke or 50% Pepsi, that is served in a hidden place and nor the subject or the experimenter know which one. 3) They roll a dice (an usual one, D6) Results: * Average before Coke = 0.5002 * Average after Coke = 3.5005 * Average before Pepsi = 0.5004 * Average after Pepsi = 3.5003 So the conclusion is that Coke improves the average (p<1E-a-lot) and Pepsi improves the average (p<1E-a-lot). Both are "technically" statistically significant (but it's caused by a horrible experiment design). Unsurprisingly, the difference in the average after drinking Coke or Pepsi is not statistically significant (p<.something). (I'm too lazy to run a simulation now, but it's not difficult to get realistic averages and p values.) In conclusion, the useful result is the comparison of the anxiety after taking both drugs, not the difference of before and after taking them. As the article says: >> So I propose a new rule: Blind trial or GTFO.
- silent_cal 2y agoFrom what I've seen, most scientists don't even publish their raw data.
- angg 2y ago> https://www.gibney.org/a_syntax_for_self-tracking https://www.gibney.org/a_syntax_for_self-tracking Loved reading this as I have been on a similar self tracking journey of my own. I am curious about your thoughts on mobile support? Specifically, have you implemented anything to make adding/editing events while away from your computer easier? Or perhaps you feel it is not important for your use case? Personally, I've found that being able to log an event in an instant, or just on a whim, is invaluable for capturing the sort of data I care about tracking the most (like what I just ate or my mood), and I find too much friction causes many events simply go unlogged as they almost always feel too unimportant or mundane in retrospect to keep a mental note of and track later. I started out with a text file system similar to yours but I've since begrudgingly resorted to google forms for most of my tracking needs. It's nice that I can arbitrarily add and remove fields while still keeping everything relatively structured and parseable, and it does have (albeit inferior) revision history. But I dislike not having ownership and control over where my data is stored and absolutely *dread* waiting for the form to load every time I want to log an event, especially on mobile.