You hit on a lot of important points. The key here is that they are making a claim about memory, a psychological construct, that is being represented by some of

You hit on a lot of important points. The key here is that they are making a claim about memory, a psychological construct, that is being represented by some of their measures (batteries). But their batteries also purport to measure other psychological constructs, too. To connect this with the slot machine analogy, it might be like if groups of slot machines had different colors, and they chose the color that yielded the best results. > but would coming back and saying "these five designs have a higher than chance level of winning!" be an incorrect conclusion? It would be if you didn't take into account that you analyzed 20 machines in your analysis .