Yes, it is the "Family-wise error rate". You need to perform "correction for multiple tests". The simplest, most conservative form is Bonferroni correction [1]

Yes, it is the "Family-wise error rate". You need to perform "correction for multiple tests". The simplest, most conservative form is Bonferroni correction [1], where you just divide the required significance threshold by the number of tests you perform. In your example, if you required significance at the 99% level (1% chance a false positive, or "Type I error"), then for 100 datasets, each individual one would require significance at the 99.99% level (0.01% false positives for each data set). (99.99%)^100 = 99.005%. This does not require any assumptions about the independence of the tests. Independent tests are the worst case, so this over-corrects if there is actual correlation between the measurements. There are other, more complicated correction methods that do not give up as much statistical power (i.e., produce fewer false negatives), but that may require making more assumptions. [1] https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction