It is very difficult to measure data set quality. The goal of a data set to give you insight into how well an approach performs for any given data point (in th

It is very difficult to measure data set quality. The goal of a data set to give you insight into how well an approach performs for any given data point (in this case an image and associated label). The problem is that we can not collect all the possible data points, so we have to settle for what we hope to be a fair sample of all data points. While bigger usually means better, if for example there was a bias in how the data points were selected the data set can actually be worse than a smaller one where there was no such bias. Building a good data set really depends on the current understanding of the nature and difficulty of the task, both on the part of the scientific field and the researchers working on the data set. The very same problem exists in the medical domain. How do you select the right patients to evaluate for a treatment? How do you know that there is not a specific genetic, racial, gender, etc. trait that leads to side effects? The answer is of course that you can not know with absolute certainty unless you include every human on the planet (and then there is the question as-of-yet unborn humans), but you can use your experience and medical knowledge to try to select patients (your data set) for the medical trial to minimise the possible impact of as-of-yet unknown side effects.