You're right that it is a problem, but defining objective criteria before knowing exactly what it is that you're looking for is potentially just as bad. This is

You're right that it is a problem, but defining objective criteria before knowing exactly what it is that you're looking for is potentially just as bad. This is part of the reason folks like Tukey defended statistical graphics early on: exploratory data analysis is tremendously important, and it's rare that you would know exactly what you're looking for. Also, this problem of making sure what you're seeing is actually in the data is one that researchers are actively working on! There's a paper due to come out at a top visualization conference on the relation between formal hypothesis testing and visualizations which goes to the heart of your "easily tricked visual cues" comment. It's not currently available (and I'm not a co-author), but if you're interested, be on the lookout for "Graphical Inference for InfoVis"