That's indeed true - and why for subjective eyeball moving picture tests you should also do things like cut out a 20 second piece of a raw/y4m file, encode it w

That's indeed true - and why for subjective eyeball moving picture tests you should also do things like cut out a 20 second piece of a raw/y4m file, encode it with various x265 settings, and then put those 4 or 5 pieces in a VLC playlist so you can quickly compare them in series. For objective comparisons there is of course VMAF, which is an essential tool for doing automated comparisons of videos vs their uncompressed original. One of the reasons why VMAF was created is that subjective eyeball evaluation of codecs (whether still or moving) will vary between person to person, and is very labor intensive. But still image PNGs of frames also serve a useful purpose when dealing with lower bitrate videos, for your personal opinion comparison of blockiness, color banding, blobs of color in areas that are mostly the same color, etc.