I tried with downsample and up sample image with opencv using INTER_AREA then compare them with SSMI. After that I pick the threshold to determine which is bad

I tried with downsample and up sample image with opencv using INTER_AREA then compare them with SSMI. After that I pick the threshold to determine which is bad and which is good, but it still not work well. I struggle when determine the threshold, some blur image have higher score than high quality image, but other blur image has lower score than high quality image. So I wonder where i'm wrong or have some better approaches to resolve this.