Indeed, we need to look at external factors like time/energy expenditure to determine the efficiency of the operation in order to have a hope at constraining it

Indeed, we need to look at external factors like time/energy expenditure to determine the efficiency of the operation in order to have a hope at constraining its method. If the NSA were to suddenly publish a list of 1,000,000,000 broken SHA512 keys, and there was no sign of any spikes in energy expenditures, we could strongly suspect that they did not generate them from brute force means. Similarly, if we have hundreds of millions of distinct species on the planet, and only a few years of time on the planet (relative to the rate at which we've observed beneficial random mutations actually occurring), we can suspect that the bulk of the speciation did not occur through stochastic means.