The downside is that the secure versions take longer to download if you don't get a local cache hit. I did a speed test and loaded jQuery 20 times from Google u

The downside is that the secure versions take longer to download if you don't get a local cache hit. I did a speed test and loaded jQuery 20 times from Google using my local machine with an unprimed cache. For the http version, the average load time was 90ms. For the https version, the average load time was 192ms. That’s over twice as slow for the SSL version. Since you can't guarantee the cache hit, it's a trade-off that doesn't have an obvious answer.