Because addresses are longer they will take up more space, which will lead to your CPUs instruction-cache (I-Cache) being more full, which directly translates i

Because addresses are longer they will take up more space, which will lead to your CPUs instruction-cache (I-Cache) being more full, which directly translates into performance losses due to needed things being evicted from I-Cache. The same is true for L1/L2/L3 caches too. The same is true to a much lesser extent for taking up more space in ram, and to an even lesser extent for reading things from storage.