64 bit pointers require double the space in memory, hence your caches inside the processor can only hold half the number of pointers. This means more cache miss

64 bit pointers require double the space in memory, hence your caches inside the processor can only hold half the number of pointers. This means more cache misses and thus more waiting for abysmally slow (when compared with L1/L2 caches) RAM. The impact of this is AFAIK generally in the single-digit percentage area, but if you have an algorithm that just fits everything into the caches with 32 bit pointers, the change to 64 bit pointers might push it over the edge, which can then result in significant (>= double-digit percentage) performance decrease.