Even RAM has overhead for random access. It takes time to change which row of the memory you're accessing, and there's also a "burst mode" that reads or writes

Even RAM has overhead for random access. It takes time to change which row of the memory you're accessing, and there's also a "burst mode" that reads or writes a batch of consecutive locations with reduced overhead. [EDITED to add:] And of course cacheing means that the RAM+cache system collectively makes random accesses slower, and would do even if all accesses to RAM took exactly the same amount of time.